Increasing number of tasks causes ImportError

[FurkanTheHuman]

Checklist

• I have verified that the issue exists against the master branch of Celery. (v5.0.2)
• This has already been asked to the discussion group first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• [x ] I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version: 5.0.2

celery report Output:

software -> celery:5.0.2 (singularity) kombu:5.0.2 py:3.6.9
            billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
            kernel version:5.4.0-53-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost/

broker_url: 'redis://localhost:6379//'
result_backend: 'redis://localhost/'
deprecated_settings: None
result_extended: True

Steps to Reproduce

Required Dependencies

• Minimal Python Version: 3.6.9

• Minimal Celery Version: 5.0.2

• Minimal Kombu Version: 5.0.2

• Minimal Broker Version: Redis server v=6.0.8 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=75cef67090587c6

• Minimal Result Backend Version: Redis server v=6.0.8 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=75cef67090587c6

• Minimal OS and/or Kernel Version: Linux Machine 5.4.0-53-generic Unsuccessful import in views.py #59~18.04.1-Ubuntu SMP Wed Oct 21 12:14:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Minimal Broker Client Version: N/A or Unknown

• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aioredis==1.3.1
alabaster==0.7.8
amqp==5.0.2
appdirs==1.4.4
apturl==0.5.2
argon2-cffi==20.1.0
asgiref==3.3.1
asn1crypto==0.24.0
astroid==1.6.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.2.0
autobahn==20.7.1
Automat==20.2.0
Babel==2.3.4
backcall==0.2.0
beautifulsoup4==4.9.0
billiard==3.6.3.0
black==20.8b1
bleach==3.2.1
blinker==1.4
Brlapi==0.6.6
bs4==0.0.1
celery==5.0.2
certifi==2020.6.20
cffi==1.14.3
channels==2.4.0
channels-redis==3.1.0
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
cloudpickle==0.5.2
colorama==0.4.3
colored==1.4.2
command-not-found==0.3
constantly==15.1.0
cryptography==3.1.1
cupshelpers==1.0
cycler==0.10.0
daphne==2.5.0
dataclasses==0.8
decorator==4.4.2
defer==1.0.6
defusedxml==0.6.0
distlib==0.3.1
distro==1.5.0
distro-info===0.18ubuntu0.18.04.1
Django==3.1.3
django-allauth==0.42.0
django-automated-logging==6.0.0
django-cors-headers==3.5.0
django-filter==2.4.0
django-ipware==3.0.2
django-picklefield==3.0.1
django-rest-auth==0.9.5
djangorestframework==3.12.1
docutils==0.12
elasticsearch==7.9.1
entrypoints==0.3
fastapi==0.61.2
feedparser==5.2.1
filelock==3.0.12
Flask==1.1.2
Flask-Cors==3.0.8
future==0.18.2
greenlet==0.4.12
h11==0.11.0
hiredis==1.1.0
html5lib==0.999999999
httpie==0.9.8
httplib2==0.9.2
hyperlink==20.0.1
idna==2.10
imagesize==0.7.1
importlib-metadata==2.0.0
importlib-resources==3.0.0
incremental==17.5.0
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.4
itsdangerous==1.1.0
jedi==0.17.2
Jinja2==2.11.2
joblib==0.17.0
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyterlab==2.2.9
jupyterlab-pygments==0.1.2
jupyterlab-server==1.2.0
kafka-python==2.0.2
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.3.1
kombu==5.0.2
language-selector==0.1
launchpadlib==1.10.6
lazr.restfulclient==0.13.5
lazr.uri==1.0.3
lazy-object-proxy==1.3.1
logilab-common==1.4.1
louis==3.5.0
lxml==4.5.0
macaroonbakery==1.1.3
Mako==1.0.7
Markdown==3.3
MarkupSafe==1.1.1
marshmallow==3.9.1
matplotlib==3.3.3
mccabe==0.6.1
mistune==0.8.4
msgpack==1.0.0
mypy-extensions==0.4.3
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
neovim==0.2.0
nest-asyncio==1.4.2
netifaces==0.10.9
notebook==6.1.4
numpy==1.19.4
numpydoc==0.7.0
oauth==1.0.1
oauthlib==3.1.0
olefile==0.45.1
packaging==20.4
pandas==1.0.3
pandocfilters==1.4.3
parso==0.7.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.0.1
prometheus-client==0.8.0
prompt-toolkit==3.0.8
protobuf==3.0.0
psutil==5.4.2
ptk==1.3.8
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycodestyle==2.3.1
pycparser==2.20
pycrypto==2.6.1
pycups==1.9.73
pydantic==1.7.2
pyflakes==1.6.0
Pygments==2.7.2
pygobject==3.26.1
PyHamcrest==2.0.2
PyJWT==1.5.3
pylint==1.8.3
pymacaroons==0.13.0
PyNaCl==1.1.2
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyRFC3339==1.0
pyrsistent==0.17.3
PySocks==1.7.1
python-apt==1.6.5+ubuntu0.3
python-dateutil==2.8.1
python-debian==0.1.32
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2020.4
pyxdg==0.25
PyYAML==3.12
pyzmq==19.0.2
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.3.1
redis==3.5.3
regex==2020.7.14
reportlab==3.4.0
requests==2.24.0
requests-futures==1.0.0
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
roman==2.0.0
rope==0.10.5
scikit-learn==0.23.2
scipy==1.5.4
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Send2Trash==1.5.0
service-identity==18.1.0
simplegeneric==0.8.1
simplejson==3.13.2
six==1.15.0
soupsieve==2.0
Sphinx==1.6.7
spyder==3.2.6
sqlparse==0.4.1
starlette==0.13.6
stem==1.8.0
system-service==0.3
systemd-python==234
terminado==0.9.1
testpath==0.4.4
text-unidecode==1.3
threadpoolctl==2.1.0
toml==0.10.1
torch==1.7.0
torchvision==0.8.1
tornado==6.1
torrequest==0.1.0
traitlets==4.3.3
Twisted==20.3.0
txaio==20.4.1
typed-ast==1.4.1
typing-extensions==3.7.4.3
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unity-scope-calculator==0.1
unity-scope-chromiumbookmarks==0.1
unity-scope-colourlovers==0.1
unity-scope-devhelp==0.1
unity-scope-firefoxbookmarks==0.1
unity-scope-manpages==0.1
unity-scope-openclipart==0.1
unity-scope-texdoc==0.1
unity-scope-tomboy==0.1
unity-scope-virtualbox==0.1
unity-scope-yelp==0.1
unity-scope-zotero==0.1
unity-tweak-tool==0.0.7
urllib3==1.25.11
usb-creator==0.3.3
uvicorn==0.12.2
vine==5.0.0
virtualenv==20.0.35
wadllib==1.3.2
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
websockets==8.1
Werkzeug==1.0.1
wrapt==1.9.0
xkit==0.0.0
xonsh==0.9.24
zipp==3.4.0
zope.interface==5.1.2

Other Dependencies

N/A

Minimally Reproducible Test Case

## tasks.py
from celery import Celery
import os
from time import sleep
redis_url = os.getenv("REDIS_URL", default="redis://localhost")

app = Celery('tasks', backend=redis_url, broker=redis_url)

app.conf.update(
   result_extended=True
)

@app.task(bind=True)
def dyn_sleeper(self,s):
    print('Executing task id {0.id}, args: {0.args!r} kwargs: {0.kwargs!r}'.format(
            self.request))
    sleep(s)
    return s

## producer.py
from test import dyn_sleeper 
from time import sleep


task_list = []

for i in range(5, 15, 2):
    print("example task initiated, SLEEP:",i)
    task_list.append(dyn_sleeper.delay(i))



print("num of tasks:", len(task_list))

while True:
    completed_tasks = [task for task in task_list if task.ready() ]
    if len(completed_tasks) ==len(task_list):
        print("All tasks finished.")
        for c in task_list:
            print(">> ",c.name, c.args," :", c.result)
        break
    print("There is", len(completed_tasks), "completed tasks out of", len(task_list), "tasks")
    print(">> ", len(completed_tasks)/len(task_list)*100)
    sleep(1)
    
sleep(10)

Expected Behavior

Exit without an error

Actual Behavior

I have a tasks like this

## celery -A tasks worker -l INFO --concurrency=6
@app.task(bind=True)
def dyn_sleeper(self,s):
    print('Executing task id {0.id}, args: {0.args!r} kwargs: {0.kwargs!r}'.format(
            self.request))
    sleep(s)
    return s

When I start celery and run the task like this:

## producer.py
task_list = []

for i in range(5, 35, 2):
    print("example task initiated, SLEEP:",i)
    task_list.append(dyn_sleeper.delay(i))

I also check if all task finished:

## producer.py
while True:
    completed_tasks = [task for task in task_list if task.ready() ]
    if len(completed_tasks) ==len(task_list):
        print("All tasks finished.")
        for c in task_list:
            print(">> ",c.name, c.args," :", c.result)
        break
    print("There is", len(completed_tasks), "completed tasks out of", len(task_list), "tasks")
    sleep(1)

Everything works fine but after the producer.py finishes I get an error. I complete all the tasks but I just get this error while I exit from producer.py.

Traceback (most recent call last):
  File "/home/username/.local/lib/python3.6/site-packages/celery/result.py", line 391, in __del__
  File "/home/username/.local/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 186, in remove_pending_result
  File "/home/username/.local/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 194, in on_result_fulfilled
  File "/home/username/.local/lib/python3.6/site-packages/celery/backends/redis.py", line 174, in cancel_for
  File "/home/username/.local/lib/python3.6/site-packages/redis/client.py", line 3600, in unsubscribe
  File "/home/username/.local/lib/python3.6/site-packages/redis/client.py", line 3468, in execute_command
  File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 1192, in get_connection
  File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 567, in connect
  File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 630, in on_connect
  File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 401, in on_connect
ImportError: sys.meta_path is None, Python is likely shutting down

Weird thing is if, I don't use the loop above I don't get an error. Also if I don't use many tasks (tasks less than 20) again I don't get an error.

[thedrow]

I'm not sure what this error means. I've never seen it before.

[FurkanTheHuman]

There are some unrelated projects that shows the same error.

openai/gym#893
h5netcdf/h5netcdf#50
https://stackoverflow.com/questions/41480148/importerror-sys-meta-path-is-none-python-is-likely-shutting-down

Common theme is unclosed connections.

This might be related to my own general system setup. I will try to reproduce it on docker.

[FurkanTheHuman]

Didn't observe the same effect on docker.