3.1.25 >3.1.26 Regression: `ImportError: cannot import name '_detect_environment'`

[ciarancourtney]

I think this is due to 3.1.26 expecting Kombu 4.x, which AFAIK is not compatible with celery 3.x

ubuntu@ciaran:/opt/master$ venv/bin/pip install celery==3.1.26
Collecting celery==3.1.26
  Using cached celery-3.1.26-py2.py3-none-any.whl
Requirement already satisfied: billiard<3.4,>=3.3.0.23 in ./venv/lib/python3.6/site-packages (from celery==3.1.26)
Requirement already satisfied: pytz>dev in ./venv/lib/python3.6/site-packages (from celery==3.1.26)
Requirement already satisfied: kombu<3.1,>=3.0.37 in ./venv/lib/python3.6/site-packages (from celery==3.1.26)
Requirement already satisfied: amqp<2.0,>=1.4.9 in ./venv/lib/python3.6/site-packages (from kombu<3.1,>=3.0.37->celery==3.1.26)
Requirement already satisfied: anyjson>=0.3.3 in ./venv/lib/python3.6/site-packages (from kombu<3.1,>=3.0.37->celery==3.1.26)
Installing collected packages: celery
  Found existing installation: celery 3.1.25
    Uninstalling celery-3.1.25:
      Successfully uninstalled celery-3.1.25
Successfully installed celery-3.1.26

Stacktrace

/opt/master/venv/bin/python /opt/master/src/manage.py celeryd -lINFO
Traceback (most recent call last):
  File "/opt/master/src/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/master/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/opt/master/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/master/venv/lib/python3.6/site-packages/djcelery/management/base.py", line 107, in run_from_argv
    return super(CeleryCommand, self).run_from_argv(argv)
  File "/opt/master/venv/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/master/venv/lib/python3.6/site-packages/djcelery/management/base.py", line 100, in execute
    super(CeleryCommand, self).execute(*args, **options)
  File "/opt/master/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/opt/master/venv/lib/python3.6/site-packages/djcelery/management/commands/celeryd.py", line 25, in handle
    worker.run(**options)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/opt/master/venv/lib/python3.6/site-packages/celery/worker/__init__.py", line 100, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
  File "/opt/master/venv/lib/python3.6/site-packages/celery/worker/__init__.py", line 141, in setup_instance
    self.blueprint.apply(self, **kwargs)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 221, in apply
    step.include(parent)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 388, in include
    inc, ret = self._should_include(parent)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 344, in _should_include
    return True, self.create(parent)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/worker/components.py", line 245, in create
    prefetch_multiplier=w.prefetch_multiplier,
  File "/opt/master/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 340, in instantiate
    return instantiate(name, *args, **kwargs)
  File "/opt/master/venv/lib/python3.6/site-packages/celery/utils/imports.py", line 53, in instantiate
    return symbol_by_name(name)(*args, **kwargs)
  File "/opt/master/venv/lib/python3.6/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/master/venv/lib/python3.6/site-packages/celery/worker/consumer/__init__.py", line 3, in <module>
    from .consumer import Consumer
  File "/opt/master/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 19, in <module>
    from kombu.utils.compat import _detect_environment
ImportError: cannot import name '_detect_environment'

Process finished with exit code 1

Can't find what could cause this in the diff

[auvipy]

@thedrow

[spjwebster]

Can confirm we're seeing this too in our environment.

[vandorjw]

It is either pypi, or the build system pushing to pypi that is broken. The wheel and the gzip do not contain the same code. See

https://pypi.python.org/packages/c3/e3/d19b6c775f15bce93befebd5f32358a3ee897578732c9e13400b8f43fae7/celery-3.1.26-py2.py3-none-any.whl#md5=3dcb20b875fe0348390127d04f901896

and

https://pypi.python.org/packages/89/86/00f257023b3919d458b6b1df0a29c0559284f55bf885d7593ec363a413ad/celery-3.1.26.tar.gz#md5=71b31213e981797a895135baae66c61f

The 'wheel' version contains code belonging to celery 4 and is broken.
The 'tar.gz` version is correct.

[vandorjw]

@thedrow The new release didn't work. When I extract the contents of the wheel package, I still find the following files:

./celery
./celery-3.1.26.post1.dist-info
...
<SNIP>
...
./celery/worker/autoreload.py
./celery/worker/autoscale.py
./celery/worker/components.py
./celery/worker/consumer
./celery/worker/consumer.py
./celery/worker/control.py
./celery/worker/heartbeat.py
./celery/worker/__init__.py
./celery/worker/job.py
./celery/worker/loops.py
./celery/worker/pidbox.py
./celery/worker/request.py
./celery/worker/state.py
./celery/worker/strategy.py
./celery/worker/worker.py
./celery/worker/consumer/agent.py
./celery/worker/consumer/connection.py
./celery/worker/consumer/consumer.py
./celery/worker/consumer/control.py
./celery/worker/consumer/events.py
./celery/worker/consumer/gossip.py
./celery/worker/consumer/heart.py
./celery/worker/consumer/__init__.py
./celery/worker/consumer/mingle.py
./celery/worker/consumer/tasks.py
./celery-3.1.26.post1.dist-info/DESCRIPTION.rst
./celery-3.1.26.post1.dist-info/entry_points.txt
./celery-3.1.26.post1.dist-info/METADATA
./celery-3.1.26.post1.dist-info/metadata.json
./celery-3.1.26.post1.dist-info/RECORD
./celery-3.1.26.post1.dist-info/top_level.txt
./celery-3.1.26.post1.dist-info/WHEEL

./celery/worker/consumer.py and ./celery/worker/consumer/consumer.py both exist

https://pypi.python.org/pypi/celery/3.1.26-1

[vandorjw]

https://pypi.python.org/pypi/celery/4.2.0rc1 <-- this wheel is also has code which does not belong

[ch0k0bn]

.post2 version seems to fix the problem. Thx for your quick fix.

[vandorjw]

I can also confirm that post2 fixed the issue.
Thanks for looking into this!

[georgepsarakis]

@thedrow if you can share the commands you are using for the release process, perhaps I can check if I can reproduce it.

[vandorjw]

If you visit this page, download the Python Wheel and check what is inside this wheel, you'll find under celery/worker/ a file called: consumer.py. That file does not belong in Celery 4.2

https://github.com/celery/celery/tree/v4.2.0rc1/celery/worker

The MD5 digest of the wheel is: f57dc80bf33a1150f6672cf9a73ecede