pipenv install from GitHub repo (editable) not working in Docker #2872

metakermit · 2018-09-20T11:57:04Z

Issue description

When I use pipenv to install Celery directly from the master branch on GitHub inside the official Pipenv Docker repository, the package cannot be imported. This is quite a problem right now, because there is no version of Celery that works in Python 3.7 that's published on PyPI as of writing this. Regardless of Celery, though, this error should be fixed for other packages too.

Expected result

I expected that the Celery package installed that way could be imported.

Actual result

Importing Celery outputs an error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import celery
ModuleNotFoundError: No module named 'celery'

Steps to replicate

I've created a minimal GitHub repository to show how to replicate this issue here. The full and detailed steps to replicate are in the repo's README.md

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.93-linuxkit-aufs',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Jun 6 16:55:56 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

PATH
HOSTNAME
TERM
LC_ALL
LANG
HOME
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG: C.UTF-8

Contents of Pipfile ('/app/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/app/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "7e7ef69da7248742e869378f8421880cf8f0017f96d94d086813baa518a65489"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

The text was updated successfully, but these errors were encountered:

uranusjr · 2018-09-21T08:17:00Z

Hmm, following the README I get “it works”. Perhaps @kennethreitz stealthily fixed this in 7a696ce?

metakermit · 2018-09-21T08:24:01Z

@uranusjr ok, thanks for trying. Yeah, I didn't try the development version, so could very well be that it's fixed in the mean time.

metakermit · 2018-09-21T08:35:40Z

@uranusjr awesome, it does indeed work. I guess I'll close this then. Yeah, don't know what exactly changed, because it seems to be the same version of pipenv. The Python versions are a bit different (3.6.6 vs. 3.6.5 in my first report).

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.15: /usr/bin/python
2.7.15: /usr/bin/python2
3.6.6: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.93-linuxkit-aufs',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Jun 6 16:55:56 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

PATH
HOSTNAME
TERM
LC_ALL
LANG
HOME
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG: C.UTF-8

Contents of Pipfile ('/app/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
celery = {editable = true, ref = "master", git = "https://github.com/celery/celery.git"}

[dev-packages]

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/app/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "9476da2b8fb1fd760cabe4c7a1345c1ced8fb81f7013c3d734a391fd6d6c7b49"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "amqp": {
            "hashes": [
                "sha256:073dd02fdd73041bffc913b767866015147b61f2a9bc104daef172fc1a0066eb",
                "sha256:eed41946890cd43e8dee44a316b85cf6fee5a1a34bb4a562b660a358eb529e1b"
            ],
            "version": "==2.3.2"
        },
        "billiard": {
            "hashes": [
                "sha256:ed65448da5877b5558f19d2f7f11f8355ea76b3e63e1c0a6059f47cfae5f1c84"
            ],
            "version": "==3.5.0.4"
        },
        "celery": {
            "editable": true,
            "git": "https://github.com/celery/celery.git",
            "ref": "ae680f447f14a4175d676aec7a2c1963144a2311"
        },
        "kombu": {
            "hashes": [
                "sha256:86adec6c60f63124e2082ea8481bbe4ebe04fde8ebed32c177c7f0cd2c1c9082",
                "sha256:b274db3a4eacc4789aeb24e1de3e460586db7c4fc8610f7adcc7a3a1709a60af"
            ],
            "version": "==4.2.1"
        },
        "pytz": {
            "hashes": [
                "sha256:a061aa0a9e06881eb8b3b2b43f05b9439d6583c206d0a6c340ff72a7b6669053",
                "sha256:ffb9ef1de172603304d9d2819af6f5ece76f2e85ec10692a524dd876e72bf277"
            ],
            "version": "==2018.5"
        },
        "vine": {
            "hashes": [
                "sha256:52116d59bc45392af9fdd3b75ed98ae48a93e822cee21e5fda249105c59a7a72",
                "sha256:6849544be74ec3638e84d90bc1cf2e1e9224cc10d96cd4383ec3f69e9bce077b"
            ],
            "version": "==1.1.4"
        }
    },
    "develop": {}
}

kennethreitz · 2018-09-21T08:47:06Z

oops :)

uranusjr · 2018-09-21T09:00:28Z

@metakermit That commit changed the base Docker image; that could be the reason.

metakermit · 2018-09-21T10:30:15Z

@uranusjr @kennethreitz Hm… for a slightly more complex repo where I originally spotted this error (where I use docker-compose, mount the current code directory as a volume etc.) the problem still persists – even after I've updated the Dockerfile and clean-built all the images. If you have time, it'd help me out a lot if you could take a look – I'm a bit out of ideas what to do next there.

My theory is that something in the way installing an editable package from GitHub creates a src folder clashes with how multiple images sharing the same base image behave (web, worker & cli in this scenario).

$ docker-compose run web python3 -c "import celery"
Starting hellodjango_db_1    ... done
Starting hellodjango_cache_1 ... done
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'celery'

$ docker-compose up
...
web_1     | [2018-09-21 10:22:42 +0000] [11] [ERROR] Exception in worker process
web_1     | Traceback (most recent call last):
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
web_1     |     worker.init_process()
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 129, in init_process
web_1     |     self.load_wsgi()
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
web_1     |     self.wsgi = self.app.wsgi()
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/base.py", line 67, in wsgi
web_1     |     self.callable = self.load()
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
web_1     |     return self.load_wsgiapp()
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
web_1     |     return util.import_app(self.app_uri)
web_1     |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/util.py", line 350, in import_app
web_1     |     __import__(module)
web_1     |   File "/app/hellodjango/__init__.py", line 5, in <module>
web_1     |     from .celery import app as celery_app
web_1     |   File "/app/hellodjango/celery.py", line 3, in <module>
web_1     |     from celery import Celery
web_1     | ModuleNotFoundError: No module named 'celery'

cli_1     | Traceback (most recent call last):
cli_1     |   File "manage.py", line 15, in <module>
cli_1     |     execute_from_command_line(sys.argv)
cli_1     |   File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
cli_1     |     utility.execute()
cli_1     |   File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 317, in execute
cli_1     |     settings.INSTALLED_APPS
cli_1     |   File "/usr/local/lib/python3.6/dist-packages/django/conf/__init__.py", line 56, in __getattr__
cli_1     |     self._setup(name)
cli_1     |   File "/usr/local/lib/python3.6/dist-packages/django/conf/__init__.py", line 43, in _setup
cli_1     |     self._wrapped = Settings(settings_module)
cli_1     |   File "/usr/local/lib/python3.6/dist-packages/django/conf/__init__.py", line 106, in __init__
cli_1     |     mod = importlib.import_module(self.SETTINGS_MODULE)
cli_1     |   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
cli_1     |     return _bootstrap._gcd_import(name[level:], package, level)
cli_1     |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
cli_1     |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
cli_1     |   File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
cli_1     |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
cli_1     |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
cli_1     |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
cli_1     |   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
cli_1     |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
cli_1     |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
cli_1     |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
cli_1     |   File "/app/hellodjango/__init__.py", line 5, in <module>
cli_1     |     from .celery import app as celery_app
cli_1     |   File "/app/hellodjango/celery.py", line 3, in <module>
cli_1     |     from celery import Celery
cli_1     | ModuleNotFoundError: No module named 'celery'

worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/bin/celery", line 6, in <module>
worker_1  |     from pkg_resources import load_entry_point
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3088, in <module>
worker_1  |     @_call_aside
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3072, in _call_aside
worker_1  |     f(*args, **kwargs)
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
worker_1  |     working_set = WorkingSet._build_master()
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 574, in _build_master
worker_1  |     ws.require(__requires__)
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 892, in require
worker_1  |     needed = self.resolve(parse_requirements(requirements))
worker_1  |   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 778, in resolve
worker_1  |     raise DistributionNotFound(req, requirers)
worker_1  | pkg_resources.DistributionNotFound: The 'celery' distribution was not found and is required by the application
...

I can normally run the app locally (PIPENV_VENV_IN_PROJECT=1 PYTHONUNBUFFERED=1 pipenv run honcho -f Procfile.dev start) and even on Heroku from the same repo.

nkprince007 · 2020-02-17T08:35:49Z

Same problem here.

metakermit mentioned this issue Sep 20, 2018

Python 3.7 syntax error: async is a reserved keyword celery/celery#4849

Closed

metakermit closed this as completed Sep 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipenv install from GitHub repo (editable) not working in Docker #2872

pipenv install from GitHub repo (editable) not working in Docker #2872

metakermit commented Sep 20, 2018

uranusjr commented Sep 21, 2018

metakermit commented Sep 21, 2018

metakermit commented Sep 21, 2018

kennethreitz commented Sep 21, 2018

uranusjr commented Sep 21, 2018

metakermit commented Sep 21, 2018

nkprince007 commented Feb 17, 2020

pipenv install from GitHub repo (editable) not working in Docker #2872

pipenv install from GitHub repo (editable) not working in Docker #2872

Comments

metakermit commented Sep 20, 2018

Issue description

Expected result

Actual result

Steps to replicate

uranusjr commented Sep 21, 2018

metakermit commented Sep 21, 2018

metakermit commented Sep 21, 2018

kennethreitz commented Sep 21, 2018

uranusjr commented Sep 21, 2018

metakermit commented Sep 21, 2018

nkprince007 commented Feb 17, 2020