New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when making virtualenv directory with parallel pipenv invocations #3257

Closed
swgillespie opened this Issue Nov 20, 2018 · 2 comments

Comments

Projects
None yet
1 participant
@swgillespie
Contributor

swgillespie commented Nov 20, 2018

Issue description

When running multiple pipenv processes simultaneously, they may both attempt to use 'mkdir_p' at the same time, which can cause one of the pipenv processes to crash when it tries to create a directory that already exists. This is a classic TOCTOU bug that has bitten many a piece of software.

I personally hit this in my continuous integration setup where I invoke pipenv multiple times, potentially in parallel, and see crashes such as this:

[ tes/examples/python/smoke-test ] sample: /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/examples/python/smoke-test
[ etes/examples/python/guestbook ] sample: /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/examples/python/guestbook
[ tes/examples/python/smoke-test ] pulumi: /home/travis/.pulumi/bin/pulumi
[ etes/examples/python/guestbook ] pulumi: /home/travis/.pulumi/bin/pulumi
[ tes/examples/python/smoke-test ] **** Invoke '/home/travis/.local/bin/pipenv --python 3' in '/tmp/p-it-travis-job-smoke-test-aced7882-102623154'
[ etes/examples/python/guestbook ] **** Invoke '/home/travis/.local/bin/pipenv --python 3' in '/tmp/p-it-travis-job-guestbook-63f89532-423054697'
[ etes/examples/python/guestbook ] Invoke '/home/travis/.local/bin/pipenv --python 3' failed: exit status 1
[ etes/examples/python/guestbook ] Traceback (most recent call last):
[ etes/examples/python/guestbook ]   File "/home/travis/.local/bin/pipenv", line 11, in <module>
[ etes/examples/python/guestbook ]     sys.exit(cli())
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
[ etes/examples/python/guestbook ]     return self.main(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
[ etes/examples/python/guestbook ]     rv = self.invoke(ctx)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
[ etes/examples/python/guestbook ]     return Command.invoke(self, ctx)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
[ etes/examples/python/guestbook ]     return ctx.invoke(self.callback, **ctx.params)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
[ etes/examples/python/guestbook ]     return callback(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
[ etes/examples/python/guestbook ]     return ctx.invoke(f, obj, *args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
[ etes/examples/python/guestbook ]     return callback(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
[ etes/examples/python/guestbook ]     return f(get_current_context(), *args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/cli/command.py", line 203, in cli
[ etes/examples/python/guestbook ]     clear=state.clear,
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/core.py", line 565, in ensure_project
[ etes/examples/python/guestbook ]     pypi_mirror=pypi_mirror,
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/core.py", line 483, in ensure_virtualenv
[ etes/examples/python/guestbook ]     if not project.virtualenv_exists:
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 259, in virtualenv_exists
[ etes/examples/python/guestbook ]     if self.pipfile_exists and os.path.exists(self.virtualenv_location):
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 398, in virtualenv_location
[ etes/examples/python/guestbook ]     self._virtualenv_location = self.get_location_for_virtualenv()
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 272, in get_location_for_virtualenv
[ etes/examples/python/guestbook ]     name = self.virtualenv_name
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 384, in virtualenv_name
[ etes/examples/python/guestbook ]     sanitized, encoded_hash = self._get_virtualenv_hash(self.name)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 363, in _get_virtualenv_hash
[ etes/examples/python/guestbook ]     or get_workon_home().joinpath(venv_name).exists()
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/utils.py", line 1254, in get_workon_home
[ etes/examples/python/guestbook ]     mkdir_p(str(expanded_path))
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/utils.py", line 571, in mkdir_p
[ etes/examples/python/guestbook ]     os.mkdir(newdir)
[ etes/examples/python/guestbook ] OSError: [Errno 17] File exists: '/home/travis/.local/share/virtualenvs'

The root cause is that two pipenv processes are racing to create the new directory subtree rooted at /home/travis/.local/share/virtualenvs, one of them wins, and the other crashes.

Expected result

I'd except both pipenv processes to be successful.

Actual result

The pipenv that loses the race to create the directory crashes.

Steps to replicate

Launch multiple instances of pipenv --python 3 on a machine that previously has not used pipenv to create any virtualenvs. If you do this enough times, you'll hit the race and crash.

@duplicate-issues

This comment has been minimized.

duplicate-issues bot commented Nov 20, 2018

Hey @swgillespie,

We did a quick check and this issue looks very darn similar to

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

@swgillespie swgillespie referenced this issue Nov 20, 2018

Merged

Fix a TOCTOU issue in mkdir_p #3258

2 of 2 tasks complete
@swgillespie

This comment has been minimized.

Contributor

swgillespie commented Nov 20, 2018

Fixed by #3258.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment