Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix _get_virtualenv_hash function #2883

Merged
merged 2 commits into from
Nov 3, 2018
Merged

Fix _get_virtualenv_hash function #2883

merged 2 commits into from
Nov 3, 2018

Conversation

Jamim
Copy link
Contributor

@Jamim Jamim commented Sep 23, 2018

The issue

This PR closes #3151.

Case-sensitive filesystems are handled like case-insensitive and vice versa.
This bug was introduced in #2513 and it affects only master branch and does not affect previous release 2018.10.9 and 2018.10.13 releases.

The fix

This PR adds not before the related condition.

Additional changes
  • add PyCharm's config directory to .gitignore
CC
The checklist
  • Associated issue
  • A news fragment in the news/ directory to describe this fix with the extension .bugfix.

@Jamim
Copy link
Contributor Author

Jamim commented Sep 24, 2018

Hi @uranusjr,
I believe you can help me fix _get_virtualenv_hash for Windows as well.
Thank you!

@Jamim
Copy link
Contributor Author

Jamim commented Oct 4, 2018

@kennethreitz, @uranusjr, @techalchemy
Does anyone care that there is a bug which should be fixed?

@techalchemy
Copy link
Member

Hello @Jamim -- of course we care about the library we maintain, but I would suggest that the phrasing is probably not the most effective way to have a positive interaction.

I appreciate your contribution, however, the pr template is there to suggest that you should open and report an issue if there is not yet one on the topic, which is how we typically triage problems. This helps to avoid contributing a fix before it has been determined that it is necessary. This way, we can assess that the fix indeed is desired before you put effort into attempting it.

Before we merge any PR related to this, I would need to see a failing test and an issue explaining the problem. Thanks for your efforts

@Jamim
Copy link
Contributor Author

Jamim commented Oct 4, 2018

Thank you for the feedback, @techalchemy!

I understand that you have some workflow that you prefer to follow, but meanwhile I believe there is too much effort required for a single-line fix.
I'll create an issue a bit later today, but I strongly recommend you to just look at the patch and realize how redundant this discussion around one little not is.

@techalchemy
Copy link
Member

I still need to know exactly what is broken, preferably via a test case (prevents regression) but if you don't feel like it at least I need to see an example of what you're saying doesn't work properly, because it's hard to decipher that just from reading over the text provided

I.e. what do I need to type in to have a problem?

@Jamim
Copy link
Contributor Author

Jamim commented Oct 4, 2018

I.e. what do I need to type in to have a problem?

I found that since I submitted this PR the problem was "fixed" by #2877, but I believe it doesn't work properly. So, you need to revert one commit to see what exactly goes wrong.

git checkout master
git pull origin master
git checkout -b tmp
git revert 65ccd765b6d0b78e602d4b233ca8e31e1a6b8adb
docker rm pipenv_pipenv-tests_1  # you must delete cached image
docker-compose up

I expect that you'll get something like this:

# docker rm pipenv_pipenv-tests_1; docker-compose up
pipenv_pipenv-tests_1
Creating pipenv_pipenv-tests_1 ... done
Attaching to pipenv_pipenv-tests_1
pipenv-tests_1  | Collecting certifi
pipenv-tests_1  |   Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
pipenv-tests_1  | Installing collected packages: certifi
pipenv-tests_1  | Successfully installed certifi-2018.8.24
pipenv-tests_1  | Path: /root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
pipenv-tests_1  | Installing Pipenv…
pipenv-tests_1  | Collecting setuptools
pipenv-tests_1  |   Downloading https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py2.py3-none-any.whl (569kB)
pipenv-tests_1  | Installing collected packages: setuptools
pipenv-tests_1  | Successfully installed setuptools-40.4.3
pipenv-tests_1  | Collecting setuptools
pipenv-tests_1  |   Using cached https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py2.py3-none-any.whl
pipenv-tests_1  | Installing collected packages: setuptools
pipenv-tests_1  | Successfully installed setuptools-40.4.3
pipenv-tests_1  | Obtaining file:///pipenv
pipenv-tests_1  |   DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
pipenv-tests_1  | Collecting pip>=9.0.1 (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
pipenv-tests_1  | Requirement already up-to-date: certifi in /usr/local/lib/python2.7/dist-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Requirement already up-to-date: setuptools>=36.2.1 in /root/.local/lib/python2.7/site-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Collecting virtualenv-clone>=0.2.5 (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Using cached https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
pipenv-tests_1  | Collecting virtualenv (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Using cached https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl
pipenv-tests_1  | Requirement already up-to-date: enum34 in /usr/lib/python2.7/dist-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Collecting typing (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Downloading https://files.pythonhosted.org/packages/cc/3e/29f92b7aeda5b078c86d14f550bf85cff809042e3429ace7af6193c3bc9f/typing-3.6.6-py2-none-any.whl
pipenv-tests_1  | Installing collected packages: pip, virtualenv-clone, virtualenv, typing, pipenv
pipenv-tests_1  |   Found existing installation: pip 9.0.1
pipenv-tests_1  |     Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
pipenv-tests_1  |   Found existing installation: pipenv 2018.7.1.dev0
pipenv-tests_1  |     Can't uninstall 'pipenv'. No files were found to uninstall.
pipenv-tests_1  |   Running setup.py develop for pipenv
pipenv-tests_1  | Successfully installed pip-18.0 pipenv typing-3.6.6 virtualenv-16.0.0 virtualenv-clone-0.3.0
pipenv-tests_1  | Obtaining file:///pipenv
pipenv-tests_1  |   DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
pipenv-tests_1  | Collecting pip>=9.0.1 (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Using cached https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl
pipenv-tests_1  | Collecting certifi (from pipenv==2018.7.1.dev0)
pipenv-tests_1  |   Using cached https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl
pipenv-tests_1  | Requirement already up-to-date: setuptools>=36.2.1 in /root/.local/lib/python3.6/site-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Requirement already up-to-date: virtualenv-clone>=0.2.5 in /usr/local/lib/python3.6/dist-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Requirement already up-to-date: virtualenv in /usr/local/lib/python3.6/dist-packages (from pipenv==2018.7.1.dev0)
pipenv-tests_1  | Installing collected packages: pip, certifi, pipenv
pipenv-tests_1  |   Found existing installation: pip 9.0.1
pipenv-tests_1  |     Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
pipenv-tests_1  |   Found existing installation: certifi 2018.4.16
pipenv-tests_1  |     Uninstalling certifi-2018.4.16:
pipenv-tests_1  |       Successfully uninstalled certifi-2018.4.16
pipenv-tests_1  |   Found existing installation: pipenv 2018.7.1.dev0
pipenv-tests_1  |     Can't uninstall 'pipenv'. No files were found to uninstall.
pipenv-tests_1  |   Running setup.py develop for pipenv
pipenv-tests_1  | Successfully installed certifi-2018.8.24 pip-18.0 pipenv-2018.5.18
pipenv-tests_1  | Traceback (most recent call last):
pipenv-tests_1  |   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
pipenv-tests_1  |     "__main__", mod_spec)
pipenv-tests_1  |   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
pipenv-tests_1  |     exec(code, run_globals)
pipenv-tests_1  |   File "/pipenv/pipenv/__main__.py", line 4, in <module>
pipenv-tests_1  |     cli()
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 722, in __call__
pipenv-tests_1  |     return self.main(*args, **kwargs)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 697, in main
pipenv-tests_1  |     rv = self.invoke(ctx)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 1066, in invoke
pipenv-tests_1  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 895, in invoke
pipenv-tests_1  |     return ctx.invoke(self.callback, **ctx.params)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 535, in invoke
pipenv-tests_1  |     return callback(*args, **kwargs)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/decorators.py", line 64, in new_func
pipenv-tests_1  |     return ctx.invoke(f, obj, *args[1:], **kwargs)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/core.py", line 535, in invoke
pipenv-tests_1  |     return callback(*args, **kwargs)
pipenv-tests_1  |   File "/pipenv/pipenv/vendor/click/decorators.py", line 17, in new_func
pipenv-tests_1  |     return f(get_current_context(), *args, **kwargs)
pipenv-tests_1  |   File "/pipenv/pipenv/cli/command.py", line 249, in install
pipenv-tests_1  |     editable_packages=state.installstate.editables,
pipenv-tests_1  |   File "/pipenv/pipenv/core.py", line 1687, in do_install
pipenv-tests_1  |     pypi_mirror=pypi_mirror,
pipenv-tests_1  |   File "/pipenv/pipenv/core.py", line 594, in ensure_project
pipenv-tests_1  |     validate=validate, skip_requirements=skip_requirements, system=system
pipenv-tests_1  |   File "/pipenv/pipenv/core.py", line 319, in ensure_pipfile
pipenv-tests_1  |     if validate and project.virtualenv_exists and not PIPENV_SKIP_VALIDATION:
pipenv-tests_1  |   File "/pipenv/pipenv/project.py", line 259, in virtualenv_exists
pipenv-tests_1  |     if self.pipfile_exists and os.path.exists(self.virtualenv_location):
pipenv-tests_1  |   File "/pipenv/pipenv/project.py", line 370, in virtualenv_location
pipenv-tests_1  |     self._virtualenv_location = self.get_location_for_virtualenv()
pipenv-tests_1  |   File "/pipenv/pipenv/project.py", line 272, in get_location_for_virtualenv
pipenv-tests_1  |     name = self.virtualenv_name
pipenv-tests_1  |   File "/pipenv/pipenv/project.py", line 356, in virtualenv_name
pipenv-tests_1  |     sanitized, encoded_hash = self._get_virtualenv_hash(self.name)
pipenv-tests_1  |   File "/pipenv/pipenv/project.py", line 340, in _get_virtualenv_hash
pipenv-tests_1  |     for path in get_workon_home().iterdir():
pipenv-tests_1  |   File "/usr/lib/python3.6/pathlib.py", line 1079, in iterdir
pipenv-tests_1  |     for name in self._accessor.listdir(self):
pipenv-tests_1  |   File "/usr/lib/python3.6/pathlib.py", line 387, in wrapped
pipenv-tests_1  |     return strfunc(str(pathobj), *args)
pipenv-tests_1  | FileNotFoundError: [Errno 2] No such file or directory: '/root/.local/share/virtualenvs'
pipenv_pipenv-tests_1 exited with code 1

@Jamim
Copy link
Contributor Author

Jamim commented Oct 4, 2018

@techalchemy By the way, you asked me for some suitable test, but the main goal of this PR (and this one as well) was to make the tests works in local environment.
I found it ironic a bit.

@Jamim
Copy link
Contributor Author

Jamim commented Oct 5, 2018

Hi @techalchemy,

I created a corresponding issue and I need you to confirm that I should not create any news fragment, since the bug is in master branch only and it's not presented in the release.
And please feel free to ask me anything to clarify the issue.

Thank you!

@Jamim
Copy link
Contributor Author

Jamim commented Oct 9, 2018

ping @uranusjr

@Jamim
Copy link
Contributor Author

Jamim commented Oct 9, 2018

ping @techalchemy

@techalchemy
Copy link
Member

will look this over again before i cut the next release, sorry about that

@techalchemy
Copy link
Member

Based on the commit you referenced it seems like we just need to call normalize() on the path we create to avoid the issue, no?

And we should be normalizing it before checking as well?

@Jamim
Copy link
Contributor Author

Jamim commented Oct 9, 2018

Based on the commit you referenced it seems like we just need to call normalize() on the path we create to avoid the issue, no?

And we should be normalizing it before checking as well?

To be honest, I have no idea. Sorry, I didn't investigate this issue deeper.

By the way, did you read the Solution section of the related issue #2919?

The root cause of the issue is that os.name != "nt" condition was mistakenly converted to fnmatch.fnmatch('A', 'a') when it should have been converted to not fnmatch.fnmatch('A', 'a').

That's it. I just know that the condition was inverted by mistake and it causes the problem for me.

@techalchemy
Copy link
Member

That's it. I just know that the condition was inverted by mistake and it causes the problem for me.

Look, you opened a PR with code changes and you keep telling me a solution, but honestly it is not possible for me to assess a solution until I know the problem.

You showed me a commit you say I need to revert in order to cause the problem, but you also say this is still a problem. The commit you showed me just looks for the virtualenv directory but doesn't do any case normalization.

Ultimately, I really don't know how to ask for this differently, I need you to show me what is wrong by showing me an example of something that is currently behaving incorrectly. Telling me that someone forgot to put a not somewhere doesn't help me, this is just increasing my cognitive burden when trying to understand the issue.

Here is an example of what I am looking for:

Hi. I am a user of pipenv, and I ran this command:

> some command
...
some output

But as you can see, that directory should really have been found because it exists here:

> ls some_directory

It's going to be hard to move forward on this until I can see something showing me why the current code is wrong.

@Jamim
Copy link
Contributor Author

Jamim commented Oct 11, 2018

Hi there,

I'm ill-fated user of pipenv, and I ran these commands on my Linux machine using cae8bad revision:

mkdir pipenv-tests/{test,TEST} -p
cd pipenv-tests/test
pipenv shell
pipenv --venv
exit
cd ../TEST
pipenv shell
pipenv --venv

Here is what I've got:

~/projects $ mkdir pipenv-tests/{test,TEST} -p
~/projects $ cd pipenv-tests/test
~/projects/pipenv-tests/test $ pipenv shell
Creating a virtualenv for this project…
Pipfile: /home/mim/projects/pipenv-tests/test/Pipfile
Using /usr/bin/python3.7 (3.7.0) to create virtualenv…
⠼Already using interpreter /usr/bin/python3.7
Using base prefix '/usr'
/home/mim/.local/lib/python3.7/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
New python executable in /home/mim/.local/share/virtualenvs/test-Ic8GqeXz/bin/python3.7
Also creating executable in /home/mim/.local/share/virtualenvs/test-Ic8GqeXz/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/mim/.local/share/virtualenvs/test-Ic8GqeXz
Creating a Pipfile for this project…
Launching subshell in virtual environment…
~/projects/pipenv-tests/test $  . /home/mim/.local/share/virtualenvs/test-Ic8GqeXz/bin/activate
(test)~/projects/pipenv-tests/test $ pipenv --venv
/home/mim/.local/share/virtualenvs/test-Ic8GqeXz
(test)~/projects/pipenv-tests/test $ exit
exit
~/projects/pipenv-tests/test $ cd ../TEST
~/projects/pipenv-tests/TEST $ pipenv shell
Creating a Pipfile for this project…
Launching subshell in virtual environment…
~/projects/pipenv-tests/TEST $  . /home/mim/.local/share/virtualenvs/test-Ic8GqeXz/bin/activate
(test)~/projects/pipenv-tests/TEST $ pipenv --venv
/home/mim/.local/share/virtualenvs/test-Ic8GqeXz
(test)~/projects/pipenv-tests/TEST $ 

There should be two different virtual environments for test and TEST, but if some environment is already created, pipenv reuses it.
In my case it's /home/mim/.local/share/virtualenvs/test-Ic8GqeXz which is used for both corresponding directories.

I'm expecting that pipenv will create /home/mim/.local/share/virtualenvs/TEST-TKzGZNiV for TEST, no matter if /home/mim/.local/share/virtualenvs/test-Ic8GqeXz exists or not.

(test)~/projects/pipenv-tests/TEST $ exit
exit
~/projects/pipenv-tests/TEST $ pipenv --rm
Removing virtualenv (/home/mim/.local/share/virtualenvs/test-Ic8GqeXz)…
~/projects/pipenv-tests/TEST $ pipenv shell
Creating a virtualenv for this project…
Pipfile: /home/mim/projects/pipenv-tests/TEST/Pipfile
Using /usr/bin/python3.7m (3.7.0) to create virtualenv…
⠹Running virtualenv with interpreter /usr/bin/python3.7m
Using base prefix '/usr'
/home/mim/.local/lib/python3.7/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
New python executable in /home/mim/.local/share/virtualenvs/TEST-TKzGZNiV/bin/python3.7m
Also creating executable in /home/mim/.local/share/virtualenvs/TEST-TKzGZNiV/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/mim/.local/share/virtualenvs/TEST-TKzGZNiV
Launching subshell in virtual environment…
~/projects/pipenv-tests/TEST $  . /home/mim/.local/share/virtualenvs/TEST-TKzGZNiV/bin/activate
(TEST)~/projects/pipenv-tests/TEST $ 

@Jamim
Copy link
Contributor Author

Jamim commented Oct 11, 2018

@techalchemy Should I explain why does it happen?

Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is correct. Could you modify the news fragment to be more targeted to the user? The fragment is used to format our release notes, and “fix an internal function” doesn’t really mean anything to an end user. Something like “Fix hash comparison logic on case insensitive system so projects with different casing do not reuse the same virtual environment” would be better (you don’t need to use my version, I am terrible at writing sentences like this).

Jamim and others added 2 commits November 2, 2018 20:40
Case-sensitive filesystems were handled
like case-insensitive and vice versa.

Close #3151

This changes also:

 - Add PyCharm's config directory to .gitignore
@techalchemy techalchemy added the PR: awaiting-merge The PR related to this issue has been reviewed and is awaiting merge. label Nov 3, 2018
@techalchemy techalchemy merged commit 8799431 into pypa:master Nov 3, 2018
@Jamim Jamim deleted the fix/get-virtualenv-hash branch May 11, 2019 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: awaiting-merge The PR related to this issue has been reviewed and is awaiting merge.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Collisions of virtual environments
3 participants