when trying to create a folder gracefully fall back when it already exists #1083
Conversation
|
See #1063 for the CI failures. |
I believe this is likely to be safe in general, but have only tested with 2.7 on on OS X. Without this change, I am seeing error: ``` $ git clone git@github.com:pypa/setuptools Cloning into 'setuptools'... remote: Counting objects: 24321, done. remote: Compressing objects: 100% (23/23), done. remote: Total 24321 (delta 11), reused 20 (delta 7), pack-reused 24291 Receiving objects: 100% (24321/24321), 28.77 MiB | 7.38 MiB/s, done. Resolving deltas: 100% (8669/8669), done. $ cd setuptools $ git rev-parse HEAD 995d309 $ python bootstrap.py adding minimal entry_points Traceback (most recent call last): File "bootstrap.py", line 63, in <module> __name__ == '__main__' and main() File "bootstrap.py", line 59, in main ensure_egg_info() File "bootstrap.py", line 37, in ensure_egg_info build_egg_info() File "bootstrap.py", line 47, in build_egg_info ep.write(minimal_egg_info) TypeError: must be unicode, not str ```
|
@benoit-pierre Thank you for the quick pointer. With that change CI passes. |
| @@ -12,7 +12,7 @@ | |||
| import io | |||
|
|
|||
|
|
|||
| minimal_egg_info = textwrap.dedent(""" | |||
| minimal_egg_info = textwrap.dedent(u""" | |||
There was a problem hiding this comment.
Use from __future__ import unicode_literals if unicode literals are required... which I don't think they are here. Scrap this change.
There was a problem hiding this comment.
The change from ##1063 is only being pulled in here to make CI pass.
| @@ -2958,8 +2959,11 @@ def _find_adapter(registry, ob): | |||
| def ensure_directory(path): | |||
| """Ensure that the parent directory of `path` exists""" | |||
| dirname = os.path.dirname(path) | |||
| if not os.path.isdir(dirname): | |||
| try: | |||
There was a problem hiding this comment.
This is good, and I was going to accept it straight-up, but given that this technique is superseded in Python 3.2 with the exists_ok parameter. Therefore, I'd prefer to use the py27compat or py31compat module to house this legacy approach. Can you put it in py31compat (as a makedirs function that accepts exists_ok kwarg)?
There was a problem hiding this comment.
That is why the branch is called "makedirs_exist_ok" 😉 But since the code needs to work in Python < 3.2 I didn't use the new parameter.
|
Ugh. I now I note that since this is in pkg_resources, and pkg_resources can't import setuptools, we need a new py31compat module in pkg_resources. I'll just make the changes. |
Great, thank you for getting this fixed. Can you please comment here with a reference to the applied change once that is done. That will allow future users to trace as of which version the fix from this PR will be available. |
|
Should be fixed in v36.1.0, going out now (assuming tests pass). |
|
I took a look at the patch you committed but it seems to be incorrect: see 925dd35#commitcomment-23081646 |
|
Should be fixed in 2cd7f22. Release imminent. |
|
I'm trying to figure out now what's going on with https://travis-ci.org/pypa/setuptools/jobs/253323051. |
|
Ugh. So exists_ok had some extra complication with it before Python 3.3.6. From the docs:
Seems it also affects Python 3.2 prior to 3.2.5 and Python 3.4 prior to 3.4.1. |
|
v36.1.1 going out now. |
|
Great, thank you. |
When installing multiple packages into the same location in parallel each packages installation step tries to create the same directories (e.g. the
binfolder). Currently theisdirandmakedirscall are performed sequentially which makes this logic vulnerable to a race condition. Theisdircall will returnFalsebut the followingmakedirscall will fail since the directory has been created in the meantime by another package being installed in parallel.This patch changes the logic to use the Pythonic approach to try to create the directory and in case it already exists ignore the exception.