You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.
I was doing the chunk compression with 12 workers, so I think one of them created the subdirectory while another had already checked that it didn't exist. When it got to os.makedirs, it gave the expected error. A quick fix would just be to catch this error, but a better fix might be to just make all the directories for a NestedDirectoryStore ahead of time.
The text was updated successfully, but these errors were encountered:
Thanks Justin. On Python 3 I think we could just do os.makedirs(...,
exist_ok=True). Not available in Python 2 unfortunately.
There's also another race condition issue in DirectoryStore, would be good
to get both resolved, could be addressed in same PR:
On Fri, 6 Jul 2018, 06:02 Justin Swaney, ***@***.***> wrote:
I've encountered a FileExistsError when writing in parallel to a
NestedDirectoryStore that says:
FileExistsError: [Errno 17] File exists:
The error points to the os.makedirs(dir_path) line in __setitem__ for the
dir_path, file_name = os.path.split(file_path)if os.path.isfile(dir_path):
raise KeyError(key)if not os.path.exists(dir_path):
os.makedirs(dir_path) # Error raised here
I was doing the chunk compression with 12 workers, so I think one of them
created the subdirectory while another had already checked that it didn't
exist. When it got to os.makedirs, it gave the expected error. A quick
fix would just be to catch this error, but a better fix might be to just
make all the directories for a NestedDirectoryStore ahead of time.
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#272>, or mute the thread
It looks like there are a couple options for the Python 2 vs 3 problem listed here. They mention the exist_ok=True option in the context of the pathlib module which has been backported to Python 2 in the pathlib2 module, but it might be weird to use this OO-style path handling just for this though. Another option is to catch the general OSError and just throw an error if the directory still doesn't exist.
* #272 fix race condition to make folders in directory store
* #272 remove FileNotFoundError for python 2 support
* Change OSError to KeyError to make tests pass
* Only catch `OSError` if its `errno` equal `EEXIST`
On Python 3, we would only want to catch the race case here by checking
for the `FileExistsError` and suppressing raising for that case. However
there is no `FileExistsError` on Python 2. Only `OSError` exists on
Python 2/3. However that catches some other errors that we might not
want to suppress here. So we check the `errno` of the `OSError` instance
to verify it is caused by an existing file/directory represented by the
`EEXIST` error number. This amounts to catching the `FileExistsError` in
a Python 2/3 friendly way. All other `OSErrors` we raise as before.
* Drop generic `Exception` case in `DirectoryStore`
As we now properly handle the existing file case properly, go ahead and
drop the generic exception case. This should now raise some errors that
were hidden previously.