Race condition in storage.py for NestedDirectoryStore writes

[jmswaney]

I've encountered a FileExistsError when writing in parallel to a NestedDirectoryStore that says:

FileExistsError: [Errno 17] File exists: '/media/jswaney/Drive/Justin/coregistration/whole_brain_tde/fixed/zarr/4/10'

The error points to the os.makedirs(dir_path) line in __setitem__ for the DirectoryStore class:

dir_path, file_name = os.path.split(file_path)
if os.path.isfile(dir_path):
    raise KeyError(key)
if not os.path.exists(dir_path):
    try:
        os.makedirs(dir_path)  # Error raised here
    except Exception:
        raise KeyError(key)

I was doing the chunk compression with 12 workers, so I think one of them created the subdirectory while another had already checked that it didn't exist. When it got to os.makedirs, it gave the expected error. A quick fix would just be to catch this error, but a better fix might be to just make all the directories for a NestedDirectoryStore ahead of time.

[alimanfoo]

Thanks Justin. On Python 3 I think we could just do os.makedirs(..., exist_ok=True). Not available in Python 2 unfortunately. There's also another race condition issue in DirectoryStore, would be good to get both resolved, could be addressed in same PR: #263

…

On Fri, 6 Jul 2018, 06:02 Justin Swaney, ***@***.***> wrote: I've encountered a FileExistsError when writing in parallel to a NestedDirectoryStore that says: FileExistsError: [Errno 17] File exists: '/media/jswaney/Drive/Justin/coregistration/whole_brain_tde/fixed/zarr/4/10' The error points to the os.makedirs(dir_path) line in __setitem__ for the DirectoryStore class: dir_path, file_name = os.path.split(file_path)if os.path.isfile(dir_path): raise KeyError(key)if not os.path.exists(dir_path): try: os.makedirs(dir_path) # Error raised here except Exception: raise KeyError(key) I was doing the chunk compression with 12 workers, so I think one of them created the subdirectory while another had already checked that it didn't exist. When it got to os.makedirs, it gave the expected error. A quick fix would just be to catch this error, but a better fix might be to just make all the directories for a NestedDirectoryStore ahead of time. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#272>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QoBvx_MpUqzyNOrahocByEx2mK8aks5uDu9wgaJpZM4VE4sR> .

[jmswaney]

It looks like there are a couple options for the Python 2 vs 3 problem listed here. They mention the exist_ok=True option in the context of the pathlib module which has been backported to Python 2 in the pathlib2 module, but it might be weird to use this OO-style path handling just for this though. Another option is to catch the general OSError and just throw an error if the directory still doesn't exist.

[jakirkham]

It sounds like we are already interested in using pathlib. ( #261 ) Maybe using the pathlib-based solution will work for us and we can add pathlib2 as a requirement.

[jakirkham]

Note: Work on this is currently being pursued in PR ( #318 ).