-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for opening files with FILE_SHARE_DELETE on Windows #59449
Comments
On Unix, files (unless specifically locked) can be renamed and deleted while file descriptors for the file remain open. The file descriptors remain valid even after deletion of the file. On Windows this is not possible for files opened using io.open() or os.open(). However, by using the FILE_SHARE_DELETE flag in CreateFile() one can get Unix-like behaviour. Unfortunately, FILE_SHARE_DELETE is only available through the Win32 API, not through the CRT. Also, Issue bpo-14243 concerns the fact that on Windows temporary files cannot be reopened unless one uses the FILE_SHARE_DELETE flag. One can only reopen a file by using a share mode that is at least as permissive as the share mode for all the currently open handles. The attached patch adds a module "share" (bad name?) with functions share.open() and share.os_open() which act as substitutes for io.open() and os.open(). These by default use FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE as the share mode. (io.open() and os.open() use FILE_SHARE_READ | FILE_SHARE_WRITE instead.) To run the full regression test suite with builtins.open(), io.open() and os.open() monkey patched to use these replacements you can do
Nothing seems to break. |
Having the same semantics on both Unix and Windows with regard to validity of file handle after a file was deleted would be a very nice to have. How could we progress this? I'm adding Martin and Antoine to cc list. |
Actually, it is not quite the same semantics as Unix. After you delete the the file you cannot create a file of the same name or delete the directory which contains it until the handle has been closed. However, one can work around that by moving the file somewhere else (like the root directory) before deleting it. |
I don't understand whether you are proposing to include the patch into Python as-is; if so, I'm -1 on it for two formal reasons: a) the standard library shouldn't monkey patch itself, and b) OS interfacing should be implemented in C. That said, having maximum sharing when opnening files sounds fine to me. |
I think Richard is well aware of the constraints you specify and current patch was meant as a proof of concept; to show that all tests pass with such a change. Of course that's only my belief and we shall see what Richard has to say.
Good to hear. However I started to wonder if we are ready for all consequences of this. For example taking into account what Richard noted in http://bugs.python.org/issue14243, specifically:
forces programs which would like to open a file being opened at the same time by Python code (by means of built-in open() or os.open() with default arguments) to either use O_TEMPORARY when using msvcrt or to go low level and use CreateFile() Win32 API function with FILE_SHARE_DELETE flag. Are we ok with it? |
Am 14.03.13 03:31, schrieb Piotr Dobrogost:
That's why I was asking for an actual patch. The proposed change may An alternative approach could be that os.open stops creating CRT |
On 14/03/2013 1:00pm, Martin v. Löwis wrote:
The patch *does* create CRT fds from win32 handles by using One other issue is that I do not know of a way to determine the current In the end I am not sure it is worth the hassle. (But maybe it would be
http://bugs.python.org/issue12939 has C implementations of _io.WinFileIO |
As much as I like the idea of using OS handles everywhere, the compatibility issues are probably too significant, and you can't mix CRT methods with OS methods because the CRT does its own buffering. Of course, you can open a file with the Win32 API and immediately call open_osfhandle() safely enough. It seems this should be solvable with the opener argument of open(). Was that discussed and I missed it? Would the ability to write "open(f, opener=with_same_sharing_as(f))" be okay/useful/relevant for all platforms? |
"opener" was new in 3.3, so I suspect it didn't come up because *we're* still not entirely accustomed to having it available yet :) I agree providing a suitable opener could be a good way to go. |
Python 3's io uses the CRT's low I/O (ioinfo struct) in binary mode. It appears that the buffers pipech, pipech2, and dbcsbuffer are only used in text mode. So it should be generally OK to call get_osfhandle and use Win32 directly, if that's ever required. Of course, as this patch demonstrates, an io opener (3.3+) can call open_osfhandle immediately after CreateFile, and nothing else changes. This could be added to io / _pyio in 3.5. Initially it could simply be a pure-Python function that uses _winapi and msvcrt. 3.6 could get a C implementation in _iomodule.c. |
Deleting a file in Windows 10 has been updated to try a POSIX-style delete. For a POSIX delete, the filesystem unlinks the file even it's open, whereas a classic delete only unlinks a 'deleted' file when it's closed. The filesystem has to support it. NTFS does, which is the most important because the system drive is NTFS. This makes supporting FILE_SHARE_DELETE a more attractive option. That said, there are sill significant limits. Memory-mapped files can't be deleted (renamed, yes, but not removed). Also, most programs don't share delete access on their opens, which means they can't open a file that's currently open with delete access (e.g. NamedTemporaryFile), and their open files can't be deleted. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: