Add new io.FileIO using the native Windows API

[vstinner]

BPO	12939
Nosy	@gvanrossum, @amauryfa, @pitrou, @vstinner, @ezio-melotti, @bitdancer, @socketpair, @serhiy-storchaka, @zooba, @applio
Files	winfileio.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-28.01:41:16.135>
created_at = <Date 2011-09-08.23:19:00.700>
labels = ['expert-unicode', 'type-feature', 'expert-IO', 'OS-windows']
title = 'Add new io.FileIO using the native Windows API'
updated_at = <Date 2017-06-28.17:28:38.827>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-06-28.17:28:38.827>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-28.01:41:16.135>
closer = 'vstinner'
components = ['Unicode', 'Windows', 'IO']
creation = <Date 2011-09-08.23:19:00.700>
creator = 'vstinner'
dependencies = []
files = ['28737']
hgrepos = []
issue_num = 12939
keywords = ['patch']
message_count = 40.0
messages = ['143741', '143742', '146769', '146788', '146789', '146792', '146795', '146796', '146797', '146837', '178900', '178923', '178943', '178969', '179181', '179229', '179232', '179269', '179809', '179811', '179943', '179944', '179973', '179975', '180014', '180043', '180102', '199413', '222661', '222663', '252160', '252235', '252236', '252241', '252247', '252261', '297130', '297178', '297183', '297201']
nosy_count = 15.0
nosy_names = ['gvanrossum', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'ezio.melotti', 'rpetrov', 'r.david.murray', 'santoso.wijaya', 'BreamoreBoy', 'socketpair', 'sbt', 'piotr.dobrogost', 'serhiy.storchaka', 'steve.dower', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12939'
versions = ['Python 3.6']

[vstinner]

On Windows, Python uses the POSIX API (file descriptors), instead of the native API (file handles). Some features cannot be used using the POSIX API, like setting security attributes. It would be nice to have a io.FileIO using Windows file handlers to get access to all Windows features. It would help feature bpo-12105 to implement "O_CLOEXEC" flag using the lpSecurityAttributes argument.

We can maybe try with a prototype written in Python. Using _pyio.RawIOBase, only readinto(), write(), seek() and truncate() have to be implemented. But it is better to implement also close() :-)

[vstinner]

See also issue bpo-1602: a prototype of a console object has been proposed to use the native Windows console API, instead of the POSIX API (read from fd 0, write to fd 1 or 2). The prototype is implemented in Python using ctypes.

[pitrou]

Instead of rewriting your own RawIO implementation, why not use _open_osfhandle?
This should be simple now with the "opener" argument.
http://msdn.microsoft.com/en-us/library/bdts1c9x.aspx

[socketpair]

why not use _open_osfhandle?

Because it is wrapper for other CRT functions for Windows, like close(). In other words it is an emulation. I think Python should not create wrapper around wrapper around wrapper...

For example, in Python3, open() implemented using open() and not using fopen(). Why we should use another wrapper on Windows platform?

[pitrou]

why not use _open_osfhandle?

Because it is wrapper for other CRT functions for Windows, like
close(). In other words it is an emulation. I think Python should not
create wrapper around wrapper around wrapper...

Why do you think it makes a difference?

[amauryfa]

An implementation of RawIO with the win32 API can be useful (and I'd be interested to compare the performance)
But maybe not for all usages: some Python interfaces are defined in terms of file descriptors, imp.load_module(), and PyTokenizer_FindEncoding for example.

[socketpair]

Why do you think it makes a difference?
Because adding one more dependency on unneeded libraries add the pain. Also it limit us on very restricted API of that wrapper. Windows native API is stable. So it's OK to rely on it's documented imlementation.

Suppose, we receive file-descriptor from _open_osfhandle.... For example it is a socket. It still unusable for stdin. Many standard functions does not work with such handle. The list of available functions : http://msdn.microsoft.com/en-us/library/kdfaxaay.aspx . As we see it is very narrow and function names are not POSIX-compatible (_chsize vs ftruncate). Documentation is very poor. For example, _close() is documented only here: http://msdn.microsoft.com/en-US/library/40bbyw78(v=VS.80).aspx .

[socketpair]

The good example of implemetation is QT. I think we can get some things from this library. (Like using "\\?\" for long paths (or always, as in QT)).

[pitrou]

> Why do you think it makes a difference?
Because adding one more dependency on unneeded libraries add the pain.

MSVCRT is unneeded?? What are you talking about?

Also it limit us on very restricted API of that wrapper. Windows
native API is stable. So it's OK to rely on it's documented
imlementation.

Please provide a patch to demonstrate the claimed improvement.

Like using "\\?\" for long paths

That's completely unrelated to this issue.

[vstinner]

Instead of rewriting your own RawIO implementation, why not use
_open_osfhandle?

I don't know yet what is the best approach. One important point is to keep the
HANDLE, to be able to manipulate the open file using the Windows API (e.g. call
WriteFile instead of write).

I propose a RawIO to call directly the Windows API: file.write() would call
directly WriteFile() instead of the POSIX wrapper (write()).

We may use it to implement new features like async I/O on Windows (e.g.
WriteFileEx).

Modules/_multiprocessing/win32_functions.c exposes already some low-level
Windows functions like WriteFile(). We may reuse the RawIO to offer an object
oriented API for the Windows multiprocessing pipes. Such RawIO would have
extra methods, maybe a file.WriteFile() method to give access to extra options
like "overlapped".

[vstinner]

I'm not really motivated to work on a Windows-specific issue. The issue has no patch, even no proof-of-concept implemented in Python. Can I close the issue, or is someone interested to work on this topic?

Python 3.3 adds a opener argument to open(). It may be enough to workaround this issue?

[socketpair]

Yes, re-writing windows IO to direct API, without intemediate layer is still needed.

Please don't close bug. Maybe someone will implement this.