document the special features (eg: fdclose=False) of the standard streams

[snaphat]

BPO	22673
Nosy	@rbtcollins, @tjguk, @bitdancer, @4kir4, @vadmium, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-10-19.19:00:58.005>
labels = ['type-bug', '3.8', '3.9', 'expert-IO', '3.10', 'docs']
title = 'document the special features (eg: fdclose=False) of the standard streams'
updated_at = <Date 2021-02-23.11:40:13.770>
user = 'https://bugs.python.org/snaphat'

bugs.python.org fields:

activity = <Date 2021-02-23.11:40:13.770>
actor = 'eryksun'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'IO']
creation = <Date 2014-10-19.19:00:58.005>
creator = 'snaphat'
dependencies = []
files = []
hgrepos = []
issue_num = 22673
keywords = []
message_count = 8.0
messages = ['229690', '229694', '229699', '229701', '229703', '229707', '229708', '233093']
nosy_count = 11.0
nosy_names = ['rbcollins', 'tim.golden', 'Arfrever', 'r.david.murray', 'docs@python', 'akira', 'martin.panter', 'zach.ware', 'eryksun', 'steve.dower', 'snaphat']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22673'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[snaphat]

If I execute the following code, the file descriptor for CONOUT$ has a fileno != 1. With CONIN$ the fileno != 0. Similar code in another language such as perl produces the desired results.

sys.stdout.close();
sys.stdout = open("CONOUT$", "w");
sys.stderr.write(str(sys.stdout.fileno()));

I believe it has to do with the fact that stdout is an object in python and in perl or c, you are operating directly on the stream ala freopen() or the equivalent.

[bitdancer]

This is Windows specific, right?

From a quick google is not clear that CONOUT$ *should* point 1, since it supposed to get at the console regardless of whether there has been any redirection. Certainly, relying on it doing so would seem to be problematic. What does PERL return if stdout is redirected, and where does the output go in that case?

The io module does not use the c IO library, unlike the default open in python2 (the io module is the default in python3), so the behavior may also be different in the two cases.

[eryksun]

In Python 2, the closer for sys.stdout is _check_and_flush, which flushes the stream without closing the underlying FILE.

https://hg.python.org/cpython/file/ee879c0ffa11/Python/sysmodule.c#l1377

In Python 3, io.FileIO won't close the underlying file descriptor if closefd is False.

    >>> sys.stdout.buffer.raw.closefd
    False

It isn't a writable attribute. It gets initialized to False by create_stdio (line 1034).

https://hg.python.org/cpython/file/ab2c023a9432/Python/pythonrun.c#l1006

You'll can call os.close.

Python 2:

    >>> sys.stdout.close(); os.close(1); sys.stdout = open('CONOUT$', 'w', buffering=0)
    >>> sys.stdout.fileno()
    1

Python 3:

    >>> sys.stdout.close(); os.close(1); sys.stdout = open('CONOUT$', 'w') 
    >>> sys.stdout.fileno()
    1

[snaphat]

Yeah, it is windows specific. The problem is that if you open conout$ and the descriptor isn't 1, the buffer doesn't flush normally so the console won't display anything unless you manually flush it.

Now, why would you want to redirect STDOUT to the console in the first place like this? It's in the case where AllocConsole() has been called and since the console is new, STDOUT is directed elsewhere or nowhere. So you can't dup it or restore it or anything like that. In order, to get the normal behavior for the windows console the redirect is needed for STDOUT with fileno=1. But, as far as I can tell, it is impossible to actually do the redirect in python.

Anyway, let me discuss a bit the behavior between C, Python, and Perl w.r.t. to opening files. It'll help clarify where python is lacking in comparison to perl and c. You basically have two different 'flavors' of prototypes between the languages:

1. handle fd = open(filename, mode)
2. open(filename, mode, fd)

In C, the first prototype matches 'fopen' and is used to open a file. And the second prototype matches 'freopen' and is used to associate a different file with the an existing file descriptor. Moreover, you can't use stdout as an lvalue so you simply can't redefine it.

In code terms:
stdout = fopen("CONOUT$", "w"); //Invalid lvalue
FILE* file= fopen("CONOUT$", "w"); // Valid fileno=3 to N.
freopen("CONOUT$", "w", stdout); // Valid fileno=1.
freopen("CONOUT$", "w", file); //Valid IFF fd!=0, the value of fd is whatever it was previously.

In perl the open function is similar to the second prototype mentioned above, but the parameters are backwards. Moreover, It can either be used for new file descriptors or for reusing file descriptors unlike C's version of the second prototype.

In code terms:
open (STDOUT, ">", "CONOUT$"); //Valid fileno=1. Reuses the desciptor number.
open (FILE, ">", "CONOUT$"); //Valid fileno=3 to N. Reuses the fileno if a valid descriptor. Otherwise, creates a new one.

In python, the os.open function matches the first prototype with the exception that it can be used to redirect stdout unlike C's version.

In code terms:
sys.stdout = os.open("CONOUT$", "w"); // Valid fileno=3 to N.

So when you look at the behavior, you'll note that both C and Perl allow you to redirect using an existing descriptor; where as, Python seems to be unable to do this. Normally, it doesn't matter since you can just save stdout ahead of time, but it matters in the case where you don't start with a console (pythonw), or deallocated the console and allocated a new one later on.

As it stands, I don't believe this is fixable without an API extension/change of some sort that would allow passing filehandles... either that or... just making $CONOUT always allocate to 1 regardless. Though the latter would change the behavior for anyone who currently uses CONOUT$ in that it currently needs to be manually flushed to display to the console; where as, it wouldn't with the change.

Boy that is long winded, but hopefully it describes the issue adequately.

[snaphat]

Note, I just read eryksun's response. That does indeed fix the issues without API changes.

[bitdancer]

OK, do we have a documentation issue here with respect to "actually" closing the standard streams, or do we just close this as not a bug? Now that bpo-17401 is closed, in python 3.5 you will at least be able to tell that the standard stream fdclose is set False by its repr.

[rbtcollins]

I think its worth a note that the stdio streams are constructed specially, even with the repr improvements.

[4kir4]

Two minor details:

1. It is possible that fileno(stdout) != 1 even in C [1].
   I don't know what happens if the code from the answer is
   run on Windows.

   In principle, it may break eryksun's workaround. I don't
   know how likely it is in practice.

2. you can redirect at the file descriptor level in Python [2]
   using os.dup2(). I don't know whether the code in the
   answer works on Windows.

[1] http://stackoverflow.com/questions/25516375/is-it-possible-that-filenostdout-1-on-a-posix-system
[2] http://stackoverflow.com/questions/4675728/redirect-stdout-to-a-file-in-python/22434262#22434262