sys.stdin.read() doesn't return after first EOF on Windows

[rmosaic]

BPO	5505
Nosy	@pitrou, @vstinner, @ned-deily, @anacrolix, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-03-14.21:52:23.873>
created_at = <Date 2009-03-18.09:20:28.147>
labels = ['type-bug', 'library']
title = "sys.stdin.read() doesn't return after first EOF on Windows"
updated_at = <Date 2016-03-14.21:52:23.872>
user = 'https://bugs.python.org/rmosaic'

bugs.python.org fields:

activity = <Date 2016-03-14.21:52:23.872>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-14.21:52:23.873>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2009-03-18.09:20:28.147>
creator = 'r_mosaic'
dependencies = []
files = []
hgrepos = []
issue_num = 5505
keywords = []
message_count = 19.0
messages = ['83736', '83742', '83755', '83756', '84145', '84166', '91154', '91296', '139994', '139995', '140322', '140326', '140561', '140562', '140563', '140564', '140572', '140573', '261723']
nosy_count = 10.0
nosy_names = ['ggenellina', 'pitrou', 'vstinner', 'ocean-city', 'r_mosaic', 'ned.deily', 'cassava', 'famart', 'anacrolix', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5505'
versions = ['Python 3.2']

[rmosaic]

Platform: Windows Vista x64
Python version: 3.0.1 x64

When I use sys.stdin.readlines(), it correctly ends when I enter ^Z
(ctrl-Z) on the console (this is the EOF on the console). However when
I call sys.stdin.read(), it doesn't end when ^Z is entered. It ends
when I enter the second ^Z.

Repro steps:

1. Open python.
2. Type:
   import sys
   sys.stdin.read()
3. Type:
   Hello
   ^Z
4. Note the above ^Z should be followed by a Return.
   Result:
   The function call doesn't end. If I enter another ^Z, it ends.
   Expected result:
   The function call ends.

[pitrou]

(cannot reproduce under Linux)

[ocean-city]

I can reproduce this on coLinux(debian). But coLinux is running on
windows, so its console behavior may not be same as native linux.
(You need to use Ctrl+D on coLinux instead of Ctrl+Z)

[ocean-city]

With following patch for investigation, (release30-maint)

Index: Lib/io.py
===================================================================

--- Lib/io.py	(revision 70450)
+++ Lib/io.py	(working copy)
@@ -57,6 +57,7 @@
 
 import os
 import abc
+import sys
 import codecs
 import _fileio
 # Import _thread instead of threading to reduce startup cost
@@ -931,6 +932,7 @@
             while True:
                 # Read until EOF or until read() would block.
                 chunk = self.raw.read()
+                print("============>", repr(chunk), file=sys.stderr)
                 if chunk in empty_values:
                     nodata_val = chunk
                     break

///////////////////////

I got this result.

>>> sys.stdin.read()
abc
^Z
============> b'abc\n'
^Z
============> b''
'abc\n'

To get empty chunk, we need to hit CTRL-Z twice.

[rmosaic]

Perhaps using just one read() is enough? I mean perhaps no loop is
necessary?

[cassava]

This is happening on Arch Linux as well:

Python 3.0.1 (r301:69556, Feb 22 2009, 02:43:30)
[GCC 4.3.3] on linux2

I tried python2.6 and it ends with one Ctrl+D
I tried python3.0.1 and it needs two Ctrl+D before it ends

[famart]

Looks like python needs eof() or something for file objects, just like
any other languages.

Since read() is using the system call, that's the right behavior: read()
blocks until EOF, and returns whatever was buffered. EOF character is
consumed, but since it's a stdin, it is never closed. The next read()
will again wait for normal input.

The 2nd EOF mark without anything in-between will return an empty string.

[ggenellina]

This is a duplicate of bpo-1633941

[vstinner]

I am still able to reproduce the problem with Python 3.2.1RC1 (64 bits) on Windows Seven, but not on Python 3.3 (alpha) compiled myself (using Visual C++ Express 2008).

I don't know if something changed in Python 3.3, or it is related to how I compiled Python? I don't think that it is related to issue bpo-12016 because Python 3.2.1RC1 contains the fix for bpo-12016.

[vstinner]

I don't know if something changed in Python 3.3, or ...

Yes, something changed in Python 3.3. I fixed this issue "by mistake" :-) The fix is the following commit:

New changeset 3c7792ec4547 by Victor Stinner in branch 'default':
Issue bpo-12175: BufferedReader.read(-1) now calls raw.readall() if available.
http://hg.python.org/cpython/rev/3c7792ec4547

It is a bug in BufferedReader, a bug or an unexpected behaviour. Before this commit (e.g. in Python 3.2.0), BufferedReader.read(-1) calls raw.read() two times for this issue: the first call returns an non empty string (the string until the first ^Z), the second call returns an empty string (the string until the second ^Z). BufferedReader.read(-1) loop ends with raw.read() returns an empty string.

You can workaround this issue by calling sys.stdin.buffer.raw.readall() or sys.stdin.buffer.raw.read(-1).

I chose to not port my BufferedRead.read(-1) fix (call raw.readall() if available) in Python 3.2 because it changes the behaviour, and I don't want to do that in a minor release.

Is anyone in favor of changing the behaviour of BufferedRead.read(-1) in a minor release (next 2.7.3 and 3.2.2)? Or can I close the issue (the bug is already fixed in Python 3.3)?

[anacrolix]

I get this on Linux with ^D

[vstinner]

I get this on Linux with ^D

With which Python version? Did you try Python 3.3 (development version)?

[anacrolix]

Feel like a total noob: Where do I get the latest source? I can't find any pre-release tarballs for 3.3, and the suggested py3k checkout doesn't work: $ hg clone http://hg.python.org/cpython#py3k py3k
abort: unknown revision 'py3k'!

[ned-deily]

See the developer's guide: http://docs.python.org/devguide/setup.html#getting-the-source-code

hg clone http://hg.python.org/cpython directory_name

[anacrolix]

This version is fixed for me:

$ ./python
Python 3.3.0a0 (default:7520f1bf0a81, Jul 18 2011, 17:12:12)
[GCC 4.1.2 20070115 (SUSE Linux)] on linux2

[vstinner]

@pitrou: Antoine, do you think that the following commit should be backported from 3.3 to 3.2?

New changeset 3c7792ec4547 by Victor Stinner in branch 'default':
Issue bpo-12175: BufferedReader.read(-1) now calls raw.readall() if available.
http://hg.python.org/cpython/rev/3c7792ec4547

It changes BufferedReader.read() behaviour a *little* bit. Only a little because FileIO.read(-1) calls FileIO.readall() internally for example.

[pitrou]

Antoine, do you think that the following commit should be backported
from 3.3 to 3.2?

No, I don't think so.

[vstinner]

No, I don't think so.

The issue is already fixed in 3.3, so you agree to not fix it in Python 3.2?

[vadmium]

Since this was apparenly only a bug in 3.2, can we close it as being out of date?