Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall: reading os.Stdin does not treat ^Z as EOF on Windows #6303

Closed
mattn opened this issue Sep 2, 2013 · 11 comments
Closed

syscall: reading os.Stdin does not treat ^Z as EOF on Windows #6303

mattn opened this issue Sep 2, 2013 · 11 comments
Milestone

Comments

@mattn
Copy link
Member

@mattn mattn commented Sep 2, 2013

1. call io.Copy(os.Stdout, os.Stdin)
2. Type CTRL-Z on console


What is the expected output?

  Interrupt stdin

What do you see instead?

  Seeing ^Z character

Which compiler are you using (5g, 6g, 8g, gccgo)?

  8g

Which operating system are you using?

  Windows 7

Which version are you using?  (run 'go version')

  go version devel +2f13cc5d7043 Tue Aug 27 14:53:57 2013 +1000 windows/386


Please provide any additional information below.

  https://groups.google.com/forum/#!topic/golang-dev/xnjP0VlL2nw
@rsc
Copy link
Contributor

@rsc rsc commented Oct 18, 2013

Comment 1:

Labels changed: added priority-later, go1.3maybe, removed priority-triage.

Status changed to Accepted.

@rsc
Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 2:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 3:

Labels changed: added repo-main.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 27, 2014

Comment 4:

CL https://golang.org/cl/95780046 mentions this issue.
@peterGo
Copy link
Contributor

@peterGo peterGo commented Apr 27, 2014

Comment 5:

More complaints:
Ctrl-Z and EOF in Windows shells
https://groups.google.com/forum/#!topic/golang-nuts/7rMVIUW8Fic
One proposed solution to this issue:
os,syscall: use ReadFile/MultiByteToWideChar to read from console
https://golang.org/cl/45150045/
Another proposed solution to this issue:
os,syscall: On Windows, recognize Control-Z for console input
https://golang.org/cl/95780046/
@peterGo
Copy link
Contributor

@peterGo peterGo commented May 4, 2014

Comment 6:

A minimal test program.

Attachments:

  1. i6303.go (143 bytes)
@mattn mattn added accepted labels May 4, 2014
@techtonik
Copy link

@techtonik techtonik commented Mar 17, 2015

+1 for fixing this. Looks like the only way to terminate input from stdin in Go 1.4.2 is to be prepared for it.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 25, 2015

CL https://golang.org/cl/4310 mentions this issue.

@techtonik
Copy link

@techtonik techtonik commented Oct 22, 2015

Maybe make that optional - ability to choose what combination can be used to terminate the stream.

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2016

CL https://golang.org/cl/33451 mentions this issue.

gopherbot pushed a commit that referenced this issue Nov 29, 2016
Go 1.5 worked with Unicode console input but not ^Z.
Go 1.6 did not work with Unicode console input but did handle one ^Z case.
Go 1.7 did not work with Unicode console input but did handle one ^Z case.

The intent of this CL is for Go 1.8 to work with Unicode console input
and also handle all ^Z cases.

Here's a simple test program for reading from the console.
It prints a "> " prompt, calls read, prints what it gets, and repeats.

	package main

	import (
	    "fmt"
	    "os"
	)

	func main() {
	    p := make([]byte, 100)
	    fmt.Printf("> ")
	    for {
	        n, err := os.Stdin.Read(p)
	        fmt.Printf("[%d %q %v]\n> ", n, p[:n], err)
	    }
	}

On Unix, typing a ^D produces a break in the input stream.
If the ^D is at the beginning of a line, then the 0 bytes returned
appear as an io.EOF:

	$ go run /tmp/x.go
	> hello
	[6 "hello\n" <nil>]
	> hello^D[5 "hello" <nil>]
	> ^D[0 "" EOF]
	> ^D[0 "" EOF]
	> hello^Dworld
	[5 "hello" <nil>]
	> [6 "world\n" <nil>]
	>

On Windows, the EOF character is ^Z, not ^D, and there has
been a long-standing problem that in Go programs, ^Z on Windows
does not behave in the expected way, namely like ^D on Unix.
Instead, the ^Z come through as literal ^Z characters:

	C:\>c:\go1.5.4\bin\go run x.go
	> ^Z
	[3 "\x1a\r\n" <nil>]
	> hello^Zworld
	[13 "hello\x1aworld\r\n" <nil>]
	>

CL 4310 attempted to fix this bug, then known as #6303,
by changing the use of ReadConsole to ReadFile.
This CL was released as part of Go 1.6 and did fix the case
of a ^Z by itself, but not as part of a larger input:

	C:\>c:\go1.6.3\bin\go run x.go
	> ^Z
	[0 "" EOF]
	> hello^Zworld
	[13 "hello\x1aworld\r\n" <nil>]
	>

So the fix was incomplete.
Worse, the fix broke Unicode console input.

ReadFile does not handle Unicode console input correctly.
To handle Unicode correctly, programs must use ReadConsole.
Early versions of Go used ReadFile to read the console,
leading to incorrect Unicode handling, which was filed as #4760
and fixed in CL 7312053, which switched to ReadConsole
and was released as part of Go 1.1 and still worked as of Go 1.5:

	C:\>c:\go1.5.4\bin\go run x.go
	> hello
	[7 "hello\r\n" <nil>]
	> hello world™
	[16 "hello world™\r\n" <nil>]
	>

But in Go 1.6:

	C:\>c:\go1.6.3\bin\go run x.go
	> hello
	[7 "hello\r\n" <nil>]
	> hello world™
	[0 "" EOF]
	>

That is, changing back to ReadFile in Go 1.6 reintroduced #4760,
which has been refiled as #17097. (We have no automated test
for this because we don't know how to simulate console input
in a test: it appears that one must actually type at a keyboard
to use the real APIs. This CL at least adds a comment warning
not to reintroduce ReadFile again.)

CL 29493 attempted to fix #17097, but it was not a complete fix:
the hello world™ example above still fails, as does Shift-JIS input,
which was filed as #17939.

CL 29493 also broke ^Z handling, which was filed as #17427.

This CL attempts the never before successfully performed trick
of simultaneously fixing Unicode console input and ^Z handling.
It changes the console input to use ReadConsole again,
as in Go 1.5, which seemed to work for all known Unicode input.
Then it adds explicit handling of ^Z in the input stream.
(In the case where standard input is a redirected file, ^Z processing
should not happen, and it does not, because this code path is only
invoked when standard input is the console.)

With this CL:

	C:\>go run x.go
	> hello
	[7 "hello\r\n" <nil>]
	> hello world™
	[16 "hello world™\r\n" <nil>]
	> ^Z
	[0 "" EOF]
	> [2 "\r\n" <nil>]
	> hello^Zworld
	[5 "hello" <nil>]
	> [0 "" EOF]
	> [7 "world\r\n" <nil>]

This almost matches Unix:

	$ go run /tmp/x.go
	> hello
	[6 "hello\n" <nil>]
	> hello world™
	[15 "hello world™\n" <nil>]
	> ^D
	[0 "" EOF]
	> [1 "\n" <nil>]
	> hello^Dworld
	[5 "hello" <nil>]
	> [6 "world\n" <nil>]
	>

The difference is in the handling of hello^Dworld / hello^Zworld.
On Unix, hello^Dworld terminates the read of hello but does not
result in a zero-length read between reading hello and world.
This is dictated by the tty driver, not any special Go code.

On Windows, in this CL, hello^Zworld inserts a zero length read
result between hello and world, which is treated as an interior EOF.
This is implemented by the Go code in this CL, but it matches the
handling of ^Z on the console in other programs:

	C:\>copy con x.txt
	hello^Zworld
	        1 file(s) copied.

	C:\>type x.txt
	hello
	C:\>

A natural question is how to test all this. As noted above, we don't
know how to write automated tests using the actual Windows console.
CL 29493 introduced the idea of substituting a different syscall.ReadFile
implementation for testing; this CL continues that idea but substituting
for syscall.ReadConsole instead. To avoid the regression of putting
ReadFile back, this CL adds a comment warning against that.

Fixes #17427.
Fixes #17939.

Change-Id: Ibaabd0ceb2d7af501d44ac66d53f64aba3944142
Reviewed-on: https://go-review.googlesource.com/33451
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Quentin Smith <quentin@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@golang golang locked and limited conversation to collaborators Nov 22, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.