os: Windows readConsole fails on short read of multi-byte rune

[mattn]

What version of Go are you using (go version)?

go version devel +6fd8c00 Wed Sep 14 08:42:28 2016 +0000 windows/amd64

What operating system and processor architecture are you using (go env)?

windows/amd64

Windows7 64bit

What did you do?

package main

import (
    "fmt"
)

func main() {
    var foo string
    fmt.Scan(&foo)
    fmt.Println(foo)
}

And type multi-byte string for example 世界

What did you expect to see?

print 世界

What did you see instead?

print empty

[davecheney]

Please always check your errors before reporting a bug. fmt.Scan returns an error value, it will tell you what went wrong.

[mattn]

yes, I will explaining in next comment. please wait.

[mattn]

fmt.Scan return EOF with multi-byte string because fmt.Scan call io.ReadFull() with sized 1.

go/src/fmt/scan.go

Lines 321 to 333 in 510fb63

	func (r *readRune) readByte() (b byte, err error) {
	if r.pending > 0 {
	b = r.pendBuf[0]
	copy(r.pendBuf[0:], r.pendBuf[1:])
	r.pending--
	return
	}
	n, err := io.ReadFull(r.reader, r.pendBuf[:1])
	if n != 1 {
	return 0, err
	}
	return r.pendBuf[0], err
	}

os.Stdin.Read i.e. readConsole desn't decode the 1 byte to utf-8. So f.readbuf always nil.

go/src/os/file_windows.go

Lines 298 to 306 in 510fb63

	for i, r := range f.readbuf {
	if utf8.RuneLen(r) > len(b) {
	f.readbuf = f.readbuf[i:]
	return n, nil
	}
	nr := utf8.EncodeRune(b, r)
	b = b[nr:]
	n += nr
	}

[davecheney]

I think what you are seeing is your input is not utf8. A string only
accepts utf8, so if you're using big-5 or some other encoding you'll have
to wrap that input in a translator.

On Wed, 14 Sep 2016, 21:33 mattn notifications@github.com wrote:

fmt.Scan return EOF with multi-byte string because fmt.Scan call
io.ReadFull() with sized 1.

go/src/fmt/scan.go

Lines 321 to 333 in 510fb63

	func (r *readRune) readByte() (b byte, err error) {
	if r.pending > 0 {
	b = r.pendBuf[0]
	copy(r.pendBuf[0:], r.pendBuf[1:])
	r.pending--
	return
	}
	n, err := io.ReadFull(r.reader, r.pendBuf[:1])
	if n != 1 {
	return 0, err
	}
	return r.pendBuf[0], err
	}

os.Stdin.Read i.e. readConsole desn't decode the 1 byte to utf-8. So
f.readbuf always nil.

go/src/os/file_windows.go

Lines 298 to 306 in 510fb63

	for i, r := range f.readbuf {
	if utf8.RuneLen(r) > len(b) {
	f.readbuf = f.readbuf[i:]
	return n, nil
	}
	nr := utf8.EncodeRune(b, r)
	b = b[nr:]
	n += nr
	}

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#17097 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA1IZ0xA8EFvPlhsyptThTuXKp96Lks5qp9tsgaJpZM4J8qnv
.

[mattn]

On windows, the input bytes are encoded as Double Byte Character Set. For example, in japanese, it is cp932. So readConsole read bytes and convert it to utf-8. However, the buffer size is not enough to decode bytes. I guess, to fix this problem, os.File need to have buffer that can store multi-byte letter enough.

[mattn]

we can change code page of cmd.exe like chcp 65001. So the buffer size should be max bytes of utf-8 rune.

[mattn]

FYI, bufio.Buffer have bytes enough to read utf-8 bytes, so this can work correctly.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    var foo string
    fmt.Fscan(bufio.NewReader(os.Stdin), &foo)
    fmt.Println(foo)
}

But not same as non-Windows OSs.

[ianlancetaylor]

At first glance I think that readConsole is doing the wrong thing. It buffers runes but it is supposed to be returning a UTF-8 sequence of bytes, so in the case utf8.RuneLen(r) > len(b) when n == 0 && len(b) > 0 it needs to break up r into a byte sequence, return the bytes that fit into b, and buffer the remaining bytes for the next call.

[ianlancetaylor]

CC @alexbrainman

[alexbrainman]

so in the case utf8.RuneLen(r) > len(b) when n == 0 && len(b) > 0 it needs to break up r into a byte sequence, return the bytes that fit into b, and buffer the remaining bytes for the next call.

Are you saying that it is wrong for readConsole(b) to return 0, nil when len(b) > 0? But why?

Alex

[mattn]

var b [1]byte
readConsole(b[:])
fmt.Println("b1=%X", b[0])
readConsole(b[:])
fmt.Println("b2=%X", b[0])
readConsole(b[:])
fmt.Println("b3=%X", b[0])

Typing あ on cp932. (0x82 0xA0 in cp932, 0xE3 0x81 0x82 in utf-8)

I think right behavior is:

b1=E3
b2=81
b3=82

[ianlancetaylor]

@alexbrainman It's not entirely wrong for readConsole to return 0 bytes for a 1 byte read. But it's confusing and unexpected. Go programs expect that when they try to read 1 byte, they will either get 1 byte or they will get 0 bytes and io.EOF. They don't expect to have to retry the read with a larger buffer.

As the documentation for io.Reader says: "Implementations of Read are discouraged from returning a zero byte count with a nil error, except when len(p) == 0."

[alexbrainman]

@ianlancetaylor thank you for explaining. I think I see the problem now. I will try to build new test using @mattn example.

Thank you both again.

Alex

[gopherbot]

CL https://golang.org/cl/29493 mentions this issue.

[mattn]

@alexbrainman thanks for fixing this. While trying this fix, I notice ^Z doesn't work. Maybe, this part is better to exit the loop.

diff --git a/src/os/file_windows.go b/src/os/file_windows.go
index ed06b55..aa5b218 100644
--- a/src/os/file_windows.go
+++ b/src/os/file_windows.go
@@ -233,6 +233,9 @@ func (f *File) readOneUTF16FromConsole() (uint16, error) {
            return 0, err
        }
        if nmb == 0 {
+           if len(mbytes) == 0 {
+               return 0, io.EOF
+           }
            continue
        }
        mbytes = append(mbytes, buf[0])

Or

diff --git a/src/os/file_windows.go b/src/os/file_windows.go
index ed06b55..e79fc5b 100644
--- a/src/os/file_windows.go
+++ b/src/os/file_windows.go
@@ -233,7 +233,7 @@ func (f *File) readOneUTF16FromConsole() (uint16, error) {
            return 0, err
        }
        if nmb == 0 {
-           continue
+           return 0, io.EOF
        }
        mbytes = append(mbytes, buf[0])

How do you think?

[bradfitz]

@mattn, this is a closed bug, so you might want to file a new one to be sure it won't get lost.

[gopherbot]

CL https://golang.org/cl/33451 mentions this issue.