internal/poll: Reading data piped through os.Stdin hangs on Windows version

[jakans]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.9 darwin/amd64, but cross-compiling for PC/CYGWIN

Does this issue reproduce with the latest release?

Yes, issue is only on go1.9 and Windows, go1.8 is fine on all platforms

What operating system and processor architecture are you using (go env)?

Windows under CYGWIN. (Go compiler is not installed on target machines.)

What did you do?

curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" |
go run pipehangs.go

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Source code of pipehangs.go is inserted here:

package main

import (
"fmt"
"io/ioutil"
"os"
)

func main() {

str, err := ioutil.ReadAll(os.Stdin)
fmt.Fprintf(os.Stderr, "n: %d\n", len(str))
if err != nil {
	fmt.Fprintf(os.Stderr, "err: %s\n", err)
}

}

File attached, with .txt suffix added to get past data format test:

pipehangs.go.txt

What did you expect to see?

n: 3910

What did you see instead?

Program hangs.

It's fine if you pipe a file:

cat testfile.txt | go run pipehangs.go

or redirect stdin:

go run pipehangs.go < testfile.txt

Only reading a network result under Windows and go1.9 causes it to hang.

[ianlancetaylor]

Probably has something to do with the poller changes, but I don't know what.

CC @alexbrainman

[alexbrainman]

@jakans your program works with Windows type command:

c:\Users\Alex\dev\src\issue\go\22024>type pipehangs.go
package main

import (
        "fmt"
        "io/ioutil"
        "os"
)

func main() {

        str, err := ioutil.ReadAll(os.Stdin)
        fmt.Fprintf(os.Stderr, "n: %d\n", len(str))
        if err != nil {
                fmt.Fprintf(os.Stderr, "err: %s\n", err)
        }
}

c:\Users\Alex\dev\src\issue\go\22024>type pipehangs.go | go run pipehangs.go
n: 215

c:\Users\Alex\dev\src\issue\go\22024>

I do not have curl program installed on my computer to test your scenario. How do I install it?
But even after I install curl program, I am not sure how to debug this - it could be curl program that hangs for all we know.

What are you trying to do? If you just want to download some file from the Internet, Go standard library have plenty of tools to do that.

Alex

MSYS (with separately downloaded cURL)

$ curl.exe --version
curl 7.50.2 (i386-pc-win32) libcurl/7.50.2 OpenSSL/1.0.2e zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp
smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL libz

$ go.exe version
go version go1.9 windows/386

$ curl.exe -s \
> "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
> go.exe run pipehangs.go
n: 0

$ 

MSYS2

http://nurmi-labs.blogspot.com/2016/11/git.html

$ curl.exe --version
curl 7.50.1 (i686-w64-mingw32) libcurl/7.50.1 OpenSSL/1.0.2h zlib/1.2.8 libidn/1
.33 libssh2/1.7.0 nghttp2/1.13.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s
 rtmp rtsp scp sftp smtp smtps telnet tftp
Features: IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 Me
talink

$ go.exe version
go version go1.9 windows/386

$ curl.exe -s \
> "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
> go.exe run pipehangs.go
n: 3910

$

[ktye]

Does it hang also, if you run it from cmd.exe?
It could have something to do with the terminal (cygwin, msys/mintty vs console).
For me on mintty, pipe sometimes work, sometimes hang. On cmd.exe they always work.

the content of #22024 (comment) indicates NO outout from curl 7.50.2

in both examples from that comment the Windows Console had been utilised

for what its worth here is MSYS2's mintty.exe running a C Shell

% curl.exe -s \
? "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
? go.exe run pipehangs.go
n: 3910
%

You don't mention what cURL version you are using, or if you've run your cURL command without the pipe.

[alexbrainman]

@forskning thanks for testing. Looks like "go run pipehangs.go" does not hang in either of your scenarios.

I wonder why curl 7.50.2 does not output any data. What happens if you redirect curl output to a file? Do you get anything written to the file?

Thank you

Alex

@alexbrainman I should have looked into that prior to adding the MSYS content

http://nurmi-labs.blogspot.com/2015/11/bcc55.html

It was Dirk Paehl's cURL 7.50.2 (Download WITH SUPPORT SSL) I used for a Borland/Dmake compile of perl-5.10.1.tar.gz, the CPAN setup, and, as those were http addresses, I never added the SSL support files.

Perl/perl5@378eeda#diff-65539b463d4890c68be9c1e3de589c4d

You can read about the events which led up to the "The Borland Chainsaw Massacre" here:

http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2011-09/msg00034.html

[as]

Unable to reproduce with various versions of curl 7.55.1.
n: 0 when curl fails to initialize; but a hang never occurs.

http://www.paehl.com/open_source/?CURL_7.55.1

set URL=https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed^&id=26422376^&version=2.0

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1
n: 0
curl_755_1

No crash, instant return

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1_ssh2_ssl_sspi
n: 0
curl_755_1_ssh2_ssl_sspi

curl -s "%URL%" | go run pipehangs.go & echo curl_755_1
n: 3910
curl_755_1

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1_rtmp_ssh2_ssl_sspi
n: 0
curl_755_1_rtmp_ssh2_ssl_sspi

go get github.com/as/torgo/hget
go run %GOPATH%\src\github.com\as\torgo\hget\hget.go "%URL%" | go run pipehangs.go
n: 3910

@jakans

Does your curl command when run without the pipe output the xml to the console?

[alexbrainman]

@alexbrainman I should have looked into that prior to adding the MSYS content
...

Simply speaking you curl.exe was broken. Cool. Thank you for explaining.

Unable to reproduce with various versions of curl 7.55.1.
n: 0 when curl fails to initialize; but a hang never occurs.

Thank you @as for trying.

Alex

moot point

curl_750_2_ssl (without adding the SSL support files) returns output for http not https

$ curl.exe -s "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://eutils.ncbi.nlm.nih.gov/entrez/eutils
/esummary.fcgi?db=pubmed&amp;id=26422376&amp;version=2.0">here</a>.</p>
</body></html>

$ curl.exe -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0"

$ curl.exe --version
curl 7.50.2 (i386-pc-win32) libcurl/7.50.2 OpenSSL/1.0.2e zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp
smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL libz

$

"That they be not forced to sue the lawe, wrapped with so infinite crickes and moot poyntes."

Laurence Humphrey - 1563

[jakans]

The problem was discovered by users of Entrez Direct (https://www.ncbi.nlm.nih.gov/books/NBK179288/) who had installed the latest release on their PCs. While most of EDirect is implemented as a Perl wrapper to NCBI's URL-based Entrez Utilities (https://www.ncbi.nlm.nih.gov/books/NBK25501/), the xtract component is written in Go, and it failed to read pipes from the upstream Perl steps. I quickly reverted to the Go 1.8 xtract binary for CYGWIN (Mac and Linux versions are fine under Go 1.9), and then isolated the problem and created the minimal pipehangs.go program for debugging.

Entrez Direct provides a simple way for non-programmers to do sophisticated data mining in PubMed and other interconnected NCBI databases. The target audience includes scientists, medical librarians, bibliometric researchers, and academic administrators, as well as computational biologists who are looking for an easy way to get specific information from our databases. Users supply query details in command-line arguments. Separate steps in the query are connected by Unix pipes. Everything else is handled behind the scenes, with no additional coding required.

The sample ad hoc EDirect query shown below searches for journal articles in PubMed and retrieves the records in a defined XML form. The XML data set is piped to xtract, which limits results to journals published in the U.S. and then visits each author, printing one author name per line. This output is piped to a standard script that produces a frequency table of publications per author:

esearch -db pubmed -query "rattlesnake phospholipase" |
efetch -format xml |
xtract -pattern PubmedArticle
-if MedlineJournalInfo/Country -equals "United States"
-block Author -tab "\n" -sep " " -element LastName,Initials |
sort-uniq-count-rank

9 Wells MA
8 Marangoni S
8 Toyama MH
6 Dennis EA
5 Bon C
5 Kézdy FJ
5 Sigler PB
5 Soares AM
4 Carlini CR
4 Faure G
4 Francischetti IM
4 Guimarães JA
4 HANAHAN DJ
4 Heinrikson RL
...

To answer the earlier question, curl.exe --version produces:
curl 7.43.0 (x86_64-unknown-cygwin) libcurl/7.43.0 OpenSSL/1.0.2d zlib/1.2.8 libidn/1.29 libssh2/1.5.0