Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
crypto/tls: Can't read final bytes off of TLS socket before close #7414
What steps will reproduce the problem? Suppose a server emits some data on a TLS socket and then immediately closes that socket. In the current implementation of tls.Conn, the client side of that socket may or may not be able to read those final bytes off the socket if the client is also concurrently streaming writes to the server. I've attached a sample client.go and server.go that demonstrate the issue. The client essentially does this: - Writes 1, 2, 3, ... in a loop until a write error occurs - Concurrently, does a blocking read for another integer from the server The server reads integers in a loop from the client until it reads 2048, then echoes 2048 back to the client and exits. To test, fire up the server with some throwaway self-signed cert and key: $ go run server.go -port=10000 -cert=server.crt -key=server.key Then run the client. Sometimes, the reader wins and is able to read the final message from the server (2048) before the writer encounters an error: $ go run client.go -addr='secure.nhaminated.com:9999' -useTLS 2014/02/24 20:10:14.722902 read value: 2048 2014/02/24 20:10:14.723218 write error on iteration 6642 : write tcp 188.8.131.52:9999: broken pipe Other times, the writer wins and an encounters an error before the reader is able to read the final bytes off the socket: $ go run client.go -addr='secure.nhaminated.com:9999' -useTLS 2014/02/24 20:10:23.934537 write error on iteration 6494 : write tcp 184.108.40.206:9999: broken pipe 2014/02/24 20:10:23.934632 read error: write tcp 220.127.116.11:9999: broken pipe In both cases, the server actually wrote this final value of 2048, and the client's TCP stack received and acked that value, before closing the socket in an orderly fashion with a FIN. This can be verified by looking at the attached packet trace in wireshark, using the result.key master key to decrypt the SSL connection. This issue appears to be bacause tls.Conn has a single connErr field. Both tls.Conn.Read() and tls.Conn.Write() update this field and return errors from that field. The issue is that if the writer encounters an error first before the reader, then the writer sets connErr and the reader can no longer read the final bytes off the socket, even though those bytes could be read if go were to emit the read() syscall. You can omit the -useTLS flag on the client and the -key/-cert flags on the server to confirm that this issue does not occur when using raw sockets. The type of protocol I've described is extremely widely used in production. For instance, in Apple's APNs protocol, the client continuously streams push messages with ids to the server. If the server encounters a malformed push message, it returns the id of the malformed push message and then closes the socket. It is critical for the client to be able to read this error message because the server expects the client to re-connect and restart streaming from the push just after the failed id. What is the expected output? The reader should be able to read the last bytes off the socket; in the test program, it should always be able to read the integer value 2048. What do you see instead? Sometimes the reader can read the last value, and sometimes it just returns an error. Which operating system are you using? OS X 10.9.2 and Ubuntu 13.10 (Linux 3.11.0-12-generic) Which version are you using? (run 'go version' or 'gccgo --version') go version go1.2 darwin/amd64 and go version go1.2 linux/amd64
FiloSottile pushed a commit to FiloSottile/go that referenced this issue
Oct 12, 2018
Currently a write error will cause future reads to return that same error. However, there may have been extra information from a peer pending on the read direction that is now unavailable. This change splits the single connErr into errors for the read, write and handshake. (Splitting off the handshake error is needed because both read and write paths check the handshake error.) Fixes golang#7414. LGTM=bradfitz, r R=golang-codereviews, r, bradfitz CC=golang-codereviews https://golang.org/cl/69090044
This issue was closed.