crypto/tls: Can't read final bytes off of TLS socket before close #7414
Description
What steps will reproduce the problem? Suppose a server emits some data on a TLS socket and then immediately closes that socket. In the current implementation of tls.Conn, the client side of that socket may or may not be able to read those final bytes off the socket if the client is also concurrently streaming writes to the server. I've attached a sample client.go and server.go that demonstrate the issue. The client essentially does this: - Writes 1, 2, 3, ... in a loop until a write error occurs - Concurrently, does a blocking read for another integer from the server The server reads integers in a loop from the client until it reads 2048, then echoes 2048 back to the client and exits. To test, fire up the server with some throwaway self-signed cert and key: $ go run server.go -port=10000 -cert=server.crt -key=server.key Then run the client. Sometimes, the reader wins and is able to read the final message from the server (2048) before the writer encounters an error: $ go run client.go -addr='secure.nhaminated.com:9999' -useTLS 2014/02/24 20:10:14.722902 read value: 2048 2014/02/24 20:10:14.723218 write error on iteration 6642 : write tcp 162.243.46.153:9999: broken pipe Other times, the writer wins and an encounters an error before the reader is able to read the final bytes off the socket: $ go run client.go -addr='secure.nhaminated.com:9999' -useTLS 2014/02/24 20:10:23.934537 write error on iteration 6494 : write tcp 162.243.46.153:9999: broken pipe 2014/02/24 20:10:23.934632 read error: write tcp 162.243.46.153:9999: broken pipe In both cases, the server actually wrote this final value of 2048, and the client's TCP stack received and acked that value, before closing the socket in an orderly fashion with a FIN. This can be verified by looking at the attached packet trace in wireshark, using the result.key master key to decrypt the SSL connection. This issue appears to be bacause tls.Conn has a single connErr field. Both tls.Conn.Read() and tls.Conn.Write() update this field and return errors from that field. The issue is that if the writer encounters an error first before the reader, then the writer sets connErr and the reader can no longer read the final bytes off the socket, even though those bytes could be read if go were to emit the read() syscall. You can omit the -useTLS flag on the client and the -key/-cert flags on the server to confirm that this issue does not occur when using raw sockets. The type of protocol I've described is extremely widely used in production. For instance, in Apple's APNs protocol, the client continuously streams push messages with ids to the server. If the server encounters a malformed push message, it returns the id of the malformed push message and then closes the socket. It is critical for the client to be able to read this error message because the server expects the client to re-connect and restart streaming from the push just after the failed id. What is the expected output? The reader should be able to read the last bytes off the socket; in the test program, it should always be able to read the integer value 2048. What do you see instead? Sometimes the reader can read the last value, and sometimes it just returns an error. Which operating system are you using? OS X 10.9.2 and Ubuntu 13.10 (Linux 3.11.0-12-generic) Which version are you using? (run 'go version' or 'gccgo --version') go version go1.2 darwin/amd64 and go version go1.2 linux/amd64
Attachments:
- server.go (2240 bytes)
- client.go (1206 bytes)
- result.key (1679 bytes)
- result.log (717 bytes)