strange "connection reset by peer" error

[nicklan]

I'm seeing odd behavior in Click for some servers. I've traced it down a bit, and can reproduce it, but it's a bit hard to give a full repro here since there's some private information. However, here's what I do know:

1. With hyper-sync-rustls, for some servers, after a first connection every other request will fail with: Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
2. This doesn't happen with hyper-rustls
3. It only seems to happen if the connection is using a client cert and key to authenticate (i.e. tls.set_single_client_cert(..) is used

I've modified the example clients from the two crates to support adding the client key/cert, and added a second get call, to reproduce it locally. Here's some logs from that.

First from hyper-sync-rustl:

$ RUST_LOG=debug cargo run --example client "https://[server]/[path] ca.cert client.cert client.key
[2020-02-13T22:33:35Z DEBUG rustls::anchors] add_pem_file processed 4 valid and 0 invalid certs
[2020-02-13T22:33:35Z DEBUG hyper::net] http scheme
[2020-02-13T22:33:35Z DEBUG hyper::net] https scheme
[2020-02-13T22:33:35Z DEBUG rustls::client::hs] No cached session for DNSNameRef("[server]")
[2020-02-13T22:33:35Z DEBUG rustls::client::hs] Not resuming any session
[2020-02-13T22:33:35Z DEBUG hyper::http::h1] request line: Get "/[path]" Http11
[2020-02-13T22:33:35Z DEBUG hyper::http::h1] headers=Headers { Host: [server]
    , }
[2020-02-13T22:33:35Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] Not resuming
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: []
[2020-02-13T22:33:35Z DEBUG rustls::client::hs] ALPN protocol is None
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] Got CertificateRequest CertificateRequestPayloadTLS13 { context: PayloadU8([]), extensions: [Unknown(UnknownExtension { typ: StatusRequest, payload: Payload([]) }), Unknown(UnknownExtension { typ: SCT, payload: Payload([]) }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_SHA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy]), AuthorityNames([PayloadU16([SNIP])])] }
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] Attempting client auth
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] Server cert is [Certificate([SNIP])]
[2020-02-13T22:33:35Z DEBUG rustls::client::tls13] Ticket saved
[2020-02-13T22:33:35Z DEBUG hyper::client::response] version=Http11, status=Forbidden
[2020-02-13T22:33:35Z DEBUG hyper::client::response] headers=Headers { Audit-Id: [snip]
    , Cache-Control: no-cache, private
    , Content-Type: application/json
    , X-Content-Type-Options: nosniff
    , Date: Thu, 13 Feb 2020 22:33:35 GMT
    , Content-Length: 264
    , }
Response: 403 Forbidden
Headers:
Audit-Id: [snip]
Cache-Control: no-cache, private
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Thu, 13 Feb 2020 22:33:35 GMT
Content-Length: 264

[THE BODY]

Sending second request

[2020-02-13T22:33:35Z DEBUG hyper::net] http scheme
[2020-02-13T22:33:35Z DEBUG hyper::net] https scheme
[2020-02-13T22:33:35Z DEBUG rustls::client::hs] Resuming session
[2020-02-13T22:33:35Z DEBUG hyper::http::h1] request line: Get "/[path]" Http11
[2020-02-13T22:33:35Z DEBUG hyper::http::h1] headers=Headers { Host: [server]
    , }
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })', src/libcore/result.rs:1188:5

and then for hyper-rustls:

$ RUST_LOG=debug cargo run --example client "https://[server]/[path] ca.cert client.cert client.key
[2020-02-13T22:44:12Z DEBUG rustls::anchors] add_pem_file processed 4 valid and 0 invalid certs
[2020-02-13T22:44:12Z DEBUG hyper::client::connect::dns] resolving host="[server]"
[2020-02-13T22:44:12Z DEBUG hyper::client::connect::http] connecting to [ip]:443
[2020-02-13T22:44:12Z DEBUG hyper::client::connect::http] connected to [ip]:443
[2020-02-13T22:44:12Z DEBUG rustls::client::hs] No cached session for DNSNameRef("[server]")
[2020-02-13T22:44:12Z DEBUG rustls::client::hs] Not resuming any session
[2020-02-13T22:44:12Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] Not resuming
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: []
[2020-02-13T22:44:12Z DEBUG rustls::client::hs] ALPN protocol is None
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] Got CertificateRequest CertificateRequestPayloadTLS13 { context: PayloadU8([]), extensions: [Unknown(UnknownExtension { typ: StatusRequest, payload: Payload([]) }), Unknown(UnknownExtension { typ: SCT, payload: Payload([]) }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_SHA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy]), AuthorityNames([SNIP])] }
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] Attempting client auth
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] Server cert is [Certificate([SNiP)]
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] flushed 98 bytes
[2020-02-13T22:44:12Z DEBUG rustls::client::tls13] Ticket saved
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] read 495 bytes
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] parsed 6 headers
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::conn] incoming body is content-length (264 bytes)
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::conn] incoming body completed
[2020-02-13T22:44:12Z DEBUG hyper::client::pool] pooling idle connection for ("https", [server])
Status:
403 Forbidden
Headers:
{
    "audit-id": "[snip]",
    "cache-control": "no-cache, private",
    "content-type": "application/json",
    "x-content-type-options": "nosniff",
    "date": "Thu, 13 Feb 2020 22:44:12 GMT",
    "content-length": "264",
}
Body:
[THE BODY]

Sending second request

[2020-02-13T22:44:12Z DEBUG hyper::client::pool] reuse idle connection for ("https", [server])
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] flushed 98 bytes
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] read 495 bytes
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::io] parsed 6 headers
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::conn] incoming body is content-length (264 bytes)
[2020-02-13T22:44:12Z DEBUG hyper::proto::h1::conn] incoming body completed
[2020-02-13T22:44:12Z DEBUG hyper::client::pool] pooling idle connection for ("https", [server])
Status2:
403 Forbidden
Headers2:
{
    "audit-id": "[snip]",
    "cache-control": "no-cache, private",
    "content-type": "application/json",
    "x-content-type-options": "nosniff",
    "date": "Thu, 13 Feb 2020 22:44:12 GMT",
    "content-length": "264",
}
Body2:
[THE BODY]

I realize there's a lot of moving parts here, but if you have any ideas I'd love to help track this down. Thanks!