curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

[bog-dan-ro]

Curl fails on some sites e.g.:
https://hazards.fema.gov/gis/nfhl/rest/services/KMZ/KMZ/MapServer?f=json
I test curl from debian sid and also I build it from source, with the same results.
The previous URL works perfectly on browsers (chrome & firefox) and also with wget[2].

[jay]

I can't reproduce this. Please give us your version and the verbose output.

curl 7.66.0 (x86_64-pc-linux-gnu) libcurl/7.66.0 OpenSSL/1.1.1d zlib/1.2.8 nghttp2/1.39.2 librtmp/2.3
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

[bagder]

I get it with what appears to be almost the same version:

curl 7.66.0 (x86_64-pc-linux-gnu) libcurl/7.66.0 OpenSSL/1.1.1d zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.39.2 librtmp/2.3
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

[bog-dan-ro]

Here is the version & output

$ curl --version
curl 7.66.0 (x86_64-pc-linux-gnu) libcurl/7.66.0 OpenSSL/1.1.1d zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.39.2 librtmp/2.3
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

$ curl -vv "https://hazards.fema.gov/gis/nfhl/rest/services/KMZ/KMZ/MapServer?f=json"        
*   Trying 216.81.81.119:443...
* TCP_NODELAY set
* Connected to hazards.fema.gov (216.81.81.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: businessCategory=Government Entity; jurisdictionC=US; serialNumber=Government Entity; C=US; ST=DC; L=Washington; O=Federal Emergency Management Agency; OU=RISKMAPCDS; CN=hazards.fema.gov
*  start date: Jun 26 00:00:00 2019 GMT
*  expire date: Aug 25 12:00:00 2020 GMT
*  subjectAltName: host "hazards.fema.gov" matched cert's "hazards.fema.gov"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
> GET /gis/nfhl/rest/services/KMZ/KMZ/MapServer?f=json HTTP/1.1
> Host: hazards.fema.gov
> User-Agent: curl/7.66.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 3821
< content-type: application/json;charset=UTF-8
< date: Tue, 24 Sep 2019 09:05:41 GMT
< etag: e89aa453
< p3p: CP="NON CUR OTPi OUR NOR UNI"
< server: Microsoft-IIS/7.5, 
< vary: Origin
< cache-control: private, must-revalidate, max-age=0
< x-powered-by: ASP.NET
< x-aspnet-version: 4.0.30319
< strict-transport-security:  max-age=31536000; includeSubDomains
< 
{"currentVersion":10.6,"serviceDescription":"","mapName":"Layers","description":"","copyrightText":"","supportsDynamicLayers":false,"layers":[{"id":0,"name":"NFHL Availability","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":0,"maxScale":250001},{"id":1,"name":"FIRM Panels","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":2,"name":"LOMRs","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":3,"name":"LOMAs","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":4,"name":"River Mile Markers","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":5,"name":"Cross-Sections","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":24000,"maxScale":0},{"id":6,"name":"Coastal Transects","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":7,"name":"Base Flood Elevations","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":24000,"maxScale":0},{"id":9,"name":"Political Jurisdictions","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":10,"name":"Levees","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":11,"name":"General Structures","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":12,"name":"Flood Hazard Boundaries","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":13,"name":"Profile Baselines","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":14,"name":"Transect Baselines","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":15,"name":"Limit of Moderate Wave Action","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":16,"name":"PLSS","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":17,"name":"Flood Hazard Zones","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":[18,19],"minScale":0,"maxScale":0},{"id":18,"name":"Flood Hazard Zone Labels","parentLayerId":17,"defaultVisibility":true,"subLayerIds":null,"minScale":5000,"maxScale":0},{"id":19,"name":"Flood Hazard Zones","parentLayerId":17,"defaultVisibility":true,"subLayerIds":null,"minScale":50000,"maxScale":0}],"tables":[],"spatialReference":{"wkid":4269,"latestWkid":4269},"singleFusedMapCache":false,"initialExtent":{"xmin":-136.44085554865705,"ymin":46.74731364098251,"xmax":-51.13717848964153,"ymax":80.5356599237725,"spatialReference":{"wkid":4269,"latestWkid":4269}},"fullExtent":{"xmin":-170.875000074836,"ymin":-14.4375082555395,"xmax":278.438048536257,"ymax":66.6250124339811,"spatialReference":{"wkid":4269,"latestWkid":4269}},"minScale":0,"maxScale":0,"units":"esri* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Closing connection 0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
DecimalDegrees","supportedImageFormatTypes":"PNG32,PNG24,PNG,JPG,DIB,TIFF,EMF,PS,PDF,GIF,SVG,SVGZ,BMP","documentInfo":{"Title":"","Author":"","Comments":"","Subject":"","Category":"","AntialiasingMode":"None","TextAntialiasingMode":"Force","Keywords":""},"capabilities":"Map,Query,Data","supportedQueryFormats":"JSON, AMF, geoJSON","exportTilesAllowed":false,"datumTransformations":[{"geoTransforms":[{"wkid":108001,"latestWkid":1241,"transformForward":true,"name":"NAD_1927_To_NAD_1983_NADCON"}]},{"geoTransforms":[{"wkid":108001,"latestWkid":1241,"transformForward":false,"name":"NAD_1927_To_NAD_1983_NADCON"}]}],"supportsDatumTransfo

Again, I can reproduce it with the HEAD of the master of the git repo

[bagder]

errno 104 is Connection reset by peer. (#4411 just filed that makes the error message for this case slightly more readable). I don't see what curl does wrong here. OpenSSL returns this error to us...

[bog-dan-ro]

@bagder This means it's an OpenSSL issue?

[bagder]

Additional info: I rebuilt and tried with GnuTLS 3.6.9 instead and that too fails in a similar fashion after 3708 bytes received :

== Info: GnuTLS recv error (-54): Error in the pull function.
== Info: Marked for [closure]: Transfer returned error

[bog-dan-ro]

Why wget[2] and the browsers work? Maybe they are setting some mysterious OpenSSL flags?

[bagder]

I don't know.

[jay]

I made a mistake it looks I did reproduce it but what happened was the curl error was embedded in the terminal output instead of at the end of it which doesn't usually happen:

{"currentVersion":10.6,"serviceDescription":"","mapName":"Layers","description":"","copyrightText":"","supportsDynamicLayers":false,"layers":[{"id":0,"name":"NFHL Availability","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":0,"maxScale":250001},{"id":1,"name":"FIRM Panels","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":2,"name":"LOMRs","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":3,"name":"LOMAs","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":4,"name":"River Mile Markers","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":5,"name":"Cross-Sections","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":null,"minScale":24000,"maxScale":0},{"id":6,"name":"Coastal Transects","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":7,"name":"Base Flood Elevations","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":24000,"maxScale":0},{"id":9,"name":"Political Jurisdictions","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":250000,"maxScale":0},{"id":10,"name":"Levees","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":11,"name":"General Structures","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":12,"name":"Flood Hazard Boundaries","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":13,"name":"Profile Baselines","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":14,"name":"Transect Baselines","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":15,"name":"Limit of Moderate Wave Action","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":50000,"maxScale":0},{"id":16,"name":"PLSS","parentLayerId":-1,"defaultVisibility":false,"subLayerIds":null,"minScale":100000,"maxScale":0},{"id":17,"name":"Flood Hazard Zones","parentLayerId":-1,"defaultVisibility":true,"subLayerIds":[18,19],"minScale":0,"maxScale":0},{"id":18,"name":"Flood Hazard Zone Labels","parentLayerId":17,"defaultVisibility":true,"subLayerIds":null,"minScale":5000,"maxScale":0},{"id":19,"name":"Flood Hazard Zones","parentLayerId":17,"defaultVisibility":true,"subLayerIds":null,"minScale":50000,"maxScale":0}],"tables":[],"spatialReference":{"wkid":4269,"latestWkid":4269},"singleFusedMapCache":false,"initialExtent":{"xmin":-136.44085554865705,"ymin":46.74731364098251,"xmax":-51.13717848964153,"ymax":80.5356599237725,"spatialReference":{"wkid":4269,"latestWkid":4269}},"fullExtent":{"xmin":-170.875000074836,"ymin":-14.4375082555395,"xmax":278.438048536257,"ymax":66.6250124339811,"spatialReference":{"wkid":4269,"latestWkid":4269}},"minScale":0,"maxScale":0,"units":"esricurl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
DecimalDegrees","supportedImageFormatTypes":"PNG32,PNG24,PNG,JPG,DIB,TIFF,EMF,PS,PDF,GIF,SVG,SVGZ,BMP","documentInfo":{"Title":"","Author":"","Comments":"","Subject":"","Category":"","AntialiasingMode":"None","TextAntialiasingMode":"Force","Keywords":""},"capabilities":"Map,Query,Data","supportedQueryFormats":"JSON, AMF, geoJSON","exportTilesAllowed":false,"datumTransformations":[{"geoTransforms":[{"wkid":108001,"latestWkid":1241,"transformForward":true,"name":"NAD_1927_To_NAD_1983_NADCON"}]},{"geoTransforms":[{"wkid":108001,"latestWkid":1241,"transformForward":false,"name":"NAD_1927_To_NAD_1983_NADCON"}]}],"supportsDatumTransfo

I tried some versions on Windows and I get the same error. A very old version of curl works fine though:

curl 7.39.0 (i386-pc-win32) libcurl/7.39.0 OpenSSL/1.0.2n zlib/1.2.8 libidn/1.32 libssh2/1.4.3 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN Largefile SSPI SPNEGO NTLM SSL libz

I still suspect the server but I'll do a bisect later

[OldTimer571]

I believe this may be an invalid Content Length Error?
The curl error buffer (-v) is not being cleared before outputting all of the additional available message thus it is being erroneously inserted into the content.?

[bagder]

The curl error buffer (-v) is not being cleared before outputting all of the additional available message thus it is being erroneously inserted into the content.?

I don't think that's the case. I don't see any evidence of such a wrong-doing, not to mention how difficult it would be to actually do such a mistake in the code.

[OldTimer571]

Perhaps this info will help trace this code loop?
(I am not a C programmer)

Stderr output (at exactly the same place in the returned input)
Without squid proxy (as per report)
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

With Squid Proxy
curl: (18) transfer closed with 113 bytes remaining to read

[bagder]

That hints on the same problem: the connection gets closed prematurely. "Connection reset by peer".

I don't know why you'd only see this with curl (or why the others hide the problem) but clearly it also happens when you use squid, which then isn't curl getting the problem first-hand anymore.

[jay]

I don't have a way to bisect this. In Ubuntu I see server returning RST with curl w GnuTLS or OpenSSL and they error. No error consistently with this super old version, but building from the repo doesn't work that far back for me:

curl 7.17.0 (x86_64-unknown-linux-gnu) libcurl/7.17.0 OpenSSL/1.0.2g zlib/1.2.8 libidn/1.32
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: IDN IPv6 Largefile NTLM SSL libz 

In Windows I get errors curl 7.59 w mbedTLS but I don't remember how it was built. No errors latest curl with wolfSSL, OpenSSL or schannel.

Can anyone reproduce both ways with a single sslbackend to get a bisect? It is likely a server issue but still I'm curious what makes the difference.