Raw compressed output when not using --compressed but server returns gzip data

[nikosdion]

I did this

curl -L -s "https://downloads.joomla.org/latest" -o -"

I expected the following

Get a screen full of HTML data. What happened is that I got a gzip stream.

curl/libcurl version

curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

operating system

Ubuntu 18.04

Linux myhostname 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Further debugging

This seems to only happen with sites hosted on Rochen. I've tried the aforementioned site https://downloads.joomla.org/latest as well as https://www.dionysopoulos.me.

Using --verbose I see that cURL does not send an Accept-Encoding header at all. The server returns compressed output with the HTTP header content-encoding: gzip. However, this response header seems to be ignored by cURL 7.58.0. It also looks like cURL 7.47.0 (on Ubuntu 16.04) does not exhibit this behavior, i.e. it parses the compressed content correctly.

Passing the --compressed option to cURL the server output is uncompressed and returned correctly. In this case I see that the Accept-Encoding: gzip, deflate header is sent by cURL. The server still returns compressed output with the HTTP header content-encoding: gzip but this time cURL parses everything correctly.

In the name of ruling out a false positive I tried requests to two other sites, https://www.akeebabackup.com hosted with a different host (SiteGround, using NginX in front of Apache) and https://translate.akeeba.com (self-hosted on Linode using straight up Apache 2.4 Ubuntu 16.04). These servers seem to behave themselves. Without the --compressed option the response is sent uncompressed from the server and displayed correctly by cURL. With the --compressed option the response is sent compressed with deflate by the server, the content-encoding: deflate HTTP header is set and cURL displays the correct, uncompressed output.

I am not sure if this is a cURL bug or an issue with the way Rochen's servers behave when no Accept-Encoding HTTP header is sent at all. As far as I can see, the server's behavior is correct per RFC 2616 but incorrect per RFC 7231 (which obsoleted RFC 2616). I don't know if I'm on the right track because these RFC's are for HTTP/1.1 whereas these servers talk HTTP/2.

If it's indeed a server issue (not respecting RFC 7231) please feel free to tell me so and close this GitHub issue. I'll then file a bug report with the host.

Thank you in advance!

[bagder]

The server shouldn't send content-encoding: gzip without the client having signaled that it is acceptable.

Besides, when you don't use --compressed with curl, you tell the command line tool you rather store the exact stream (compressed or not). I don't see a curl bug here...

[nikosdion]

Thank you! I will let the hosting company know that their server configuration is wrong as I suspected.