nix 1.12: x-gzip encoding causes an error

[mguentner]

Nix 1.12 behavior:

[nix-shell:~]$ nix-prefetch-url http://www.mr511.de/software/libelf-0.8.13.tar.gz --print-path --type sha256 0vf7s9dwk2xkmhb79aigqm0x0yfbw1j0b9ksm51207qwr179n6jr
error: unsupported Content-Encoding ‘x-gzip’
[nix-shell:~]$ nix-prefetch-url --version
nix-prefetch-url (Nix) 1.12pre5511_c94f3d55

Nix 1.11 behavior:

 ~  nix-prefetch-url http://www.mr511.de/software/libelf-0.8.13.tar.gz --print-path --type sha256 0vf7s9dwk2xkmhb79aigqm0x0yfbw1j0b9ksm51207qwr179n6jr
downloading ‘http://www.mr511.de/software/libelf-0.8.13.tar.gz’... [0/0 KiB, 0.0 KiB/s]
0vf7s9dwk2xkmhb79aigqm0x0yfbw1j0b9ksm51207qwr179n6jr
/nix/store/ggpgsklddx6h8dk0wkmlnk4brcxdb0jh-libelf-0.8.13.tar.gz
 ~  nix-prefetch-url --version
nix-prefetch-url (Nix) 1.11.14

curl output:

curl http://www.mr511.de/software/libelf-0.8.13.tar.gz -v -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 217.160.231.164...
* TCP_NODELAY set
* Connected to www.mr511.de (217.160.231.164) port 80 (#0)
> GET /software/libelf-0.8.13.tar.gz HTTP/1.1
> Host: www.mr511.de
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/gzip
< Content-Length: 148529
< Connection: keep-alive
< Keep-Alive: timeout=15
< Date: Fri, 15 Sep 2017 17:36:19 GMT
< Server: Apache
< Last-Modified: Sun, 01 Nov 2009 13:12:13 GMT
< ETag: "24431-4774f026c19bd"
< Accept-Ranges: bytes
< Content-Encoding: x-gzip
<
{ [1069 bytes data]
100  145k  100  145k    0     0   145k      0  0:00:01 --:--:--  0:00:01  516k
* Connection #0 to host www.mr511.de left intact

[mguentner]

This is btw no "exotic" mirror. A more common example (redirected from http://ftpmirror.gnu.org):

curl http://mirror.checkdomain.de/gnu/hello/hello-2.10.tar.gz -v -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 46.4.50.2...
* TCP_NODELAY set
* Connected to mirror.checkdomain.de (46.4.50.2) port 80 (#0)
> GET /gnu/hello/hello-2.10.tar.gz HTTP/1.1
> Host: mirror.checkdomain.de
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 15 Sep 2017 17:47:25 GMT
< Server: Apache/2.2.22
< Last-Modified: Sun, 16 Nov 2014 12:08:45 GMT
< ETag: "18d0e9c-b13ba-507f8b942cd40"
< Accept-Ranges: bytes
< Content-Length: 725946
< Content-Type: application/x-gzip
< Content-Encoding: x-gzip
<
{ [1100 bytes data]
100  708k  100  708k    0     0   708k      0  0:00:01  0:00:01 --:--:--  675k
* Connection #0 to host mirror.checkdomain.de left intact

[edolstra]

This is a server configuration error. The HTTP standard says about Content-Encoding:

   The "Content-Encoding" header field indicates what content codings
   have been applied to the representation, beyond those inherent in the
   media type, and thus what decoding mechanisms have to be applied in
   order to obtain data in the media type referenced by the Content-Type
   header field.

Thus the server is saying that we need to remove the x-gzip encoding to obtain the application/x-gzip file.

This is not just a problem for Nix. For example, saving http://mirror.checkdomain.de/gnu/hello/hello-2.10.tar.gz in Chromium will give you an uncompressed tar file.

[edolstra]

Also from the standard:

   If the media type includes an inherent encoding, such as a data
   format that is always compressed, then that encoding would not be
   restated in Content-Encoding even if it happens to be the same
   algorithm as one of the content codings.  Such a content coding would
   only be listed if, for some bizarre reason, it is applied a second
   time to form the representation.