Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libstore/filetransfer: set CURLOPT_ACCEPT_ENCODING #4191

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

@puckipedia
Copy link
Contributor

@puckipedia puckipedia commented Oct 27, 2020

This allows Nix to use transport-level compression, where supported.

As documented on https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html, setting an empty string will use libcurl's default.

Also, no tests ran, because nix-build failed:

ran test tests/gc.sh... [FAIL]
    ++ nix-instantiate dependencies.nix
    gc.sh: line 3: nix-instantiate: command not found
    + drvPath=
This allows Nix to use transport-level compression, where supported.
@domenkozar domenkozar requested a review from edolstra Nov 2, 2020
@edolstra
Copy link
Member

@edolstra edolstra commented Nov 2, 2020

This breaks nix log:

$ nix log /nix/store/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv > /dev/null
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 290 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 686 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 1103 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 2812 ms
error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding)

@edolstra
Copy link
Member

@edolstra edolstra commented Nov 2, 2020

That file is brotli-compressed BTW. Curl gives the following warning:

curl: Unrecognized content encoding type. libcurl understands deflate, gzip content encodings.

Note that FileTransfer already does its own decompression so Accept-Encoding may interfere with that.

@puckipedia
Copy link
Contributor Author

@puckipedia puckipedia commented Nov 2, 2020

This is caused by a misconfiguration of the object's properties:

content-encoding: br
content-type: text/plain; charset=utf-8

So curl's behavior is correct here.

@puckipedia
Copy link
Contributor Author

@puckipedia puckipedia commented Nov 2, 2020

Or, well. Technically it is correct, and modern browsers will accept it, but older curl doesn't understand it (7.57.0 and up do br natively). It'd be possible to set Accept-Encoding manually, or just upgrade curl and drop the FileTransfer decompression altogether.

@edolstra
Copy link
Member

@edolstra edolstra commented Nov 2, 2020

How is it misconfigured? The file is encoded using Brotli.

@puckipedia
Copy link
Contributor Author

@puckipedia puckipedia commented Nov 2, 2020

See my reply immediately after. I misparsed it as the nix log files being inherently encoded as brotli, since i've not seen them in another format otherwise.

@stale
Copy link

@stale stale bot commented Jun 2, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Jun 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants