Description
I did this
Currently the Cargo package manager for Rust uses Curl to download files over HTTP, namely dependency libraries. Recently Cargo, on Rust's nightly channel, updated from curl 7.76.0 to curl 7.77.0. Since updating users are reporting an increase in errors seen while downloading.
I've personally been seeing this in a number of CI builds that use Cargo with curl 7.77.0, and the errors as-reported by Cargo is:
error: failed to download from `https://crates.io/api/v1/crates/wat/1.0.38/download`
Caused by:
[55] Failed sending data to the peer (Connection died, tried 5 times before giving up)
I'm able to reproduce this locally on a machine where it just downloads dependencies in a loop, and frequently it will hit this error while downloading. Since this was an apparent regression in Curl I did a bisection of commits between 7.76.0 and 7.77.0 where on each commit I built that version of Curl into Cargo and then downloaded all of Cargo's own dependencies 100 times, where if any download failed that mean the commit was "bad". Bisection resulted in pointing to this commit -- 252790c -- where download would fail spuriously after that but would succeed at least 100 times before that.
Unfortunately reproduction is unlikely to be super easy unless you want to dive into Cargo (which I'm assuming you'd rather not). Cargo uses Curl as a library, so Cargo's usage is pretty far removed from the curl
command line tool for example. In theory Cargo is doing pretty a pretty bland request of "please download all these files over HTTP/2" with some reasonable timeout and retry limits configured. Cargo enables HTTP/2 and pipewait with a maximum of 2 connections per-host at this time.
To hopefully help in assisting with debugging this I captured debug logs from curl during a failed transfer. These debug logs were captured at 252790c with DEBUG_HTTP2
enabled as well.
I'm happy to help test out patches and/or get more debugging information if necessary. I can post reproduction steps as well but it requires using Rust to build Cargo and having a nontrivial setup to thread through a custom Curl dependency into Cargo.
I expected the following
Given the frequency of these errors it seems unlikely that this is caused by an actual spurious error, so the expectation is that something is retried internally as necessary or requeued as necessary without bubbling up an error.
curl/libcurl version
This relates to curl 7.77.0, specifically an apparent regression introduced with 252790c
operating system
The system I tested this on was:
Linux 4.19.0-9-arm64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) aarch64 GNU/Linux
but this seems to appear frequently with GitHub Actions runners as well, and I've seen it locally on my macbook too.