Exceeding maxfilesize once breaks future downloads on same http/2 connection

[Bluewind]

When using pacman (which uses libcurl for downloading) with an HTTP/2 server, further download requests break when the server returns a response that is bigger than the maxfilesize set via CURLOPT_MAXFILESIZE_LARGE. libcurl returns the error to pacman as seen in the log below (look for "Maximum file size exceeded"), but when pacman tries to download another file over the same connection, the connection doesn't appear to work correctly any more. The curl docs say that applications should continue to work unmodified so I'm posting this issue here before contacting the pacman developers.

Edit: The filesize exceed error is expected (kind of). Arch Linux does not sign their databases, but pacman supports signed databases and so tries to download a signature file. Since databases are not signed, the signature does not exist and a 404 error is returned. In case of one particular server, the 404 error page is bigger than our max file size for signatures and thus triggers this error.

When the server only supports HTTP/1.1, pacman works correctly.
When trying to reproduce the problem with curl --max-filesize 16384 -sv https://mirror.ams1.nl.leaseweb.net/archlinux/staging/os/x86_64/staging.db{.sig,} -o /dev/null I can't seem to trigger the same issue.

You can view pacman's download code here. I've modified it locally to enable CURLOPT_VERBOSE to get the output shown below.

curl/libcurl version

curl 7.62.0 (x86_64-pc-linux-gnu) libcurl/7.62.0 OpenSSL/1.1.1a zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.4) libssh2/1.8.0 nghttp2/1.34.0
Release-Date: 2018-10-31
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

operating system

Arch Linux

:: Synchronizing package databases...
[10:31:38] debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/staging/os/x86_64/staging.db
[10:31:38] debug: maxsize: 26214400
[10:31:38] debug: opened tempfile for download: /var/lib/pacman/sync/staging.db.part (wb)
* Couldn't find host mirror.ams1.nl.leaseweb.net in the .netrc file; using defaults
*   Trying 5.79.108.33...
* TCP_NODELAY set
* Connected to mirror.ams1.nl.leaseweb.net (5.79.108.33) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* NPN, negotiated HTTP2 (h2)
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=mirror.leaseweb.com
*  start date: Nov  2 09:36:03 2018 GMT
*  expire date: Jan 31 09:36:03 2019 GMT
*  subjectAltName: host "mirror.ams1.nl.leaseweb.net" matched cert's "mirror.ams1.nl.leaseweb.net"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557913c7c5d0)
> GET /archlinux/staging/os/x86_64/staging.db HTTP/2
Host: mirror.ams1.nl.leaseweb.net
User-Agent: pacman/5.1.0 (Linux x86_64) libalpm/5.1.0
Accept: */*

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx/1.12.2
< date: Wed, 19 Dec 2018 09:31:38 GMT
< content-type: application/octet-stream
< content-length: 29
< last-modified: Tue, 18 Dec 2018 01:59:22 GMT
< etag: "5c18547a-1d"
< strict-transport-security: max-age=31536000
< x-frame-options: DENY
< x-content-type-options: nosniff
< accept-ranges: bytes
< 
downloading staging.db...
* Connection #0 to host mirror.ams1.nl.leaseweb.net left intact
[10:31:38] debug: curl returned error 0 from transfer
[10:31:38] debug: response code: 200
[10:31:38] debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/staging/os/x86_64/staging.db.sig
[10:31:38] debug: maxsize: 16384
[10:31:38] debug: opened tempfile for download: /var/lib/pacman/sync/staging.db.sig.part (wb)
* Couldn't find host mirror.ams1.nl.leaseweb.net in the .netrc file; using defaults
* Found bundle for host mirror.ams1.nl.leaseweb.net: 0x557913c79bf0 [can multiplex]
* Re-using existing connection! (#0) with host mirror.ams1.nl.leaseweb.net
* Connected to mirror.ams1.nl.leaseweb.net (5.79.108.33) port 443 (#0)
* Using Stream ID: 3 (easy handle 0x557913c7c5d0)
> GET /archlinux/staging/os/x86_64/staging.db.sig HTTP/2
Host: mirror.ams1.nl.leaseweb.net
User-Agent: pacman/5.1.0 (Linux x86_64) libalpm/5.1.0
Accept: */*

< HTTP/2 404 
< server: nginx/1.12.2
< date: Wed, 19 Dec 2018 09:31:38 GMT
< content-type: text/html
* Maximum file size exceeded
* Connection #0 to host mirror.ams1.nl.leaseweb.net left intact
[10:31:38] debug: curl returned error 63 from transfer
[10:31:38] debug: failed retrieving file 'staging.db.sig' from mirror.ams1.nl.leaseweb.net : Maximum file size exceeded
[10:31:38] debug: "/var/lib/pacman/sync/staging.db.sig" is not readable: No such file or directory
[10:31:38] debug: sig path /var/lib/pacman/sync/staging.db.sig could not be opened
[10:31:38] debug: missing optional signature
[10:31:38] debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/multilib-staging/os/x86_64/multilib-staging.db
[10:31:38] debug: maxsize: 26214400
[10:31:38] debug: opened tempfile for download: /var/lib/pacman/sync/multilib-staging.db.part (wb)
* Couldn't find host mirror.ams1.nl.leaseweb.net in the .netrc file; using defaults
* Found bundle for host mirror.ams1.nl.leaseweb.net: 0x557913c79bf0 [can multiplex]
* 1876 bytes stray data read before trying h2 connection
* Re-using existing connection! (#0) with host mirror.ams1.nl.leaseweb.net
* Connected to mirror.ams1.nl.leaseweb.net (5.79.108.33) port 443 (#0)
* Using Stream ID: 5 (easy handle 0x557913c7c5d0)
> GET /archlinux/multilib-staging/os/x86_64/multilib-staging.db HTTP/2
Host: mirror.ams1.nl.leaseweb.net
User-Agent: pacman/5.1.0 (Linux x86_64) libalpm/5.1.0
Accept: */*

* Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
* Connection #0 to host mirror.ams1.nl.leaseweb.net left intact
[10:31:48] debug: curl returned error 28 from transfer
[10:31:48] error: failed retrieving file 'multilib-staging.db' from mirror.ams1.nl.leaseweb.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
[10:31:48] debug: failed to sync db: download library error
[10:31:48] error: failed to update multilib-staging (download library error)
[10:31:48] debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/community-staging/os/x86_64/community-staging.db
[10:31:48] debug: maxsize: 26214400
[10:31:48] debug: opened tempfile for download: /var/lib/pacman/sync/community-staging.db.part (wb)
* Couldn't find host mirror.ams1.nl.leaseweb.net in the .netrc file; using defaults
* 1156 bytes stray data read before trying h2 connection
* Found bundle for host mirror.ams1.nl.leaseweb.net: 0x557913c79bf0 [can multiplex]
* Re-using existing connection! (#0) with host mirror.ams1.nl.leaseweb.net
* Connected to mirror.ams1.nl.leaseweb.net (5.79.108.33) port 443 (#0)
* Using Stream ID: 7 (easy handle 0x557913c7c5d0)
> GET /archlinux/community-staging/os/x86_64/community-staging.db HTTP/2
Host: mirror.ams1.nl.leaseweb.net
User-Agent: pacman/5.1.0 (Linux x86_64) libalpm/5.1.0
Accept: */*

[falconindy]

Here's a much smaller standalone program that reproduces the problem:

#include <curl/curl.h>

int main(void) {
  curl_global_init(CURL_GLOBAL_ALL);

  CURL* curl = curl_easy_init();

  FILE* devnull = fopen("/dev/null", "w");

  curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
  curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L);
  curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 5L);

  /* remote returns 200 */
  curl_easy_setopt(curl, CURLOPT_URL, "https://mirror.ams1.nl.leaseweb.net/archlinux/core/os/x86_64/core.db");
  curl_easy_setopt(curl, CURLOPT_WRITEDATA, devnull);
  curl_easy_perform(curl);

  /* remote returns 404, but response body exceeds max filesize */
  curl_easy_setopt(curl, CURLOPT_URL, "https://mirror.ams1.nl.leaseweb.net/archlinux/core/os/x86_64/core.db.sig");
  curl_easy_setopt(curl, CURLOPT_MAXFILESIZE_LARGE, 16384L);
  curl_easy_perform(curl);

  /* this hangs, eventually times out due to low speed limit */
  curl_easy_setopt(curl, CURLOPT_URL, "https://mirror.ams1.nl.leaseweb.net/archlinux/extra/os/x86_64/extra.db");
  curl_easy_perform(curl);

  /* this hangs, eventually times out due to low speed limit */
  curl_easy_setopt(curl, CURLOPT_URL, "https://mirror.ams1.nl.leaseweb.net/archlinux/extra/os/x86_64/extra.db.sig");
  curl_easy_perform(curl);

  curl_easy_cleanup(curl);

  fclose(devnull);

  return 0;
}

Amending this program to include:

curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

...makes the program work as expected.

[bagder]

With the patch in #3399 it seems the problem is fixed for me. Would appreciate if you verify how it behaves for you!

[Bluewind]

#3399 fixes the problem with pacman for me too. Thanks!