Merge branch 'curl:master' into sftp_sha256

RELEASE-NOTES

@@ -4,7 +4,7 @@ curl and libcurl 7.79.0
  Command line options:         242
  curl_easy_setopt() options:   290
  Public functions in libcurl:  85
- Contributors:                 2474
+ Contributors:                 2477
 
 This release includes the following changes:
 
@@ -44,6 +44,8 @@ This release includes the following bugfixes:
  o curl-openssl.m4: show correct output for OpenSSL v3 [75]
  o curl.1: mention "global" flags [7]
  o curl: add warning for ignored data after quoted form parameter [60]
+ o curl: better error message when -O fails to get a good name [88]
+ o curl_easy_setopt.3: improve the string copy wording [89]
  o curl_setup.h: sync values for HTTP_ONLY [82]
  o curl_url_get.3: clarify about path and query [45]
  o CURLMOPT_TIMERFUNCTION.3: remove misplaced "time" [5]
@@ -57,11 +59,15 @@ This release includes the following bugfixes:
  o getparameter: fix the --local-port number parser [58]
  o happy-eyeballs-timeout-ms.d: polish the wording [10]
  o hostip: Make Curl_ipv6works function independent of getaddrinfo [26]
+ o http2: revert a change that broke upgrade to h2c [57]
  o http2: revert call the handle-closed function correctly on closed stream [25]
+ o http: disallow >3-digit response codes [80]
  o http_proxy: clear 'sending' when the outgoing request is sent [6]
  o http_proxy: fix user-agent and custom headers for CONNECT with hyper [38]
+ o http_proxy: only wait for writable socket while sending request [78]
  o libcurl-errors.3: clarify two CURLUcode errors [72]
  o limit-rate.d: clarify base unit [17]
+ o mailing lists: move from cool.haxx.se to lists.haxx.se
  o mbedTLS: initial 3.0.0 support [33]
  o mbedtls_threadlock: fix unused variable warning [11]
  o mksymbolsmanpage.pl: Fix showing symbol's last used version [76]
@@ -71,6 +77,7 @@ This release includes the following bugfixes:
  o ngtcp2: replace deprecated functions with nghttp3_conn_shutdown_stream_read [28]
  o ngtcp2: reset the oustanding send buffer again when drained [53]
  o ngtcp2: rework the return value handling of ngtcp2_conn_writev_stream [29]
+ o ngtcp2: stop buffering crypto data [85]
  o ngtcp2: utilize crypto API functions to simplify [52]
  o openssl: when creating a new context, there cannot be an old one [48]
  o output.d: add method to suppress response bodies [49]
@@ -81,12 +88,14 @@ This release includes the following bugfixes:
  o setopt: enable CURLOPT_IGNORE_CONTENT_LENGTH for hyper [70]
  o symbols-in-versions: fix CURLSSLBACKEND_QSOSSL last used version [73]
  o test1138: remove trailing space to make work with hyper [71]
+ o test1280: CRLFify the response to please hyper [86]
  o test1565: fix windows build errors [27]
  o tests/*server.pl: flush output before executing subprocess [41]
  o tests/*server.py: remove pidfile on server termination [1]
  o tests/runtests.pl: cleanup copy&paste mistakes and unused code
  o tests/server/*.c: align handling of portfile argument and file [56]
  o tests: be explicit about using 'python3' instead of 'python' [67]
+ o tests: enable test 1129 for hyper builds [87]
  o tests: make three tests pass until 2037 [22]
  o tool/tests: fix potential year 2038 issues [20]
  o tool_operate: Fix --fail-early with parallel transfers [62]
@@ -109,12 +118,13 @@ advice from friends like these:
   Carlo Marcelo Arenas Belón, Christian Weisgerber, Colin O'Dell, Dan Fandrich,
   Daniel Gustafsson, Daniel Stenberg, Daniel Woelfel, Dan Jacobson, David Cook,
   Don J Olmstead, Ehren Bendler, Emil Engler, Gambit Communications,
-  Gergely Nagy, Gleb Ivanovsky, Ikko Ashimine, Jan Verbeek, Jeff Mears,
-  Jeremy Falcon, Josh Soref, Marc Hörsken, Michael Kaufmann, Michał Antoniak,
-  modbw on github, Oleg Pudeyev, Patrick Monnerat, Randall S. Becker,
-  Ray Satiro, Rui Pinheiro, Sergey Markelov, Tatsuhiro Tsujikawa,
-  Viktor Szakats, Vincent Grande, z2-2z on github, zloi-user on github,
-  (47 contributors)
+  Gergely Nagy, Gleb Ivanovsky, Ikko Ashimine, Jan Schaumann, Jan Verbeek,
+  Jeff Mears, Jeremy Falcon, Josh Soref, Marc Hörsken, Michael Kaufmann,
+  Michał Antoniak, modbw on github, Oleg Pudeyev, Oleguer Llopart,
+  Patrick Monnerat, Randall S. Becker, Ray Satiro, Rui Pinheiro,
+  Sergey Markelov, Tatsuhiro Tsujikawa, Tk Xiong, Viktor Szakats,
+  Vincent Grande, Yaobin Wen, z2-2z on github, zloi-user on github,
+  (51 contributors)
 
 References to bug reports and discussions on issues:
 
@@ -174,6 +184,7 @@ References to bug reports and discussions on issues:
  [54] = https://curl.se/bug/?i=7549
  [55] = https://curl.se/bug/?i=7624
  [56] = https://curl.se/bug/?i=7574
+ [57] = https://curl.se/bug/?i=7633
  [58] = https://curl.se/bug/?i=7582
  [59] = https://curl.se/bug/?i=7595
  [60] = https://curl.se/bug/?i=7394
@@ -193,8 +204,15 @@ References to bug reports and discussions on issues:
  [75] = https://curl.se/bug/?i=7606
  [76] = https://github.com/curl/curl/commit/4e53b94#commitcomment-55239509
  [77] = https://github.com/curl/curl/commit/4e53b9430c7504de8984796e2a2091ec16f27136#commitcomment-55239253
+ [78] = https://curl.se/bug/?i=7589
  [79] = https://curl.se/bug/?i=7594
+ [80] = https://curl.se/bug/?i=7641
  [81] = https://curl.se/bug/?i=7603
  [82] = https://curl.se/bug/?i=7601
  [83] = https://curl.se/bug/?i=7581
  [84] = https://curl.se/bug/?i=7580
+ [85] = https://curl.se/bug/?i=7637
+ [86] = https://curl.se/bug/?i=7639
+ [87] = https://curl.se/bug/?i=7638
+ [88] = https://curl.se/bug/?i=7628
+ [89] = https://curl.se/bug/?i=7632

docs/DEPRECATE.md

@@ -2,7 +2,7 @@
 
 If any of these deprecated features is a cause for concern for you, please
 email the
-[curl-library mailing list](https://cool.haxx.se/mailman/listinfo/curl-library)
+[curl-library mailing list](https://lists.haxx.se/listinfo/curl-library)
 as soon as possible and explain to us why this is a problem for you and
 how your use case can't be satisfied properly using a workaround.
 

docs/HELP-US.md

@@ -5,7 +5,7 @@ looking for ways to contribute and help out, this document aims to give a few
 good starting points.
 
 A good idea is to start by subscribing to the [curl-library mailing
-list](https://cool.haxx.se/mailman/listinfo/curl-library) to keep track of the
+list](https://lists.haxx.se/listinfo/curl-library) to keep track of the
 current discussion topics.
 
 ## Scratch your own itch

docs/KNOWN_BUGS

@@ -163,6 +163,7 @@ problems may have been fixed or changed somewhat since this was written!
  18.5 HTTP/3 download with quiche halts after a while
  18.6 HTTP/3 multipart POST with quiche fails
  18.7 HTTP/3 quiche upload large file fails
+ 18.8 HTTP/3 doesn't support client certs
 
 ==============================================================================
 
@@ -1156,3 +1157,9 @@ problems may have been fixed or changed somewhat since this was written!
 18.7 HTTP/3 quiche upload large file fails
 
  https://github.com/curl/curl/issues/7532
+
+18.8 HTTP/3 doesn't support client certs
+
+ aka "mutual authentication".
+
+ https://github.com/curl/curl/issues/7625

docs/examples/multithread.c

@@ -5,7 +5,7 @@
  *                            | (__| |_| |  _ <| |___
  *                             \___|\___/|_| \_\_____|
  *
- * Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
+ * Copyright (C) 1998 - 2021, Daniel Stenberg, <daniel@haxx.se>, et al.
  *
  * This software is licensed as described in the file COPYING, which
  * you should have received as part of this distribution. The terms
@@ -41,9 +41,9 @@
 */
 const char * const urls[NUMT]= {
   "https://curl.se/",
-  "ftp://cool.haxx.se/",
-  "https://www.cag.se/",
-  "www.haxx.se"
+  "ftp://example.com/",
+  "https://example.net/",
+  "www.example"
 };
 
 static void *pull_one_url(void *url)

include/curl/curl.h

@@ -25,9 +25,6 @@
 /*
  * If you have libcurl problems, all docs and details are found here:
  *   https://curl.se/libcurl/
- *
- * curl-library mailing list subscription and unsubscription web interface:
- *   https://cool.haxx.se/mailman/listinfo/curl-library/
  */
 
 #ifdef CURL_NO_OLDIES

lib/c-hyper.c

@@ -400,6 +400,11 @@ CURLcode Curl_hyper_stream(struct Curl_easy *data,
       /* end of transfer */
       *done = TRUE;
       infof(data, "hyperstream is done!");
+      if(!k->bodywrites) {
+        /* hyper doesn't always call the body write callback */
+        bool stilldone;
+        result = Curl_http_firstwrite(data, data->conn, &stilldone);
+      }
       break;
     }
     else if(t != HYPER_TASK_RESPONSE) {

lib/http.c

@@ -2903,6 +2903,20 @@ CURLcode Curl_http_firstwrite(struct Curl_easy *data,
 {
   struct SingleRequest *k = &data->req;
   DEBUGASSERT(conn->handler->protocol&(PROTO_FAMILY_HTTP|CURLPROTO_RTSP));
+  if(data->req.ignore_cl) {
+    k->size = k->maxdownload = -1;
+  }
+  else if(k->size != -1) {
+    /* We wait until after all headers have been received to set this so that
+       we know for sure Content-Length is valid. */
+    if(data->set.max_filesize &&
+       k->size > data->set.max_filesize) {
+      failf(data, "Maximum file size exceeded");
+      return CURLE_FILESIZE_EXCEEDED;
+    }
+    Curl_pgrsSetDownloadSize(data, k->size);
+  }
+
   if(data->req.newurl) {
     if(conn->bits.close) {
       /* Abort after the headers if "follow Location" is set
@@ -3403,17 +3417,8 @@ CURLcode Curl_http_header(struct Curl_easy *data, struct connectdata *conn,
                                     NULL, 10, &contentlength);
 
     if(offt == CURL_OFFT_OK) {
-      if(data->set.max_filesize &&
-         contentlength > data->set.max_filesize) {
-        failf(data, "Maximum file size exceeded");
-        return CURLE_FILESIZE_EXCEEDED;
-      }
       k->size = contentlength;
       k->maxdownload = k->size;
-      /* we set the progress download size already at this point
-         just to make it easier for apps/callbacks to extract this
-         info as soon as possible */
-      Curl_pgrsSetDownloadSize(data, k->size);
     }
     else if(offt == CURL_OFFT_FLOW) {
       /* out of range */
@@ -3504,6 +3509,12 @@ CURLcode Curl_http_header(struct Curl_easy *data, struct connectdata *conn,
                                          TRUE);
     if(result)
       return result;
+    if(!k->chunk) {
+      /* if this isn't chunked, only close can signal the end of this transfer
+         as Content-Length is said not to be trusted for transfer-encoding! */
+      connclose(conn, "HTTP/1.1 transfer-encoding without chunks");
+      k->ignore_cl = TRUE;
+    }
   }
   else if(!k->http_bodyless && checkprefix("Content-Encoding:", headp) &&
           data->set.str[STRING_ENCODING]) {

lib/http2.c

@@ -2336,15 +2336,8 @@ CURLcode Curl_http2_switched(struct Curl_easy *data,
 
   DEBUGASSERT(httpc->nread_inbuf == 0);
 
-  /* Good enough to call it an end once the remaining payload is copied to the
-   * connection buffer.
-   * Some servers (e.g. nghttpx v1.43.0) may fulfill stream 1 immediately
-   * following the protocol switch other than waiting for the client-side
-   * connection preface. If h2_process_pending_input is invoked here to parse
-   * the remaining payload, stream 1 would be marked as closed too early and
-   * thus ignored in http2_recv (following 252790c53).
-   * The logic in lib/http.c and lib/transfer.c guarantees a following
-   * http2_recv would be invoked very soon. */
+  if(-1 == h2_process_pending_input(data, httpc, &result))
+    return CURLE_HTTP2;
 
   return CURLE_OK;
 }

lib/http_proxy.c

@@ -148,7 +148,7 @@ int Curl_connect_getsock(struct connectdata *conn)
   DEBUGASSERT(conn->connect_state);
   http = &conn->connect_state->http_proxy;
 
-  if(http->sending)
+  if(http->sending == HTTPSEND_REQUEST)
     return GETSOCK_WRITESOCK(0);
 
   return GETSOCK_READSOCK(0);

lib/progress.c

@@ -377,7 +377,12 @@ static curl_off_t trspeed(curl_off_t size, /* number of bytes */
 {
   if(us < 1)
     return size * 1000000;
-  return (curl_off_t)((long double)size/(long double)us * 1000000);
+  else if(size < CURL_OFF_T_MAX/1000000)
+    return (size * 1000000) / us;
+  else if(us >= 1000000)
+    return size / (us / 1000000);
+  else
+    return CURL_OFF_T_MAX;
 }
 
 /* returns TRUE if it's time to show the progress meter */

lib/urldata.h

@@ -704,14 +704,15 @@ struct SingleRequest {
 #ifndef CURL_DISABLE_DOH
   struct dohdata *doh; /* DoH specific data for this request */
 #endif
-  BIT(header);       /* incoming data has HTTP header */
+  BIT(header);        /* incoming data has HTTP header */
   BIT(content_range); /* set TRUE if Content-Range: was found */
-  BIT(upload_done);  /* set to TRUE when doing chunked transfer-encoding
-                        upload and we're uploading the last chunk */
-  BIT(ignorebody);   /* we read a response-body but we ignore it! */
+  BIT(upload_done);   /* set to TRUE when doing chunked transfer-encoding
+                         upload and we're uploading the last chunk */
+  BIT(ignorebody);    /* we read a response-body but we ignore it! */
   BIT(http_bodyless); /* HTTP response status code is between 100 and 199,
                          204 or 304 */
-  BIT(chunk); /* if set, this is a chunked transfer-encoding */
+  BIT(chunk);         /* if set, this is a chunked transfer-encoding */
+  BIT(ignore_cl);     /* ignore content-length */
   BIT(upload_chunky); /* set TRUE if we are doing chunked transfer-encoding
                          on upload */
   BIT(getheader);    /* TRUE if header parsing is wanted */

m4/curl-compilers.m4

@@ -64,7 +64,7 @@ AC_DEFUN([CURL_CHECK_COMPILER], [
 ***
 *** If you wish to help the curl project to better support your compiler
 *** you can report this and the required info on the libcurl development
-*** mailing list: https://cool.haxx.se/mailman/listinfo/curl-library/
+*** mailing list: https://lists.haxx.selistinfo/curl-library/
 ***
 _EOF
   fi

tests/data/DISABLED

@@ -44,7 +44,6 @@
 262
 265
 266
-319
 326
 357
 358
@@ -73,15 +72,10 @@
 1071
 1077
 1078
-1122
-1123
-1125
 1144
 1156
 1160
 1164
-1170
-1171
 1176
 1266
 1267

tests/data/Makefile.inc

@@ -60,7 +60,7 @@ test325 test326 test327 test328 test329 test330 test331 test332 test333 \
 test334 test335 test336 test337 test338 test339 test340 test341 test342 \
 test343 test344 test345 test346 test347 test348 test349 test350 test351 \
 test352 test353 test354 test355 test356 test357 test358 test359 test360 \
-test361 test362 test363 test364 \
+test361 test362 test363 test364 test365 \
 \
 test392 test393 test394 test395 test396 test397 \
 \

tests/data/test1122

@@ -11,16 +11,17 @@ Transfer-Encoding
 # Server-side
 <reply>
 <data base64="yes">
-SFRUUC8xLjEgMjAwIE9LDQpEYXRlOiBNb24sIDI5IE5vdiAyMDA0IDIxOjU2OjUzIEdNVA0KU2Vy
-dmVyOiBBcGFjaGUvMS4zLjMxIChEZWJpYW4gR05VL0xpbnV4KSBtb2RfZ3ppcC8xLjMuMjYuMWEg
-UEhQLzQuMy45LTEgbW9kX3NzbC8yLjguMjAgT3BlblNTTC8wLjkuN2QgbW9kX3BlcmwvMS4yOQ0K
-VmFyeTogQWNjZXB0LUVuY29kaW5nDQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD1J
-U08tODg1OS0xDQpUcmFuc2Zlci1FbmNvZGluZzogZ3ppcA0KQ29udGVudC1MZW5ndGg6IDQ0DQoN
-Ch+LCAh5nqtBAANsYWxhbGEAy8nMS1Uw5FLIAdFGXAoQhjEXAAoCcWAYAAAA
+SFRUUC8xLjEgMjAwIE9LIHN3c2Nsb3NlDQpEYXRlOiBNb24sIDI5IE5vdiAyMDA0IDIxOjU2OjUz
+IEdNVA0KU2VydmVyOiBBcGFjaGUvMS4zLjMxIChEZWJpYW4gR05VL0xpbnV4KSBtb2RfZ3ppcC8x
+LjMuMjYuMWEgUEhQLzQuMy45LTEgbW9kX3NzbC8yLjguMjAgT3BlblNTTC8wLjkuN2QgbW9kX3Bl
+cmwvMS4yOQ0KVmFyeTogQWNjZXB0LUVuY29kaW5nDQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsg
+Y2hhcnNldD1JU08tODg1OS0xDQpUcmFuc2Zlci1FbmNvZGluZzogZ3ppcA0KQ29udGVudC1MZW5n
+dGg6IDQ0DQoNCh+LCAh5nqtBAANsYWxhbGEAy8nMS1Uw5FLIAdFGXAoQhjEXAAoCcWAYAAAA
 </data>
 
+# This ignore Content-Length
 <datacheck>
-HTTP/1.1 200 OK
+HTTP/1.1 200 OK swsclose
 Date: Mon, 29 Nov 2004 21:56:53 GMT
 Server: Apache/1.3.31 (Debian GNU/Linux) mod_gzip/1.3.26.1a PHP/4.3.9-1 mod_ssl/2.8.20 OpenSSL/0.9.7d mod_perl/1.29
 Vary: Accept-Encoding