Hyper slowness issue

[gvanem]

I did this

Trying to transfer 3 URLs like:

https://api.adsb.lol/api/0/route/SAS4133 
https://api.adsb.lol/api/0/route/BBD6810
https://api.adsb.lol/api/0/route/THY19Y

to get Airplane flight-information (in series or parallel), causes curl to hang for a really long time between each URL.
But only when built with -DUSE_HYPER !? Entering the URLs either from a config-file or set on the command-line doesn't matter.
A simple get-urls.bat file to reproduce:

@echo off
setlocal
set CURL_SSL_BACKEND=schannel
echo url=https://api.adsb.lol/api/0/route/SAS4133  > %TEMP%\curl_urls.txt
echo url=https://api.adsb.lol/api/0/route/BBD6810 >> %TEMP%\curl_urls.txt
echo url=https://api.adsb.lol/api/0/route/THY19Y  >> %TEMP%\curl_urls.txt
curl -s -K - < %TEMP%\curl_urls.txt | grep "_airport_codes_iata"

Running the above .bat file takes at least 1 minute to complete! With --parallel or not.

I expected the following

The transfer to complete in < 1 sec. as the curl.exe bundled with Win-10 does:

timer & c:\windows\system32\curl.exe -s https://api.adsb.lol/api/0/route/SAS4133 https://api.adsb.lol/api/0/route/BBD6810 https://api.adsb.lol/api/0/route/THY19Y | grep "_airport_codes_iata"  & timer
Timer 1 on:  9:42:26
  "_airport_codes_iata": "CGN-BLL-KEF",
  "_airport_codes_iata": "SFO-IST",
  "_airport_codes_iata": "BGO-SVG",
Timer 1 off:  9:42:26  Elapsed: 0:00:00,45

c:\windows\system32\curl -V
curl 8.0.1 (Windows) libcurl/8.0.1 Schannel WinIDN
Release-Date: 2023-03-20

Adding the --parallel option I'd assume the URLs would be issued in parallel. Doesn't look so to me.

Version info:

curl -V
curl 8.2.0-DEV (x86_64-pc-win32) libcurl/8.2.0-DEV (OpenSSL/3.2.0) Schannel 
  (rustls-ffi/0.10.0/rustls/0.21.0) zlib/1.2.12 brotli/1.0.9 zstd/1.5.2 c-ares/1.19.0 libssh2/1.10.1_DEV Hyper/1.0.0-rc.3
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s scp sftp smb smbs smtp 
  smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe 
  TLS-SRP UnixSockets zstd

I'm on Windows-10, 22H2 (OS-build 19045.2965).

[gvanem]

After updating Rust with:

  rustup update
  rustup toolchain install nightly-x86_64-pc-windows-msvc
  rustup override set nightly-x86_64-pc-windows-msvc

a git pull of Hyper's master and rebuilding everything, this issue persists.

[bagder]

Reproduces for me as well on Linux.

[icing]

Added some debug info's to this. I see that in Curl_hyper_stream()which is called by curl's transfer loop:

1. socket gives POLLIN event
2. Curl_hyper_stream() is called
3. hyper_waker_wake(h->read_waker) is called
4. socket input reads the HTTP response
5. hyper_executor_poll(h->exec); gives HYPER_TASK_EMPTY
6. Curl_hyper_stream() returns
7. time passes
8. transfer loop calls this transfer again after 100ms or more
9. no POLL event, no hyper_wake..., NO network IO
10. hyper_executor_poll(h->exec); gives HYPER_TASK_RESPONSE

this explains the "slowness" of handling hyper transfers.

Experimentally, if I change step 5 to call hyper_executor_poll(h->exec); a second time, directly after the EMPTY result, it delivers HYPER_TASK_RESPONSE and the transfers are fast.

This seem to be y hyper internal synchronization issue that make IO result not available though they have been processed internally. Only when calling the hyper poll function again, the result is returned.

[gvanem]

@icing

Only when calling the hyper poll function again, the result is returned.

Hope someone (you?) is working on a fix. I'm clueless about such internal details.

[icing]

@gvanem you are right. I wanted to document my observations to drag the hyper project in. I think this is a bug on their behalf, but if the insist everything is fine, we'll make a workaround in curl.

[icing]

Opened hyperium/hyper#3237

[bagder]

We also have #10803 as known bug now. It is a memory-leak with hyper as a backend.

[icing]

On advisement of @seanmonstar I made some small changes that resolve the slowness for me. Pr in #11344.

I see our http/1.1 test cases work, h2 has issues with connection reuse.

[bagder]

Does not fix it for @gvanem apparently

[nnethercote]

I can replicate the slowness on Linux. I used this script:

#! /bin/sh
curl -s -K - <<EOF | grep "_airport_codes_iata"
url=https://api.adsb.lol/api/0/route/SAS4133
url=https://api.adsb.lol/api/0/route/BBD6810
url=https://api.adsb.lol/api/0/route/THY19Y
EOF

With normal curl it takes 1-2 seconds to complete. With curl+hyper it takes about a minute; the first result comes immediately, then there is a pause of about 30 seconds before the second result, then another pause of about 30 seconds before the third result.

[nnethercote]

We also have #10803 as known bug now. It is a memory-leak with hyper as a backend.

#10803 has now been fixed. It is unrelated to this issue.

[gvanem]

With everything up-to-date, .bat file above now shows Hyper is 20 sec slower than before:

Using 'c:\Windows\system32\curl.exe':
  "_airport_codes_iata": "BGO-SVG",
  "_airport_codes_iata": "CGN-BLL-KEF",
  "_airport_codes_iata": "SFO-IST",
Timer 1 off: 13:01:11  Elapsed: 0:00:01,515

Using 'curl -- tip of git master':
  "_airport_codes_iata": "CGN-BLL-KEF",
  "_airport_codes_iata": "SFO-IST",
Timer 1 off: 13:03:58  Elapsed: 0:01:01,321

Besides the response of first URL got dropped in some cases!

[nnethercote]

Some notes from my investigations:

The first response arrives quickly. After that, logging shows that Curl_hyper_stream is called once per second for 30 seconds with select_res of 0. Every time hyper_executor_poll returns NULL and so Curl_hyper_stream returns without doing anything other than setting *didwhat = KEEP_RECV. Finally after 30 calls, Curl_hyper_stream is called with select_res of 1, and then things start happening. And it's a similar story for the third response.

I grepped for 30 and found that the -y/--speed-time option affects the behaviour here. If I use -y 5 then the 30 second loop is reduced to 5 seconds, but the middle response doesn't arrive, and the whole thing takes a little over 5 seconds. In fact, any -y value less than 30 seems to cause this drop-the-middle-response behaviour.

The 1 second delay comes from a Curl_poll call with a timeout of 1000ms in multi_wait.

multi_wait calls multi_timeout, to possibly adjust the timeout down from 1000ms. Normal curl frequently adjusts this down to very small numbers: 0, 1, etc. curl+hyper never does, because it doesn't have multi->timetree.

Questions:

• Why does vanilla curl end up with a non-zero select_res (from Curl_socket_check) so much more quickly than curl+hyper?
• And what changes at the 30 second mark for curl+hyper?

[bagder]

When the 30 second hang occurs, curl wants to send off the (second) request over the connection. But due to hyper's asynch API and behavior (combined with some bug in curl obviously) curl does not know it is still sending the request but it has already moved on to the PERFORMING state and since this is a plain GET, it does not wait for the socket to be writable and thus it sits there waiting without sending the request.

(Presumably the socket gets readable after 30 seconds anyway for some reason and that triggers the send to get sent.)

This is can be seen by multi_getsock() getting called and the handle being in state MSTATE_PERFORMING before the request has been sent. The native HTTP will issue the request in the DO or DOING states.

[bagder]

My adjusted reproduce script with tracing + time stamps

#! /bin/sh
./src/curl --trace-ascii - --trace-time -K - <<EOF | tee dump
url=https://api.adsb.lol/api/0/route/SAS4133
url=https://api.adsb.lol/api/0/route/BBD6810
url=https://api.adsb.lol/api/0/route/THY19Y
EOF

[nnethercote]

Vanilla curl is using http/1.1 while curl+hyper is using h2. Is that significant?

[nnethercote]

Also, the 30 second thing might be due to an SSL timeout, which trigger a new connection?

[bagder]

Vanilla curl is using http/1.1 while curl+hyper is using h2. Is that significant?

My vanilla build uses h2 for this. But no, that does not matter.

Also, the 30 second thing might be due to an SSL timeout, which trigger a new connection?

No, the trace shows no new connection. And a curl will not silently do a new connection if a transfer fails in the middle, it would return an error.

[gvanem]

There's also an issue with Hyper + GnuTls as the backend and possibly -DCURLDEBUG.
E.g. curl.exe --tlsv1.2 -v https://mas-oyama.com/ gives:

...
* Curl_hyper_send(24)
* Curl_hyper_send(24) -> 24
> GET / HTTP/2
> Host: mas-oyama.com
> Authorization: Basic YW5vbnltb3VzOnlvdXJAZW1haWwuYWRkcmVzcw==
> User-Agent: GoogleBot/2.1
> Accept: */*
>
* STATE: DO => DID handle 0x29ebeb074b8; line 2233
* STATE: DID => PERFORMING handle 0x29ebeb074b8; line 2351
* Curl_hyper_recv(8192)
Assertion failed: (size_t)nread <= len, file vtls/vtls.c, line 1590

[icing]

@gvanem can you inspect/dump the values of that assertion failure?

[gvanem]

Adding a DEBUGF() before that Assertion and running curl.exe --tlsv1.2 -v https://mas-oyama.com/:

...
* schannel: recv returned CURLE_AGAIN
* schannel: encrypted data buffer: offset 0 length 9216
* schannel: encrypted data buffer: offset 0 length 9216
* schannel: decrypted data buffer: offset 0 length 8192
* schannel: schannel_recv cleanup
* nread: 18446744073709551615, len: 8152
Assertion failed: (size_t)nread <= len, file vtls/vtls.c, line 1591

[icing]

hmm, a 64bit -1 as unsigned. but the var nread is an ssize_t. How could that happen in:

  ssize_t nread;

  CF_DATA_SAVE(save, cf, data);
  *err = CURLE_OK;
  nread = Curl_ssl->recv_plain(cf, data, buf, len, err);
  if(nread > 0) {
    DEBUGASSERT((size_t)nread <= len);
  }
  ...

[gvanem]

Don't ask, but it seems some Rust code inside Hyper caused a panic.

[marlonbaeten]

We have looked into this issue and have found the potential problem.

I can reproduce the issue with hyper + Rustls / OpenSSL and various endpoints. Some appeared to be somewhat fast and some slow (> 60 seconds). For example:

# this takes > 60 seconds
curl https://api.adsb.lol/api/0/route/SAS4133 https://api.adsb.lol/api/0/route/BBD6810 https://api.adsb.lol/api/0/route/THY19Y
# this takes 300 ms
curl --http1.1 https://api.adsb.lol/api/0/route/SAS4133 https://api.adsb.lol/api/0/route/BBD6810 https://api.adsb.lol/api/0/route/THY19Y
# this takes around 4 seconds
curl https://github.com/curl/curl/issues/11203 https://github.com/curl/curl/issues/10803 https://github.com/curl/curl/pull/11729

So the problem appears to only occur with HTTP2 and some servers cause a delay of 30 seconds for the 2nd and 3rd request. I have confirmed this by testing several of different endpoints.

Inspecting the HTTP2 traffic with wireshark (and a SSLKEYLOGFILE) I noticed that curl with the default backend sends over:

• A HTTP2 connection preface
• Settings / window update
• Request 1
• Request 2
• Request 3

Curl with hyper sends over:

• A HTTP2 connection preface
• Settings / window update
• Request 1
• A HTTP2 connection preface
• Settings / window update
• Request 2
• A HTTP2 connection preface
• Settings / window update
• Request 3

All web servers I tested cannot handle the second / third HTTP2 connection preface / Settings / window update (i think this is called a "handshake" in c-hyper.c). In these cases curl+hyper send the second GET request, but the server will, either immediately or after a delay of 30 seconds, send a GOAWAY after which curl+hyper sets up a new connection and tries again (which will succeed). For instance, github will send the GOAWAY immediately after the second HTTP2 connection preface / settings is sent. This is why the third example above only takes 4 seconds.

The hyper API is constructed in such a way that this "handshake" is performed every time a new hyper client is constructed. So my first plan to fix this was reusing the hyper client in c-hyper.c, however this causes some problems since the executor (that is bound to the hyper client) is recreated for every request.

Would you see any problems in reusing the executor / client for multiple requests?

[seanmonstar]

Would you see any problems in reusing the executor / client for multiple requests?

At least from hyper's point of view, they are meant to be reusable.

[bagder]

Would you see any problems in reusing the executor / client for multiple requests?

No. The easy handles are already encouraged to be reused in the public libcurl API. If we can reuse more parts within it, we should.

[marlonbaeten]

Would you see any problems in reusing the executor / client for multiple requests?

No. The easy handles are already encouraged to be reused in the public libcurl API. If we can reuse more parts within it, we should.

I have looked into persisting the hyptransfer struct between requests, but I can't pinpoint the appropriate place to do this. Do you have any suggestions?

[bagder]

The hypertransfer struct itself persists between requests since it is a member of the struct Curl_easy struct, which is the CURL * handle's internal version.

Curl_http_done() gets called when the request is "done" (reaches DONE state) and that then calls Curl_hyper_done() which currently seems to free most/all hyper related resources. I presume it should keep some of them around longer and instead rather close them later. Maybe in Curl_close() which is the function that kills and frees the handle.

Note that this handle is the handle for a transfer, but not for data that associated with the connections. They are in the struct connectdata as a single transfer very well may involve one or more connections and connections can survive and be used by other transfers.

[folkertdev]

is struct curl_easy the right place for the executor then? it is most efficient to only create the executor once, and then reuse it for any request that curl makes while it runs. So maybe it should instead live in struct connectdata?

---

The hypertransfer struct itself persists between requests since it is a member of the struct Curl_easy struct, which is the CURL * handle's internal version.

Yes, but somehow the executor gets cleared and we can't figure out why. This patch

• disables a memset that would clear the pointer to the executor
• disables the freeing of the executor

diff --git a/lib/c-hyper.c b/lib/c-hyper.c
index 61ca29a3a..d72c2c575 100644
--- a/lib/c-hyper.c
+++ b/lib/c-hyper.c
@@ -912,8 +912,11 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
      the rest of the request in the PERFORM phase. */
   *done = TRUE;
 
+  infof(data, "-------------------------> the hyptransfer pointer is %p", h);
+  infof(data, "-------------------------> the executor pointer is %p", h->exec);
+
   infof(data, "Time for the Hyper dance");
-  memset(h, 0, sizeof(struct hyptransfer));
+  // memset(h, 0, sizeof(struct hyptransfer));
 
   result = Curl_http_host(data, conn);
   if(result)
@@ -1261,8 +1264,8 @@ void Curl_hyper_done(struct Curl_easy *data)
 {
   struct hyptransfer *h = &data->hyp;
   if(h->exec) {
-    hyper_executor_free(h->exec);
-    h->exec = NULL;
+    // hyper_executor_free(h->exec);
+    // h->exec = NULL;
   }
   if(h->read_waker) {
     hyper_waker_free(h->read_waker);

we see this output

*   Trying 104.26.5.74:443...
* Connected to api.adsb.lol (104.26.5.74) port 443
* ALPN: curl offers h2,http/1.1
* rustls_connection wants us to write_tls.
* Curl_socket_check: reading would block
* rustls_connection wants us to read_tls.
* rustls_connection wants us to write_tls.
* rustls_connection wants us to read_tls.
* Curl_socket_check: reading would block
* rustls_connection wants us to read_tls.
* Done handshaking
* ALPN: server accepted h2
* -------------------------> the hyptransfer pointer is 0x55b69b469a18
* -------------------------> the executor pointer is (nil)
* Time for the Hyper dance
> GET /api/0/route/SAS4133 HTTP/2
> Host: api.adsb.lol
> User-Agent: curl/8.4.0-DEV
> Accept: */*
> 
< HTTP/2 200 OK
< date: Tue, 19 Sep 2023 17:00:45 GMT
< content-type: application/json
< content-length: 659
< access-control-allow-origin: *
< access-control-allow-methods: POST, OPTIONS
< access-control-allow-headers: access-control-allow-origin,content-type
< strict-transport-security: max-age=31536000; includeSubDomains
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=1lneCHWqONSQxNCcJMd5cO9gX7h2SxL01tWW28YlzR7E5HYGTBSAfpW46NhHWrDQ2w%2ForS4K%2Fj6hgTMQCu2DpVwsDgMz58JsinMUg199oTCTxrq1U9QEBl6JaeSKNA%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 809362fd1b0efa40-AMS
< alt-svc: h3=":443"; ma=86400
< 
<http response body>
* hyperstream is done
* Connection #0 to host api.adsb.lol left intact
* Found bundle for host: 0x55b69b4648e0 [can multiplex]
* Re-using existing connection with host api.adsb.lol
* -------------------------> the hyptransfer pointer is 0x55b69b469a18
* -------------------------> the executor pointer is (nil)
* Time for the Hyper dance
> GET /api/0/route/BBD6810 HTTP/2
> Host: api.adsb.lol
> User-Agent: c
...

so hyptransfer is the same pointer, but the executor pointer does get reset. We haven't been able to figure out why this happens.

[bagder]

is struct curl_easy the right place for the executor then? it is most efficient to only create the executor once, and then reuse it for any request that curl makes while it runs. So maybe it should instead live in struct connectdata?

The executor is independent of the connection, right? Then I think the Curl_easy sounds like exactly the right place for it to be.

the executor pointer does get reset

I can find two places where curl does it:

$ git grep -n 'h->exec = NULL'
lib/c-hyper.c:1265:    h->exec = NULL;
lib/cf-h1-proxy.c:947:    h->exec = NULL;

[folkertdev]

those are the obvious places, but

• the first one is commented out by the patch I included. So it does not run
• the second one does not get run anyway (maybe because we do http2?)

so just a simple grep does not answer the question of why that field is reset. So I tried some fancier debugging by watching the memory address of the executor in the hyptransfer struct (so &h->exec).

* -----------------> the hyptransfer pointer is 0x5555555e2f20
916	  infof(data, "-----------------> the executor pointer field is %p", &h->exec);
(gdb) 
* -----------------> the executor pointer field is 0x5555555e2f30
918	  infof(data, "Time for the Hyper dance");
(gdb) watch *(long **)0x5555555e2f30
Hardware watchpoint 3: *(long **)0x5555555e2f30  # we just need to provide any pointer type here
(gdb) c
Continuing.
* Time for the Hyper dance

Thread 1 "curl" hit Hardware watchpoint 2: *0x5555555e2f30

Old value = 0
New value = 1432576368

Thread 1 "curl" hit Hardware watchpoint 3: *(long **)0x5555555e2f30

Old value = (long *) 0x0
New value = (long *) 0x555555636170
Curl_http (data=0x5555555e1a78, done=0x7fffffffd99e) at c-hyper.c:968
968	    if(!h->exec) {
(gdb) 

So far it kind of makes sense. The line number seems to be off-by-one though, the actual write happens a line earlier. So now if this field were persisted, we'd expect no further changes to it until all requests have completed, right? Well that is not what I see

* hyperstream is done
* STATE: PERFORMING => DONE handle 0x5555555e1a78; line 2550
* multi_done[DONE]: status: 0 prem: 0 done: 0
* Connection #0 to host api.adsb.lol left intact
}
Thread 1 "curl" hit Hardware watchpoint 3: *(long **)0x5555555e2f30

Old value = (long *) 0x555555636170
New value = (long *) 0x555555636100
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151	../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x00007ffff7d03df2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3480
#2  0x00007ffff7f23f2a in curl_dbg_calloc (wanted_elements=1, wanted_size=5328, line=644, source=0x7ffff7f8d840 "url.c") at memdebug.c:175
#3  0x00007ffff7f5e604 in Curl_open (curl=0x7fffffffd930) at url.c:644
#4  0x00007ffff7ef87be in curl_easy_init () at easy.c:379
#5  0x0000555555575eb2 in single_transfer (global=0x7fffffffddd0, config=0x5555555bd2a8, share=0x5555555ded88, capath_from_env=false, 
    added=0x7fffffffdc9d) at tool_operate.c:945
#6  0x000055555557d3db in transfer_per_config (global=0x7fffffffddd0, config=0x5555555bd2a8, share=0x5555555ded88, added=0x7fffffffdc9d)
    at tool_operate.c:2629
#7  0x000055555557d43b in create_transfer (global=0x7fffffffddd0, share=0x5555555ded88, added=0x7fffffffdc9d) at tool_operate.c:2645
#8  0x000055555557cf66 in serial_transfers (global=0x7fffffffddd0, share=0x5555555ded88) at tool_operate.c:2492
#9  0x000055555557d4f8 in run_all_transfers (global=0x7fffffffddd0, share=0x5555555ded88, result=CURLE_OK) at tool_operate.c:2670
#10 0x000055555557d8ab in operate (global=0x7fffffffddd0, argc=5, argv=0x7fffffffdf48) at tool_operate.c:2786
#11 0x00005555555737b3 in main (argc=5, argv=0x7fffffffdf48) at tool_main.c:274

we hit this place several more times

Old value = (long *) 0x555555636100
New value = (long *) 0x555555630000

Old value = (long *) 0x555555630000
New value = (long *) 0x555555000000

Old value = (long *) 0x555555000000
New value = (long *) 0x555500000000

Old value = (long *) 0x555500000000
New value = (long *) 0x550000000000

Old value = (long *) 0x550000000000
New value = (long *) 0x0

So that just looks like calloc at url.c:644 wiping the memory.

// url.c
/**
 * Curl_open()
 *
 * @param curl is a pointer to a sessionhandle pointer that gets set by this
 * function.
 * @return CURLcode
 */

CURLcode Curl_open(struct Curl_easy **curl)
{
  CURLcode result;
  struct Curl_easy *data;

  /* Very simple start-up: alloc the struct, init it with zeroes and return */
  data = calloc(1, sizeof(struct Curl_easy)); // <- line 644

So, the hypertransfer struct (or its exec field, anyway) just gets wiped after the first request, right? Which suggests that the whole thing has been freed and definitely is not persisted across requests.

But we never see free(data->hyp) in the source code, so how this works is not clear yet.

[bagder]

Curl_open() is the internal version of curl_easy_init(). It creates a new handle so it is not strange that it clears the struct.

So, the hypertransfer struct (or its exec field, anyway) just gets wiped after the first request, right?

Right now, yes, as I said before

But we never see free(data->hyp) in the source code

It is a sub-struct of struct Curl_easy, it is not individually allocated so there cannot be any separate free of it. It gets freed in Curl_close(), the internal version of curl_easy_cleanup().

[marlonbaeten]

@bagder thanks a lot for your help!

I was able to persist the hyper client between requests by making it part of the connectdata, which makes sense for HTTP/2, since the connection configuration should happen once per connection.

However I encountered a potential issue in the hyper C API:

• When using HTTP/1.1 the GET requests are sent immediately after calling hyper_clientconn_send and pushing the resulting task to the executor.
• When using HTTP/2 the GET request is only sent after calling hyper_clientconn_free on the corresponding hyper client.

A minimal way of reproducing this issue is by not-freeing the hyper client and trying to perform a HTTP/2 request with curl:

diff --git a/lib/c-hyper.c b/lib/c-hyper.c
index c558955c9..c85fb5569 100644
--- a/lib/c-hyper.c
+++ b/lib/c-hyper.c
@@ -1218,8 +1218,8 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
   }
   sendtask = NULL; /* ownership passed on */
 
-  hyper_clientconn_free(client);
-  client = NULL;
+  /* hyper_clientconn_free(client);
+  client = NULL; */
 
   if((httpreq == HTTPREQ_GET) || (httpreq == HTTPREQ_HEAD)) {
     /* HTTP GET/HEAD download */

@seanmonstar do you have any idea why the request is not sent if we do not free the client?

[bagder]

I was able to persist the hyper client between requests by making it part of the connectdata, which makes sense for HTTP/2, since the connection configuration should happen once per connection.

I'm not sure I follow this. What struct did you move? hyper does not manage connections for libcurl, so I fail to see how that is important here. We setup all the necessary configs per request in Curl_http() so couldn't the same structs serve multiple requests done for several connections?

• When using HTTP/2 the GET request is only sent after calling hyper_clientconn_free on the corresponding hyper client.

That sounds like a hyper bug to me. At least based on the rust API docs for hyper_clientconn_send

[seanmonstar]

do you have any idea why the request is not sent if we do not free the client?

If that's really the case, that would be a hyper bug. I can't think of anything in hyper that would need to wait until it was freed to send.