Segmentation fault when sending http3-only request by multi interface

[Cering]

I did this

• Send CURL_HTTP_VERSION_3ONLY request parallelly by using multi interface
• After some requests finished, a segmentation fault occurred
• This segmentation fault occurred when using the latest release version(libcurl/8.1.2+quiche/0.17.1), but not found in previous version(libcurl/8.0.1+quiche/0.16.0)

I expected the following

• Running the test code(see below) in gdb, the segmentation fault info:

Program received signal SIGSEGV, Segmentation fault.
bufcp_take (pchunk=<synthetic pointer>, pool=0xc783a8) at bufq.c:191
191         pool->spare = chunk->next;

(gdb) bt
#0  bufcp_take (pchunk=<synthetic pointer>, pool=0xc783a8) at bufq.c:191
#1  get_spare (q=0xfd67f8) at bufq.c:338
#2  get_non_full_tail (q=q@entry=0xfd67f8) at bufq.c:387
#3  0x00007ffff76818b0 in Curl_bufq_write (q=0xfd67f8, buf=buf@entry=0x7ffff79f19e0 "HTTP/3 ", len=len@entry=7, err=err@entry=0x7fffffffdabc) at bufq.c:412
#4  0x00007ffff76e6adb in write_resp_raw (data=<optimized out>, mem=mem@entry=0x7ffff79f19e0, memlen=memlen@entry=7, cf=<optimized out>) at vquic/curl_quiche.c:356
#5  0x00007ffff76e6bbc in cb_each_header (name=<optimized out>, name_len=<optimized out>, value=0x139ab50 "200", value_len=3, argp=0x7fffffffdb60) at vquic/curl_quiche.c:384
#6  0x00007ffff76fba3c in quiche_h3_event_for_each_header () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#7  0x00007ffff76e6fc8 in h3_process_event (ev=0x1373860, stream3_id=0, data=0x615370, cf=0x125ba00) at vquic/curl_quiche.c:515
#8  cf_poll_events (data=0x615370, cf=0x125ba00) at vquic/curl_quiche.c:591
#9  cf_process_ingress (cf=cf@entry=0x125ba00, data=data@entry=0x615370) at vquic/curl_quiche.c:680
#10 0x00007ffff76e709f in cf_quiche_recv () at vquic/curl_quiche.c:841
#11 0x00007ffff76c1117 in Curl_read (data=data@entry=0x615370, sockfd=<optimized out>, buf=buf@entry=0x610430 "\210\235\337\366\377\177", 
    sizerequested=sizerequested@entry=16384, n=n@entry=0x7fffffffdca0) at sendf.c:415
#12 0x00007ffff76d2950 in readwrite_data (comeback=0x7fffffffdd1f, done=0x7fffffffdd1d, didwhat=<synthetic pointer>, k=0x615440, conn=0x6216f0, data=0x615370)
    at transfer.c:461
#13 Curl_readwrite (conn=0x6216f0, data=data@entry=0x615370, done=done@entry=0x7fffffffdd1d, comeback=comeback@entry=0x7fffffffdd1f) at transfer.c:1115
#14 0x00007ffff76b89e4 in multi_runsingle (multi=multi@entry=0x605010, nowp=nowp@entry=0x7fffffffdd80, data=0x615370) at multi.c:2448
#15 0x00007ffff76ba3aa in curl_multi_perform (multi=0x605010, running_handles=0x7fffffffde9c) at multi.c:2745
#16 0x000000000040275b in main () at /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/test.cpp:60

Test code

#include <string>
#include <vector>
#include <sstream>
#include <unistd.h>
#include "curl/curl.h"

size_t mycurl_onrecv_body(char *ptr, size_t size, size_t nmemb, void *userdata)
{
    return size * nmemb;
}

void mycurl_process(CURLM *multi_handle)
{
    CURLMsg *curl_msg = nullptr;
    int msgs_left = 0;
    while((curl_msg = curl_multi_info_read(multi_handle, &msgs_left))) {
        if(curl_msg->msg == CURLMSG_DONE) {
            CURL *http_handle = curl_msg->easy_handle;
            CURLcode code = curl_msg->data.result;
            curl_multi_remove_handle(multi_handle, http_handle);
            printf("handle %p finished\n", http_handle);
            curl_easy_cleanup(http_handle);
        }
    }
    return;
}

int main()
{
    curl_global_init(CURL_GLOBAL_ALL);
    printf("%s\n", curl_version());

    CURLM *multi_handle = curl_multi_init();

    int reqid = 0;
    std::vector<std::string> url_list {
        "https://cloudflare-quic.com/",
        "https://cloudflare-quic.com/",
        "https://cloudflare-quic.com/",
    };

    int count = 5;
    while(count--) {
        int still_running = 0;
        for(auto url : url_list) {
            printf("New Request[%d]: %s\n", reqid++, url.c_str());
            CURL *http_handle = curl_easy_init();
            curl_easy_setopt(http_handle, CURLOPT_URL, url.c_str());
            curl_easy_setopt(http_handle, CURLOPT_VERBOSE, 1L);
            curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, mycurl_onrecv_body);
            curl_easy_setopt(http_handle, CURLOPT_CONNECTTIMEOUT_MS, 5000L);
            curl_easy_setopt(http_handle, CURLOPT_TIMEOUT_MS, 10000L);
            curl_easy_setopt(http_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_3ONLY);
            curl_multi_add_handle(multi_handle, http_handle);
        }

        do {
            int numfds = 0;
            curl_multi_wait(multi_handle, NULL, 0, 10, &numfds);
            curl_multi_perform(multi_handle, &still_running);
            mycurl_process(multi_handle);
        } while(still_running);
        sleep(5);
    }

    curl_multi_cleanup(multi_handle);
    return 0;
}

curl/libcurl version

curl 8.1.2 (x86_64-pc-linux-gnu) libcurl/8.1.2 BoringSSL zlib/1.2.7 brotli/1.0.9 nghttp2/1.41.0 quiche/0.17.1
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets

operating system

Linux mylinux 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[Cering]

sorry, forget print the bt, and already updated

[icing]

Thanks for the details and test program. I modified this a bit to compile it in plain C (see below). With this, I am unable to reproduce the issue. Do you see the crash every time?:

#include <unistd.h>
#include "curl/curl.h"

static size_t mycurl_onrecv_body(char *ptr, size_t size,
                                 size_t nmemb, void *userdata)
{
    (void)ptr;
    (void)userdata;
    return size * nmemb;
}

static void mycurl_process(CURLM *multi_handle)
{
    CURLMsg *curl_msg = NULL;
    int msgs_left = 0;
    while((curl_msg = curl_multi_info_read(multi_handle, &msgs_left))) {
        if(curl_msg->msg == CURLMSG_DONE) {
            CURL *http_handle = curl_msg->easy_handle;
            CURLcode code = curl_msg->data.result;
            curl_multi_remove_handle(multi_handle, http_handle);
            printf("handle %p finished -> %d\n", (void*)http_handle, code);
            curl_easy_cleanup(http_handle);
        }
    }
    return;
}

int main(void)
{
    CURLM *multi_handle;
    CURL *http_handle;
    int reqid = 0, i, count;
    const char *urls[] = {
        "https://cloudflare-quic.com/",
        "https://cloudflare-quic.com/",
        "https://cloudflare-quic.com/",
        NULL
    };

    curl_global_init(CURL_GLOBAL_ALL);
    printf("%s\n", curl_version());

    multi_handle = curl_multi_init();

    count = 5;
    while(count--) {
        int still_running = 0;
        for(i = 0; urls[i]; ++i) {
            printf("New Request[%d]: %s\n", reqid++, urls[i]);
            http_handle = curl_easy_init();
            curl_easy_setopt(http_handle, CURLOPT_URL, urls[i]);
            curl_easy_setopt(http_handle, CURLOPT_VERBOSE, 1L);
            curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, mycurl_onrecv_body);
            curl_easy_setopt(http_handle, CURLOPT_CONNECTTIMEOUT_MS, 5000L);
            curl_easy_setopt(http_handle, CURLOPT_TIMEOUT_MS, 10000L);
            curl_easy_setopt(http_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_3ONLY);
            curl_multi_add_handle(multi_handle, http_handle);
        }

        do {
            int numfds = 0;
            curl_multi_wait(multi_handle, NULL, 0, 10, &numfds);
            curl_multi_perform(multi_handle, &still_running);
            mycurl_process(multi_handle);
        } while(still_running);
        sleep(1);
    }

    curl_multi_cleanup(multi_handle);
    return 0;
}

[Cering]

@icing yes, I tried the test program aboved and compiled with gcc, it crash as well and always crash on the first request. The full output is:

[build]$ gdb temp 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/build/temp...done.
(gdb) r
Starting program: /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/build/temp 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
libcurl/8.1.2 BoringSSL zlib/1.2.7 brotli/1.0.9 nghttp2/1.41.0 quiche/0.17.1
New Request[0]: https://cloudflare-quic.com/
New Request[1]: https://cloudflare-quic.com/
New Request[2]: https://cloudflare-quic.com/
[New Thread 0x7ffff6a24700 (LWP 54457)]
* Found bundle for host: 0x60d3e0 [serially]
* Server doesn't support multiplex (yet)
* Connection #0 is still name resolving, can't reuse
[New Thread 0x7ffff6223700 (LWP 54458)]
* Found bundle for host: 0x60d3e0 [serially]
* Server doesn't support multiplex (yet)
* Connection #0 is still name resolving, can't reuse
* Connection #1 is still name resolving, can't reuse
[New Thread 0x7ffff5a22700 (LWP 54459)]
*   Trying 104.22.9.38:443...
[Thread 0x7ffff5a22700 (LWP 54459) exited]
[Thread 0x7ffff6a24700 (LWP 54457) exited]
[Thread 0x7ffff6223700 (LWP 54458) exited]
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Hostname 'cloudflare-quic.com' was found in DNS cache
*   Trying 104.22.9.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Hostname 'cloudflare-quic.com' was found in DNS cache
*   Trying 104.22.9.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
* ipv4 connect timeout after 2490ms, move on!
*   Trying 172.67.9.235:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* ipv4 connect timeout after 2462ms, move on!
* ipv4 connect timeout after 2449ms, move on!
*   Trying 172.67.9.235:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*   Trying 172.67.9.235:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*  subjectAltName: host "cloudflare-quic.com" matched cert's "cloudflare-quic.com"
* Connected to cloudflare-quic.com (172.67.9.235) port 443 (#0)
* using HTTP/3
* Using HTTP/3 Stream ID: 0 (easy handle 0x604700)
> GET / HTTP/3
Host: cloudflare-quic.com
Accept: */*

< HTTP/3 200 
< date: Mon, 17 Jul 2023 10:17:48 GMT
< content-type: text/html
< content-length: 125959
< server: cloudflare
< cf-ray: 7e81bcbbb83a9e6b-SJC
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host cloudflare-quic.com left intact
handle 0x604700 finished -> 0
* ipv4 connect timeout after 1245ms, move on!
*   Trying 104.22.8.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* ipv4 connect timeout after 1223ms, move on!
*   Trying 104.22.8.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*  subjectAltName: host "cloudflare-quic.com" matched cert's "cloudflare-quic.com"
* Connected to cloudflare-quic.com (104.22.8.38) port 443 (#1)
* using HTTP/3
* Using HTTP/3 Stream ID: 0 (easy handle 0x605c60)
> GET / HTTP/3
Host: cloudflare-quic.com
Accept: */*


Program received signal SIGSEGV, Segmentation fault.
bufcp_take (pchunk=<synthetic pointer>, pool=0x848d68) at bufq.c:191
191         pool->spare = chunk->next;
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-16.el7_4.2.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) 
(gdb) bt
#0  bufcp_take (pchunk=<synthetic pointer>, pool=0x848d68) at bufq.c:191
#1  get_spare (q=0x8ec878) at bufq.c:338
#2  get_non_full_tail (q=q@entry=0x8ec878) at bufq.c:387
#3  0x00007ffff76818b0 in Curl_bufq_write (q=0x8ec878, buf=buf@entry=0x7ffff79f19e0 "HTTP/3 ", len=len@entry=7, err=err@entry=0x7fffffffdb0c) at bufq.c:412
#4  0x00007ffff76e6adb in write_resp_raw (data=<optimized out>, mem=mem@entry=0x7ffff79f19e0, memlen=memlen@entry=7, cf=<optimized out>) at vquic/curl_quiche.c:356
#5  0x00007ffff76e6bbc in cb_each_header (name=<optimized out>, name_len=<optimized out>, value=0xe34980 "200", value_len=3, argp=0x7fffffffdbb0) at vquic/curl_quiche.c:384
#6  0x00007ffff76fba3c in quiche_h3_event_for_each_header () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#7  0x00007ffff76e6fc8 in h3_process_event (ev=0x935160, stream3_id=0, data=0x605c60, cf=0xd619f0) at vquic/curl_quiche.c:515
#8  cf_poll_events (data=0x605c60, cf=0xd619f0) at vquic/curl_quiche.c:591
#9  cf_process_ingress (cf=cf@entry=0xd619f0, data=data@entry=0x605c60) at vquic/curl_quiche.c:680
#10 0x00007ffff76e709f in cf_quiche_recv () at vquic/curl_quiche.c:841
#11 0x00007ffff76c1117 in Curl_read (data=data@entry=0x605c60, sockfd=<optimized out>, buf=buf@entry=0x60e690 "", sizerequested=sizerequested@entry=16384, 
    n=n@entry=0x7fffffffdcf0) at sendf.c:415
#12 0x00007ffff76d2950 in readwrite_data (comeback=0x7fffffffdd6f, done=0x7fffffffdd6d, didwhat=<synthetic pointer>, k=0x605d30, conn=0x6126a0, data=0x605c60)
    at transfer.c:461
#13 Curl_readwrite (conn=0x6126a0, data=data@entry=0x605c60, done=done@entry=0x7fffffffdd6d, comeback=comeback@entry=0x7fffffffdd6f) at transfer.c:1115
#14 0x00007ffff76b89e4 in multi_runsingle (multi=multi@entry=0x603010, nowp=nowp@entry=0x7fffffffddd0, data=0x605c60) at multi.c:2448
#15 0x00007ffff76ba3aa in curl_multi_perform (multi=0x603010, running_handles=0x7fffffffdecc) at multi.c:2745
#16 0x0000000000400f4c in main () at /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/temp.c:63

[Cering]

And found a new crash stack, with the same c test program:

Program received signal SIGSEGV, Segmentation fault.
__memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:37
37      ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
(gdb) 
(gdb) bt
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:37
#1  0x00007ffff7711fab in quiche::stream::RecvBuf::emit::hf79108df733a69dc () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#2  0x00007ffff791cb65 in quiche::Connection::stream_recv::h85cc83393b74bc7a () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#3  0x00007ffff76fcd74 in quiche::h3::stream::Stream::try_consume_data::h9ad1effc9976f6da ()
   from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#4  0x00007ffff7707d89 in quiche::h3::Connection::recv_body::hac9d766d92f7f4f3 () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#5  0x00007ffff76fbd2a in quiche_h3_recv_body () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#6  0x00007ffff76e6768 in stream_resp_read (reader_ctx=<optimized out>, buf=<optimized out>, len=<optimized out>, err=0x7fffffffdba4) at vquic/curl_quiche.c:422
#7  0x00007ffff7681dfc in chunk_slurpn (err=0x7fffffffdba4, reader_ctx=0x7fffffffdbb0, reader=0x7ffff76e6720 <stream_resp_read>, max_len=9653683, chunk=0x934e38)
    at bufq.c:109
#8  Curl_bufq_sipn (q=0x8ec878, max_len=<optimized out>, reader=0x7ffff76e6720 <stream_resp_read>, reader_ctx=0x7fffffffdbb0, err=0x7fffffffdba4) at bufq.c:597
#9  0x00007ffff7681ed3 in bufq_slurpn (max_len=0, err=0x7fffffffdba4, reader_ctx=0x7fffffffdbb0, reader=0x7ffff76e6720 <stream_resp_read>, q=0x8ec878) at bufq.c:626
#10 Curl_bufq_slurp (q=q@entry=0x8ec878, reader=reader@entry=0x7ffff76e6720 <stream_resp_read>, reader_ctx=reader_ctx@entry=0x7fffffffdbb0, err=err@entry=0x7fffffffdba4)
    at bufq.c:658
#11 0x00007ffff76e6f1a in cf_recv_body (data=0x6071c0, cf=0xe5e180) at vquic/curl_quiche.c:458
#12 h3_process_event (ev=0xf60d00, stream3_id=0, data=0x6071c0, cf=0xe5e180) at vquic/curl_quiche.c:526
#13 cf_poll_events (data=0x6071c0, cf=0xe5e180) at vquic/curl_quiche.c:591
#14 cf_process_ingress (cf=cf@entry=0xe5e180, data=data@entry=0x6071c0) at vquic/curl_quiche.c:680
#15 0x00007ffff76e709f in cf_quiche_recv () at vquic/curl_quiche.c:841
#16 0x00007ffff76c1117 in Curl_read (data=data@entry=0x6071c0, sockfd=<optimized out>, buf=buf@entry=0x6135c0 "", sizerequested=sizerequested@entry=16384, 
    n=n@entry=0x7fffffffdcf0) at sendf.c:415
#17 0x00007ffff76d2950 in readwrite_data (comeback=0x7fffffffdd6f, done=0x7fffffffdd6d, didwhat=<synthetic pointer>, k=0x607290, conn=0x6175d0, data=0x6071c0)
    at transfer.c:461
#18 Curl_readwrite (conn=0x6175d0, data=data@entry=0x6071c0, done=done@entry=0x7fffffffdd6d, comeback=comeback@entry=0x7fffffffdd6f) at transfer.c:1115
---Type <return> to continue, or q <return> to quit---
#19 0x00007ffff76b89e4 in multi_runsingle (multi=multi@entry=0x603010, nowp=nowp@entry=0x7fffffffddd0, data=0x6071c0) at multi.c:2448
#20 0x00007ffff76ba3aa in curl_multi_perform (multi=0x603010, running_handles=0x7fffffffdecc) at multi.c:2745
#21 0x0000000000400f4c in main () at /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/temp.c:63

[Cering]

@icing yes, I tried the test program aboved and compiled with gcc, it crash as well and always crash on the first request. The full output is:

fix: not always crash, but high probability

[icing]

On my machine, it never does. Still looking.

What are the values of chunk->next and chunk when it crashes? This looks like a use after free, but I find no code path for that to happen...

[Cering]

rebuild with -Og, the bt info and param are showed below. It seems that ... pool is alreay in an invalid address?

Program received signal SIGBUS, Bus error.
bufcp_take (pool=0x1a9da98, pchunk=pchunk@entry=0x7fffffffda18) at bufq.c:191
191         pool->spare = chunk->next;
(gdb) 
(gdb) bt
#0  bufcp_take (pool=0x1a9da98, pchunk=pchunk@entry=0x7fffffffda18) at bufq.c:191
#1  0x00007ffff7681445 in get_spare (q=q@entry=0x1bc0048) at bufq.c:338
#2  0x00007ffff7681568 in get_non_full_tail (q=q@entry=0x1bc0048) at bufq.c:387
#3  0x00007ffff768185c in Curl_bufq_write (q=0x1bc0048, buf=buf@entry=0x7ffff79ed982 "HTTP/3 ", len=len@entry=7, err=err@entry=0x7fffffffda9c) at bufq.c:412
#4  0x00007ffff76e2ed1 in write_resp_raw (cf=<optimized out>, data=<optimized out>, mem=mem@entry=0x7ffff79ed982, memlen=memlen@entry=7) at vquic/curl_quiche.c:356
#5  0x00007ffff76e2f71 in cb_each_header (name=<optimized out>, name_len=<optimized out>, value=0x1ccb190 "200", value_len=3, argp=0x7fffffffdb00) at vquic/curl_quiche.c:384
#6  0x00007ffff76f7d60 in quiche_h3_event_for_each_header () from /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/deps/curl/lib/libcurl.so.4
#7  0x00007ffff76e317b in h3_process_event (cf=cf@entry=0x1c85960, data=data@entry=0x615bd0, stream3_id=stream3_id@entry=0, ev=0x1d67230) at vquic/curl_quiche.c:515
#8  0x00007ffff76e32b7 in cf_poll_events (cf=cf@entry=0x1c85960, data=data@entry=0x615bd0) at vquic/curl_quiche.c:591
#9  0x00007ffff76e334d in cf_process_ingress (cf=cf@entry=0x1c85960, data=data@entry=0x615bd0) at vquic/curl_quiche.c:680
#10 0x00007ffff76e3860 in cf_quiche_recv (cf=0x1c85960, data=0x615bd0, buf=0x1ab3580 "", len=16384, err=0x7fffffffdc7c) at vquic/curl_quiche.c:841
#11 0x00007ffff7687b87 in Curl_cf_def_recv (cf=<optimized out>, data=<optimized out>, buf=<optimized out>, len=<optimized out>, err=<optimized out>) at cfilters.c:99
#12 0x00007ffff7687b87 in Curl_cf_def_recv (cf=<optimized out>, data=<optimized out>, buf=<optimized out>, len=<optimized out>, err=<optimized out>) at cfilters.c:99
#13 0x00007ffff7687b87 in Curl_cf_def_recv (cf=<optimized out>, data=<optimized out>, buf=<optimized out>, len=<optimized out>, err=<optimized out>) at cfilters.c:99
#14 0x00007ffff7687cb7 in Curl_conn_recv (data=<optimized out>, num=<optimized out>, buf=<optimized out>, len=<optimized out>, code=0x7fffffffdc7c) at cfilters.c:180
#15 0x00007ffff76be8aa in Curl_read (data=data@entry=0x615bd0, sockfd=<optimized out>, buf=buf@entry=0x1ab3580 "", sizerequested=sizerequested@entry=16384, 
    n=n@entry=0x7fffffffdcd8) at sendf.c:415
#16 0x00007ffff76ce295 in readwrite_data (data=data@entry=0x615bd0, conn=conn@entry=0x1ab7590, k=k@entry=0x615ca0, didwhat=didwhat@entry=0x7fffffffdd2c, 
    done=done@entry=0x7fffffffdd9b, comeback=comeback@entry=0x7fffffffdd87) at transfer.c:461
#17 0x00007ffff76cf5d6 in Curl_readwrite (conn=0x1ab7590, data=data@entry=0x615bd0, done=done@entry=0x7fffffffdd9b, comeback=comeback@entry=0x7fffffffdd87)
    at transfer.c:1115
#18 0x00007ffff76b7d72 in multi_runsingle (multi=multi@entry=0x603010, nowp=nowp@entry=0x7fffffffde70, data=data@entry=0x615bd0) at multi.c:2448
#19 0x00007ffff76b8425 in curl_multi_perform (multi=multi@entry=0x603010, running_handles=running_handles@entry=0x7fffffffded8) at multi.c:2745
#20 0x0000000000400ced in main () at /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/temp.c:65
(gdb)  
(gdb) p *pool
$1 = {spare = 0x56434341494b5007, chunk_size = 312159073664503089, spare_count = 3555102911988239370, spare_max = 433564935608938507}
(gdb)  
(gdb) p *chunk
Cannot access memory at address 0x56434341494b5007
(gdb)  
(gdb) p *chunk->next
Cannot access memory at address 0x56434341494b5007

[Cering]

By the way, I found that although the previous version libcurl/8.0.1+quiche/0.16.0 not crash, but will get a CURLE_WEIRD_SERVER_REPLY error, the fail info is Header without colon in http.c:3756 (curl-8.0.1 release). Maybe caused by the same problem.

New Request[6]: https://cloudflare-quic.com/
New Request[7]: https://cloudflare-quic.com/
New Request[8]: https://cloudflare-quic.com/
* Hostname cloudflare-quic.com was found in DNS cache
*   Trying 104.22.8.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Hostname cloudflare-quic.com was found in DNS cache
*   Trying 104.22.8.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Hostname cloudflare-quic.com was found in DNS cache
*   Trying 104.22.8.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
*   Trying [2606:4700:10::ac43:9eb]:443...
* Immediate connect fail for 2606:4700:10::ac43:9eb: Network is unreachable
*   Trying [2606:4700:10::6816:826]:443...
* Immediate connect fail for 2606:4700:10::6816:826: Network is unreachable
*   Trying [2606:4700:10::6816:926]:443...
* Immediate connect fail for 2606:4700:10::6816:926: Network is unreachable
* ipv4 connect timeout after 2500ms, move on!
*   Trying 104.22.9.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* ipv4 connect timeout after 2500ms, move on!
*   Trying 104.22.9.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* ipv4 connect timeout after 2500ms, move on!
*   Trying 104.22.9.38:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
*  subjectAltName: host "cloudflare-quic.com" matched cert's "cloudflare-quic.com"
* Connected to cloudflare-quic.com (104.22.9.38) port 443 (#8)
* using HTTP/3
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: cloudflare-quic.com]
* h2h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0x8470e0)
> GET / HTTP/3
Host: cloudflare-quic.com
accept: */*

*  subjectAltName: host "cloudflare-quic.com" matched cert's "cloudflare-quic.com"
* Connected to cloudflare-quic.com (104.22.9.38) port 443 (#6)
* using HTTP/3
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: cloudflare-quic.com]
* h2h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0x6133d0)
> GET / HTTP/3
Host: cloudflare-quic.com
accept: */*

< HTTP/3 200 
< date: Tue, 18 Jul 2023 03:05:13 GMT
< content-type: text/html
< content-length: 125959
< server: cloudflare
< cf-ray: 7e8780750f21aaa6-SJC
< alt-svc: h3=":443"; ma=86400
* ipv4 connect timeout after 1248ms, move on!
*   Trying 172.67.9.235:443...
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none

Breakpoint 1, verify_header (data=data@entry=0x6133d0) at http.c:3756
3756          failf(data, "Header without colon");
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-16.el7_4.2.x86_64 zlib-1.2.7-17.el7.x86_64

(gdb) bt
#0  verify_header (data=data@entry=0x6133d0) at http.c:3756
#1  0x00007ffff76b0f90 in Curl_http_readwrite_headers (data=data@entry=0x6133d0, conn=conn@entry=0x61f620, nread=nread@entry=0x7fffffffdce8, 
    stop_reading=stop_reading@entry=0x7fffffffdce0) at http.c:4309
#2  0x00007ffff76d928c in readwrite_data (data=data@entry=0x6133d0, conn=conn@entry=0x61f620, k=k@entry=0x6134a0, didwhat=didwhat@entry=0x7fffffffdd3c, 
    done=done@entry=0x7fffffffddab, comeback=comeback@entry=0x7fffffffdd97) at transfer.c:522
#3  0x00007ffff76da432 in Curl_readwrite (conn=0x61f620, data=data@entry=0x6133d0, done=done@entry=0x7fffffffddab, comeback=comeback@entry=0x7fffffffdd97) at transfer.c:1119
#4  0x00007ffff76c2d27 in multi_runsingle (multi=multi@entry=0x603010, nowp=nowp@entry=0x7fffffffde80, data=data@entry=0x6133d0) at multi.c:2443
#5  0x00007ffff76c33d3 in curl_multi_perform (multi=multi@entry=0x603010, running_handles=running_handles@entry=0x7fffffffded8) at multi.c:2729
#6  0x0000000000400ced in main () at /data/yldata/git_workspace/ThirdPartyLibs/libcurl/demo/temp.c:65

(gdb) p *k
$2 = {size = 125959, maxdownload = 125959, bytecount = 0, writebytecount = 0, headerbytecount = 179, deductheadercount = 0, pendingheader = 0, start = {tv_sec = 106326379, 
    tv_usec = 991654}, badheader = HEADER_NORMAL, headerline = 8, 
  str = 0x62069d "date: Tue, 18 Jul 2023 03:05:14 GMT\r\ncontent-type: text/html\r\ncontent-length: 125959\r\nserver: cloudflare\r\ncf-ray: 7e878079de6a9669-SJC\r\nalt-svc: h3=\":443\"; ma=86400\r\n\r\n<!DOCTYPE html>\n<html lang=\"en\" "..., offset = 0, httpcode = 200, keepon = 1, start100 = {tv_sec = 0, tv_usec = 0}, 
  exp100 = EXP100_SEND_DATA, upgr101 = UPGR101_INIT, writer_stack = 0x0, timeofdoc = 0, bodywrites = 0, location = 0x0, newurl = 0x0, upload_present = 0, 
  upload_fromhere = 0x0, p = {file = 0xf3f440, ftp = 0xf3f440, http = 0xf3f440, imap = 0xf3f440, ldap = 0xf3f440, mqtt = 0xf3f440, pop3 = 0xf3f440, rtsp = 0xf3f440, 
    smb = 0xf3f440, smtp = 0xf3f440, ssh = 0xf3f440, telnet = 0xf3f440}, doh = 0x0, setcookies = 0 '\000', writer_stack_depth = 0 '\000', header = 1, content_range = 0, 
  upload_done = 1, ignorebody = 0, http_bodyless = 0, chunk = 0, ignore_cl = 0, upload_chunky = 0, getheader = 1, forbidchunk = 0, no_body = 0}

(gdb) p header
$5 = 0x1cb2a70 "HTTP/3 200 \r\n"
(gdb) p hlen
$6 = 13
(gdb) p ptr
$7 = 0x0

(gdb) set print elements 0
(gdb) p k->str
$13 = 0x62069d "date: Tue, 18 Jul 2023 03:05:14 GMT\r\ncontent-type: text/html\r\ncontent-length: 125959\r\nserver: cloudflare\r\ncf-ray: 7e878079de6a9669-SJC\r\nalt-svc: h3=\":443\"; ma=86400\r\n\r\n<!DOCTYPE html>\n<html lang=\"en\" itemscope=\"\" itemtype=\"http://schema.org/Article\"><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n    <title>QUIC | Cloudflare</title>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <meta name=\"description\" content=\"Cloudflare Supports QUIC\">\n    <link rel=\"canonical\" href=\"https://cloudflare-quic.com\">\n    <meta itemprop=\"name\" content=\"QUIC | Cloudflare\">\n    <meta itemprop=\"description\" content=\"Cloudflare Supports QUIC\">\n    <meta itemprop=\"image\" content=\"https://www.cloudflare.com/img/cf-facebook-card.png\">\n    <meta name=\"twitter:card\" content=\"summary\">\n    <meta name=\"twitter:title\" content=\"QUIC | Cloudflare\">\n    <meta name=\"twitter:description\" content=\"Cloudflare Supports QUIC\">\n    <meta name=\"twitter:site\" content=\"@cloudflare\">\n    <meta name=\"twitter:creator\" content=\"@cloudflare\">\n    <meta name=\"twitter:image\" content=\"https://www.cloudflare.com/img/cf-twitter-card.png\">\n    <meta property=\"og:title\" content=\"QUIC | Cloudflare\">\n    <meta property=\"og:type\" content=\"article\">\n    <meta property=\"og:url\" content=\"https://cloudflare-quic.com\">\n    <meta property=\"og:image\" content=\"https://www.cloudflare.com/img/cf-facebook-card.png\">\n    <meta property=\"og:description\" content=\"Cloudflare Supports QUIC\">\n    <meta property=\"og:site_name\" content=\"Cloudflare\">\n    <meta class=\"swiftype\" name=\"language\" data-type=\"string\" content=\"en\">\n    <meta class=\"swiftype\" name=\"title\" data-type=\"string\" content=\"QUIC\">\n    <meta class=\"swiftype\" name=\"description\" data-type=\"string\" content=\"Cloudflare Supports QUIC\">\n    <meta class=\"swiftype\" name=\"url\" data-type=\"enum\" content=\"https://cloudflare-quic.com\">\n    <link rel=\"icon\" type=\"image/x-icon\" href=\"https://www.cloudflare.com/favicon.ico\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"57x57\" href=\"https://www.cloudflare.com/apple-touch-icon-57x57.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"114x114\" href=\"https://www.cloudflare.com/apple-touch-icon-114x114.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"72x72\" href=\"https://www.cloudflare.com/apple-touch-icon-72x72.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"144x144\" href=\"https://www.cloudflare.com/apple-touch-icon-144x144.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"60x60\" href=\"https://www.cloudflare.com/apple-touch-icon-60x60.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"120x120\" href=\"https://www.cloudflare.com/apple-touch-icon-120x120.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"76x76\" href=\"https://www.cloudflare.com/apple-touch-icon-76x76.png\">\n    <link rel=\"apple-touch-icon-precomposed\" sizes=\"152x152\" href=\"https://www.cloudflare.com/apple-touch-icon-152x152.png\">\n    <link rel=\"icon\" type=\"image/png\" href=\"https://www.cloudflare.com/favicon-196x196.png\" sizes=\"196x196\">\n    <link rel=\"icon\" type=\"image/png\" href=\"https://www.cloudflare.com/favicon-96x96.png\" sizes=\"96x96\">\n    <link rel=\"icon\" type=\"image/png\" href=\"https://www.cloudflare.com/favicon-32x32.png\" sizes=\"32x32\">\n    <link rel=\"icon\" type=\"image/png\" href=\"https://www.cloudflare.com/favicon-16x16.png\" sizes=\"16x16\">\n    <link rel=\"icon\" type=\"image/png\" href=\"https://www.cloudflare.com/favicon-128.png\" sizes=\"128x128\">\n    <meta name=\"application-name\" content=\"Cloudflare\">\n    <meta name=\"msapplication-TileColor\" content=\"#FFFFFF\">\n    <meta name=\"msapplication-TileImage\" content=\"/mstile-144x144.png\">\n    <meta name=\"msapplication-square70x70logo\" content=\"/mstile-70x70.png\">\n    <meta name=\"msapplication-square150x150logo\" content=\"/mstile-150x150.png\">\n    <meta name=\"msapplication-wide310x150logo\" content=\"/mstile-310x150.png\">\n    <meta name=\"msapplication-square310x310logo\" content=\"/mstile-310x310.png\">\n    <noscript><style>.nojs-hide{display: none;}</style></noscript>\n    <style>.async-hide {opacity: 0 !important}</style>\n    <style> html{line-height:1.15;-webkit-text-size-adjust:100%}body{margin:0}h1{font-size:2em;margin:.67em 0}a{background-color:transparent}b,strong{font-weight:bolder}img{border-style:none}button,input{font-family:inherit;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button{text-transform:none}[type=submit],button{-webkit-appearance:button}[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}a,---Type <return> to continue, or q <return> to quit---
body,div,footer,form,h1,h2,h4,header,html,input[type=text],li,nav,p,section,table,td,tr,ul{box-sizing:border-box}img{max-width:100%}.b{font-weight:700!important}/*!normalize.css v8.0.0 | MIT License | github.com/necolas/normalize.css*/html{line-height:1.15;-webkit-text-size-adjust:100%}body{margin:0}h1{font-size:2em;margin:.67em 0}a{background-color:transparent}b,strong{font-weight:bolder}img{border-style:none}button,input{font-family:inherit;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button{text-transform:none}[type=submit],button{-webkit-appearance:button}[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}body{font-family:-apple-system,BlinkMacSystemFont,segoe ui,Roboto,Oxygen,Ubuntu,Cantarell,fira sans,droid sans,helvetica neue,sans-serif}.hidden-element{display:none}.header-language-picker-dropdown{list-style-type:none;margin:0;padding:0}.header-language-picker-dropdown li{line-height:1.7em}.header-language-picker-dropdown li a{text-decoration:none}.top-nav__item-list a{text-decoration:none}/*!normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css*/html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}b,strong{font-weight:inherit}b,strong{font-weight:bolder}img{border-style:none}button,input{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button{text-transform:none}[type=submit],button{-webkit-appearance:button}[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}canvas{display:inline-block}*{box-sizing:border-box;font-weight:300}body{font-size:16px;line-height:1.7em;font-family:-apple-system,BlinkMacSystemFont,segoe ui,Roboto,Oxygen,Ubuntu,Cantarell,fira sans,droid sans,helvetica neue,sans-serif;color:#424242;-ms-scroll-chaining:none;overscroll-behavior:none}button{border:none;outline:0}button:focus{outline:auto;outline-color:#0055dc}.site-content{margin:100px 0 0;width:100%}@media(min-width:1000px){.site-content{margin:100px 0 0}}section{position:relative;height:100%;z-index:0}a:link{color:#3475c0;text-decoration:none;font-weight:300}a:visited{color:#3475c0;font"

[icing]

quiche seems to have bugs handling more than one connection: cloudflare/quiche#1554

I see this in local tests that quiche events get mixed up and some requests stall. Until we have verified that quiche is fixed, I would rest the analysis here.

[Cering]

Thanks!

[icing]

@Cering please check #11462 for a possible fix to your problems.