I've built my app with curl + HTTP3 (via quictls + ngtcp2 + nghttp3). On windows (built with MSVC) it crashes 100% of times, on macOS or Android it is very hard to repro the crash (easy to repro with curl 8.4.0). ASAN dump:
NB. the app works fine when built with HTTP2 only
=================================================================
==3448==ERROR: AddressSanitizer: heap-use-after-free on address 0x1209f13a7aa8 at pc 0x7ff7929a4eb3 bp 0x00a6674e9fb0 sp 0x00a6674e9fb8
READ of size 8 at 0x1209f13a7aa8 thread T0
==3448==WARNING: Failed to use and restart external symbolizer!
#0 0x7ff7929a4eb2 in cb_h3_recv_data C:\src\curl\lib\vquic\curl_ngtcp2.c:1235
#1 0x7ff7933576f3 in nghttp3_conn_on_data C:\src\vcpkg\buildtrees\nghttp3\src\v1.0.0-b55f3d7380.clean\lib\nghttp3_conn.c:1540
#2 0x7ff793353e1d in nghttp3_conn_read_bidi C:\src\vcpkg\buildtrees\nghttp3\src\v1.0.0-b55f3d7380.clean\lib\nghttp3_conn.c:1355
#3 0x7ff79334e16a in nghttp3_conn_read_stream C:\src\vcpkg\buildtrees\nghttp3\src\v1.0.0-b55f3d7380.clean\lib\nghttp3_conn.c:488
#4 0x7ff7929a2552 in cb_recv_stream_data C:\src\curl\lib\vquic\curl_ngtcp2.c:838
#5 0x7ff79327cff0 in conn_call_recv_stream_data C:\src\vcpkg\buildtrees\ngtcp2\src\v1.0.1-2d9b978f1f.clean\lib\ngtcp2_conn.c:124
#6 0x7ff7932ad075 in conn_recv_stream C:\src\vcpkg\buildtrees\ngtcp2\src\v1.0.1-2d9b978f1f.clean\lib\ngtcp2_conn.c:7225
#7 0x7ff7932a5f28 in conn_recv_pkt C:\src\vcpkg\buildtrees\ngtcp2\src\v1.0.1-2d9b978f1f.clean\lib\ngtcp2_conn.c:9311 #8 0x7ff7932b7605 in conn_recv_cpkt C:\src\vcpkg\buildtrees\ngtcp2\src\v1.0.1-2d9b978f1f.clean\lib\ngtcp2_conn.c:9725
#9 0x7ff793265117 in ngtcp2_conn_read_pkt_versioned C:\src\vcpkg\buildtrees\ngtcp2\src\v1.0.1-2d9b978f1f.clean\lib\ngtcp2_conn.c:10106
#10 0x7ff7929a9881 in recv_pkt C:\src\curl\lib\vquic\curl_ngtcp2.c:1990
#11 0x7ff7929b0e45 in recvfrom_packets C:\src\curl\lib\vquic\vquic.c:518
#12 0x7ff7929afd14 in vquic_recv_packets C:\src\curl\lib\vquic\vquic.c:544
#13 0x7ff79299ec0a in cf_progress_ingress C:\src\curl\lib\vquic\curl_ngtcp2.c:2044
#14 0x7ff7929ae8e9 in cf_ngtcp2_conn_is_alive C:\src\curl\lib\vquic\curl_ngtcp2.c:2741
#15 0x7ff7928c5533 in Curl_cf_def_conn_is_alive C:\src\curl\lib\cfilters.c:111
#16 0x7ff7928c5533 in Curl_cf_def_conn_is_alive C:\src\curl\lib\cfilters.c:111
#17 0x7ff7928c5533 in Curl_cf_def_conn_is_alive C:\src\curl\lib\cfilters.c:111
#18 0x7ff7928c97fe in Curl_conn_is_alive C:\src\curl\lib\cfilters.c:640
#19 0x7ff7928b1683 in extract_if_dead C:\src\curl\lib\url.c:807
#20 0x7ff7928b1897 in call_extract_if_dead C:\src\curl\lib\url.c:845
#21 0x7ff792899321 in Curl_conncache_foreach C:\src\curl\lib\conncache.c:332
#22 0x7ff7928b1cf6 in prune_dead_connections C:\src\curl\lib\url.c:874
#23 0x7ff7928c0b74 in create_conn C:\src\curl\lib\url.c:3649
#24 0x7ff7928ad0b6 in Curl_connect C:\src\curl\lib\url.c:3915
#25 0x7ff792878601 in multi_runsingle C:\src\curl\lib\multi.c:1927
#26 0x7ff79287bb85 in multi_socket C:\src\curl\lib\multi.c:3212
#27 0x7ff79286cce4 in curl_multi_socket_action C:\src\curl\lib\multi.c:3334
#28 0x7ff7909fe1c5 in coro::http::`anonymous namespace'::CurlHttpImpl::TimeoutEvent C:\src\coro-http\src\coro\http\curl_http.cc:540
#29 0x7ff790efd2ee in event_process_active_single_queue C:\src\vcpkg\buildtrees\libevent\src\757b64d6d1-bfa17b9727.clean\event.c:1727
#30 0x7ff790ef3dfe in event_process_active C:\src\vcpkg\buildtrees\libevent\src\757b64d6d1-bfa17b9727.clean\event.c:1819
#31 0x7ff790ede5c0 in event_base_loop C:\src\vcpkg\buildtrees\libevent\src\757b64d6d1-bfa17b9727.clean\event.c:2068
#32 0x7ff7909c4950 in coro::util::EventLoop::EnterLoop C:\src\coro-http\src\coro\util\event_loop.cc:159
#33 0x7ff78faeb932 in main C:\src\coro-cloudstorage\examples\cloudstorage.cc:100
#34 0x7ff793503fa8 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
#35 0x7ff793503efd in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288 #36 0x7ff793503dbd in __scrt_common_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:330
#37 0x7ff79350401d in mainCRTStartup D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:16
#38 0x7ff90a0d257c in BaseThreadInitThunk+0x1c (C:\WINDOWS\System32\KERNEL32.DLL+0x18001257c)
#39 0x7ff90ac4aa57 in RtlUserThreadStart+0x27 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005aa57)
0x1209f13a7aa8 is located 424 bytes inside of 5136-byte region [0x1209f13a7900,0x1209f13a8d10)
freed by thread T0 here:
#0 0x7ff78fb1ce15 in free D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win_thunk.cpp:69
#1 0x7ff7928acd7d in Curl_close C:\src\curl\lib\url.c:343
#2 0x7ff792855983 in curl_easy_cleanup C:\src\curl\lib\easy.c:804
#3 0x7ff7909f58b6 in coro::http::`anonymous namespace'::CurlHandleDeleter::operator() C:\src\coro-http\src\coro\http\curl_http.cc:81
#4 0x7ff7909fa079 in std::unique_ptr<void,coro::http::`anonymous namespace'::CurlHandleDeleter>::~unique_ptr<void,coro::http::`anonymous namespace'::CurlHandleDeleter> C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3290
#5 0x7ff7909fabe0 in coro::http::`anonymous namespace'::CurlHandle::~CurlHandle+0x70 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411aabe0)
#6 0x7ff7909fac06 in coro::http::`anonymous namespace'::CurlHandle::`scalar deleting destructor'+0x16 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411aac06)
#7 0x7ff7909fb94e in std::default_delete<coro::http::`anonymous namespace'::CurlHandle>::operator() C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3180
#8 0x7ff7909fba51 in std::unique_ptr<coro::http::`anonymous namespace'::CurlHandle,std::default_delete<coro::http::`anonymous namespace'::CurlHandle> >::~unique_ptr<coro::http::`anonymous namespace'::CurlHandle,std::default_delete<coro::http::`anonymous namespace'::CurlHandle> > C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3290
#9 0x7ff7909fbc9b in coro::http::`anonymous namespace'::CurlHttpBodyGenerator::~CurlHttpBodyGenerator+0x1b (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411abc9b)
#10 0x7ff7909ffa39 in coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator>::~Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator>+0x19 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411afa39)
#11 0x7ff7909ffa76 in coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator>::`scalar deleting destructor'+0x16 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411afa76)
#12 0x7ff7909ff6ee in std::default_delete<coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator> >::operator() C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3180 #13 0x7ff7909ff7f1 in std::unique_ptr<coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator>,std::default_delete<coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator> > >::~unique_ptr<coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator>,std::default_delete<coro::http::Response<coro::http::`anonymous namespace'::CurlHttpBodyGenerator> > > C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3290
#14 0x7ff790a00a85 in `coro::http::`anonymous namespace'::ToBody'::`21'::<parameters>::~<parameters>+0x15 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1411b0a85)
#15 0x7ff790a0400f in coro::http::`anonymous namespace'::ToBody$_ResumeCoro$1 C:\src\coro-http\src\coro\http\curl_http.cc:654
#16 0x7ff790a02ec4 in coro::http::`anonymous namespace'::ToBody$_DestroyCoro$3 C:\src\coro-http\src\coro\http\curl_http.cc:15732480
#17 0x7ff78fb160d8 in std::coroutine_handle<coro::detail::async_generator_promise<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >::destroy C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:145
#18 0x7ff78fafec99 in coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::~Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > C:\src\coro-http\src\coro\generator.h:288
#19 0x7ff78fafee69 in coro::http::Response<coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >::~Response<coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >+0x19 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x1402aee69)
#20 0x7ff7902a534e in coro::cloudstorage::GoogleDrive::GetFileContent$_ResumeCoro$1 C:\src\coro-cloudstorage\src\coro\cloudstorage\providers\google_drive.cc:196
#21 0x7ff7902a3d96 in coro::cloudstorage::GoogleDrive::GetFileContent$_DestroyCoro$3 C:\src\coro-cloudstorage\src\coro\cloudstorage\providers\google_drive.cc:15732480
#22 0x7ff78fb160d8 in std::coroutine_handle<coro::detail::async_generator_promise<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >::destroy C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:145
#23 0x7ff78fafec99 in coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::~Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > C:\src\coro-http\src\coro\generator.h:288
#24 0x7ff7900aabf5 in `coro::cloudstorage::util::internal::GetFileContentResponseBody'::`5'::<parameters>::~<parameters>+0x15 (C:\src\coro-cloudstorage\out\build\x64-AddressSanitizer\examples\cloudstorage.exe+0x14085abf5)
#25 0x7ff7900ab6c6 in coro::cloudstorage::util::internal::GetFileContentResponseBody$_ResumeCoro$1 C:\src\coro-cloudstorage\src\coro\cloudstorage\util\handler_utils.cc:18
#26 0x7ff7900aac44 in coro::cloudstorage::util::internal::GetFileContentResponseBody$_DestroyCoro$3 C:\src\coro-cloudstorage\src\coro\cloudstorage\util\handler_utils.cc:15732480
#27 0x7ff78fb160d8 in std::coroutine_handle<coro::detail::async_generator_promise<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >::destroy C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:145
#28 0x7ff78fafec99 in coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::~Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > C:\src\coro-http\src\coro\generator.h:288
previously allocated by thread T0 here:
#0 0x7ff78fb1cd9a in calloc D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win_thunk.cpp:101
#1 0x7ff7928aa907 in Curl_open C:\src\curl\lib\url.c:508
#2 0x7ff7928557f1 in curl_easy_init C:\src\curl\lib\easy.c:368
#3 0x7ff7909f5ee0 in coro::http::`anonymous namespace'::CurlHandle::CurlHandle C:\src\coro-http\src\coro\http\curl_http.cc:361
#4 0x7ff790a0c8b6 in std::make_unique<coro::http::`anonymous namespace'::CurlHandle,void * &,event_base * &,coro::http::Request<coro::Generator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const * &,coro::stdx::stop_token,coro::http::A0x7b97c9cd::CurlHttpOperation *,0> C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\memory:3477
#5 0x7ff7909fc076 in coro::http::`anonymous namespace'::CurlHttpOperation::CurlHttpOperation C:\src\coro-http\src\coro\http\curl_http.cc:488
#6 0x7ff7909fd953 in coro::http::`anonymous namespace'::CurlHttpImpl::Fetch C:\src\coro-http\src\coro\http\curl_http.cc:643
#7 0x7ff790a024c6 in coro::http::CurlHttpBase::Fetch$_ResumeCoro$1 C:\src\coro-http\src\coro\http\curl_http.cc:672
#8 0x7ff78fb1a2fa in std::coroutine_handle<void>::resume C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:85
#9 0x7ff790097a01 in coro::detail::ReturnValue<coro::detail::Task<coro::cloudstorage::util::VersionedItem>::promise_type,coro::cloudstorage::util::VersionedItem>::return_value<coro::cloudstorage::util::VersionedItem> C:\src\coro-http\src\coro\task.h:67
#10 0x7ff79007a0cd in coro::cloudstorage::util::CloudProviderAccount::GetItemById$_ResumeCoro$1 C:\src\coro-cloudstorage\src\coro\cloudstorage\util\cloud_provider_account.cc:136
#11 0x7ff78fb1a2fa in std::coroutine_handle<void>::resume C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:85
#12 0x7ff78fbe0f31 in coro::detail::ReturnValue<coro::detail::Task<std::optional<coro::cloudstorage::util::CacheManager::ItemData> >::promise_type,std::optional<coro::cloudstorage::util::CacheManager::ItemData> >::return_value<coro::cloudstorage::util::CacheManager::ItemData> C:\src\coro-http\src\coro\task.h:67
#13 0x7ff78fb53d42 in coro::cloudstorage::util::CacheManager::Get$_ResumeCoro$1 C:\src\coro-cloudstorage\src\coro\cloudstorage\util\cache_manager.cc:316
#14 0x7ff78fb1a2fa in std::coroutine_handle<void>::resume C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:85
#15 0x7ff78fbe10d1 in coro::detail::ReturnValue<coro::detail::Task<std::optional<coro::cloudstorage::util::`anonymous namespace'::DbItem> >::promise_type,std::optional<coro::cloudstorage::util::`anonymous namespace'::DbItem> >::return_value<std::optional<coro::cloudstorage::util::`anonymous namespace'::DbItem> > C:\src\coro-http\src\coro\task.h:67
#16 0x7ff78fb7dcd5 in coro::util::ThreadPool::Do$_ResumeCoro$1<`coro::cloudstorage::util::CacheManager::Get'::`2'::<lambda_1> > C:\src\coro-http\src\coro\util\thread_pool.h:60
#17 0x7ff78fb1a2fa in std::coroutine_handle<void>::resume C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:85
#18 0x7ff78fb1a3b8 in coro::detail::ReturnValue<coro::detail::Task<void>::promise_type,void>::return_void C:\src\coro-http\src\coro\task.h:76
#19 0x7ff7909cbb4a in coro::util::ThreadPool::SwitchToEventLoop$_ResumeCoro$1 C:\src\coro-http\src\coro\util\thread_pool.cc:129
#20 0x7ff78fb1a2fa in std::coroutine_handle<void>::resume C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\coroutine:85
#21 0x7ff7909caf95 in ``coro::util::ThreadPool::SwitchToEventLoop'::`2'::Awaiter::await_suspend'::`2'::<lambda_1>::operator() C:\src\coro-http\src\coro\util\thread_pool.cc:124
#22 0x7ff7909cb135 in `coro::util::EventLoop::RunOnEventLoop<``coro::util::ThreadPool::SwitchToEventLoop'::`2'::Awaiter::await_suspend'::`2'::<lambda_1> >'::`2'::<lambda_1>::operator() C:\src\coro-http\src\coro\util\event_loop.h:49
#23 0x7ff7909cdf32 in std::invoke<`coro::util::EventLoop::RunOnEventLoop<``coro::util::ThreadPool::SwitchToEventLoop'::`2'::Awaiter::await_suspend'::`2'::<lambda_1> >'::`2'::<lambda_1> &> C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\type_traits:1752
#24 0x7ff7909cb2f5 in coro::stdx::any_detail::handler_traits<void>::small_handler<`coro::util::EventLoop::RunOnEventLoop<``coro::util::ThreadPool::SwitchToEventLoop'::`2'::Awaiter::await_suspend'::`2'::<lambda_1> >'::`2'::<lambda_1> >::call C:\src\coro-http\src\coro\stdx\any_invocable.h:97
#25 0x7ff7909c8f2b in coro::stdx::any_detail::any_invocable_impl<void,0>::call C:\src\coro-http\src\coro\stdx\any_invocable.h:212
#26 0x7ff7909c8df2 in coro::stdx::any_invocable<void __cdecl(void)&& >::operator() C:\src\coro-http\src\coro\stdx\any_invocable.h:323
#27 0x7ff7909c64f4 in `coro::util::EventLoop::RunOnce'::`3'::<lambda_1>::operator() C:\src\coro-http\src\coro\util\event_loop.cc:93
#28 0x7ff7909c65fc in `coro::util::EventLoop::RunOnce'::`3'::<lambda_1>::<lambda_invoker_cdecl> C:\src\coro-http\src\coro\util\event_loop.cc:94
SUMMARY: AddressSanitizer: heap-use-after-free C:\src\curl\lib\vquic\curl_ngtcp2.c:1235 in cb_h3_recv_data
Shadow bytes around the buggy address:
0x04072f5f4f00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x04072f5f4f10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x04072f5f4f20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x04072f5f4f50: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4f90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x04072f5f4fa0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
If needed I can provide curl debug logs as well.
No crash.
I did this
I've built my app with curl + HTTP3 (via quictls + ngtcp2 + nghttp3). On windows (built with MSVC) it crashes 100% of times, on macOS or Android it is very hard to repro the crash (easy to repro with curl 8.4.0). ASAN dump:
NB. the app works fine when built with HTTP2 only
If needed I can provide curl debug logs as well.
I expected the following
No crash.
curl/libcurl version
curl head (fa71483) with quictls (openssl-3.1.4+quic), ngtcp2 (v1.0.1), nghttp3 (v1.0.0).
operating system
Windows 11