heap use after free in QUICStreamAdapter::stream via HQTransaction::get_transaction_id

[bneradt]

With the latest 10.0.x (a1665e5) built and installed on docs, I see the following heap-use-after-free reported by ASan:

=================================================================                                                                                                                                                                                                                                                             
==14752==ERROR: AddressSanitizer: heap-use-after-free on address 0x613000032068 at pc 0x55921eaaa705 bp 0x7ff0ae074a60 sp 0x7ff0ae074a50                                                                                                                                                                                      
READ of size 8 at 0x613000032068 thread T5 ([ET_NET 3])                                                                                                                                                                                                                                                                       
    #0 0x55921eaaa704 in QUICStreamAdapter::stream() /home/bneradt/src/trafficserver_10/include/iocore/net/quic/QUICStreamAdapter.h:37                                                                                                                                                                                        
    #1 0x55921eac0a23 in HQTransaction::get_transaction_id() const /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3Transaction.cc:211                                                                                                                                                                                 
    #2 0x55921eabae7a in HQSession::get_transaction(unsigned long) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3Session.cc:85                                                                                                                                                                                      
    #3 0x55921eab1144 in Http3App::_handle_bidi_stream_on_read_ready(int, VIO*) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:291                                                                                                                                                                            
    #4 0x55921eaaf159 in Http3App::main_event_handler(int, Event*) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:149                                                                                                                                                                                         
    #5 0x55921e0cd6ea in Continuation::handleEvent(int, void*) /home/bneradt/src/trafficserver_10/include/iocore/eventsystem/Continuation.h:228                                                                                                                                                                               
    #6 0x55921ea42964 in EThread::process_event(Event*, int) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:162                                                                                                                                                                                     
    #7 0x55921ea42eb8 in EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:197                                                                                                                                                     
    #8 0x55921ea4344f in EThread::execute_regular() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:255                                                                                                                                                                                              
    #9 0x55921ea43efb in EThread::execute() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:348                                                                                                                                                                                                      
    #10 0x55921ea40d11 in spawn_thread_internal /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/Thread.cc:68                                                                                                                                                                                                        
    #11 0x7ff0b50a3608 in start_thread /build/glibc-BHL3KM/glibc-2.31/nptl/pthread_create.c:477                                                                                                                                                                                                                               
    #12 0x7ff0b4fc8132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                              
0x613000032068 is located 104 bytes inside of 328-byte region [0x613000032000,0x613000032148)                                                                                                                                                                                                                                 
freed by thread T5 ([ET_NET 3]) here:                                                                                                                                                                                                                                                                                         
    #0 0x7ff0b5f4551f in operator delete(void*) ../../../../src/libsanitizer/asan/asan_new_delete.cc:165                                                                                                                                                                                                                      
    #1 0x55921eaad1cb in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> >::deallocate(std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128       
    #2 0x55921eaac9cd in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> > >::deallocate(std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> >&, std::__detail::_Hash_node<std::pa
ir<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:469                                                                                                                                                                                                      
    #3 0x55921eaac8e2 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> > >::_M_deallocate_node_ptr(std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>*) /usr/include/c++/9/bits/hash
table_policy.h:2113                                                                                                                                                                                                                                                                                                           
    #4 0x55921eaabdd9 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>*) /usr/include/c++/9/bits/hashtabl
e_policy.h:2103                                                                                                                                                                                                                                                                                                               
    #5 0x55921eab90fa in std::_Hashtable<unsigned long, std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::
__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>*) /usr/include/c++/9/bits/hashtable.h:1921
    #6 0x55921eab7c61 in std::_Hashtable<unsigned long, std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::
__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase(std::integral_constant<bool, true>, unsigned long const&) /usr/include/c++/9/bits/hashtable.h:1947                                                                                      
    #7 0x55921eab6a6a in std::_Hashtable<unsigned long, std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::
__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::erase(unsigned long const&) /usr/include/c++/9/bits/hashtable.h:804                                                                                                                              
    #8 0x55921eab5b1a in std::unordered_map<unsigned long, QUICStreamVCAdapter::IOInfo, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> > >::erase(unsigned long const&) /usr/include/c++/9/bits/unordered_map.h:816                       
    #9 0x55921eaaeb77 in Http3App::on_stream_close(QUICStream&) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:127                                                                                                                                                                                            
    #10 0x55921eb14bee in QUICStreamManager::delete_stream(unsigned long&) /home/bneradt/src/trafficserver_10/src/iocore/net/quic/QUICStreamManager.cc:121                                                                                                                                                                    
    #11 0x55921eab2a2f in Http3App::_handle_bidi_stream_on_write_complete(int, VIO*) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:443                                                                                                                                                                       
    #12 0x55921eaaf291 in Http3App::main_event_handler(int, Event*) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:174                                                                                                                                                                                        
    #13 0x55921e0cd6ea in Continuation::handleEvent(int, void*) /home/bneradt/src/trafficserver_10/include/iocore/eventsystem/Continuation.h:228                                                                                                                                                                              
    #14 0x55921ea42964 in EThread::process_event(Event*, int) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:162                                                                                                                                                                                    
    #15 0x55921ea42eb8 in EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:197                                                                                                                                                    
    #16 0x55921ea4366c in EThread::execute_regular() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:276                                                                                                                                                                                             
    #17 0x55921ea43efb in EThread::execute() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:348                                                                                                                                                                                                     
    #18 0x55921ea40d11 in spawn_thread_internal /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/Thread.cc:68                                                                                                                                                                                                        
    #19 0x7ff0b50a3608 in start_thread /build/glibc-BHL3KM/glibc-2.31/nptl/pthread_create.c:477    
previously allocated by thread T5 ([ET_NET 3]) here:
    #0 0x7ff0b5f44587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
    #1 0x55921eaad10f in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> >::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
    #2 0x55921eaac934 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> > >::allocate(std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> >&, unsigned long) /usr/include/c++/9/b
its/alloc_traits.h:443
    #3 0x55921eaabffb in std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false>* std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false> > >::_M_allocate_node<unsigned long, QUICStream&>(unsigne
d long&&, QUICStream&) /usr/include/c++/9/bits/hashtable_policy.h:2081
    #4 0x55921eaab95f in std::pair<std::__detail::_Node_iterator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false, false>, bool> std::_Hashtable<unsigned long, std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> >, s
td::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<unsigned long, QUICStream&>(std::integral_constant<bool, tru
e>, unsigned long&&, QUICStream&) /usr/include/c++/9/bits/hashtable.h:1673
    #5 0x55921eaab439 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false, false>, bool> std::_Hashtable<unsigned long, std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, std::allocator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo> >, s
td::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::emplace<unsigned long, QUICStream&>(unsigned long&&, QUICStream&) /usr/
include/c++/9/bits/hashtable.h:781
    #6 0x55921eaab0c3 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned long const, QUICStreamVCAdapter::IOInfo>, false, false>, bool> std::unordered_map<unsigned long, QUICStreamVCAdapter::IOInfo, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, QUICSt
reamVCAdapter::IOInfo> > >::emplace<unsigned long, QUICStream&>(unsigned long&&, QUICStream&) /usr/include/c++/9/bits/unordered_map.h:389
    #7 0x55921eaae928 in Http3App::on_stream_open(QUICStream&) /home/bneradt/src/trafficserver_10/src/proxy/http3/Http3App.cc:102
    #8 0x55921eb14a32 in QUICStreamManager::create_stream(unsigned long) /home/bneradt/src/trafficserver_10/src/iocore/net/quic/QUICStreamManager.cc:99
    #9 0x55921e94a08c in QUICNetVConnection::_handle_read_ready() /home/bneradt/src/trafficserver_10/src/iocore/net/QUICNetVConnection.cc:605
    #10 0x55921e946851 in QUICNetVConnection::state_established(int, Event*) /home/bneradt/src/trafficserver_10/src/iocore/net/QUICNetVConnection.cc:194
    #11 0x55921e0cd6ea in Continuation::handleEvent(int, void*) /home/bneradt/src/trafficserver_10/include/iocore/eventsystem/Continuation.h:228
    #12 0x55921e94637f in QUICNetVConnection::state_handshake(int, Event*) /home/bneradt/src/trafficserver_10/src/iocore/net/QUICNetVConnection.cc:152
    #13 0x55921e0cd6ea in Continuation::handleEvent(int, void*) /home/bneradt/src/trafficserver_10/include/iocore/eventsystem/Continuation.h:228
    #14 0x55921e94926d in QUICNetVConnection::net_read_io(NetHandler*, EThread*) /home/bneradt/src/trafficserver_10/src/iocore/net/QUICNetVConnection.cc:506
    #15 0x55921e9ac570 in NetHandler::process_ready_list() /home/bneradt/src/trafficserver_10/src/iocore/net/NetHandler.cc:276
    #16 0x55921e9acf05 in NetHandler::waitForActivity(long) /home/bneradt/src/trafficserver_10/src/iocore/net/NetHandler.cc:364
    #17 0x55921ea439b5 in EThread::execute_regular() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:299
    #18 0x55921ea43efb in EThread::execute() /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEThread.cc:348
    #19 0x55921ea40d11 in spawn_thread_internal /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/Thread.cc:68
    #20 0x7ff0b50a3608 in start_thread /build/glibc-BHL3KM/glibc-2.31/nptl/pthread_create.c:477
Thread T5 ([ET_NET 3]) created by T0 ([TS_MAIN]) here:
    #0 0x7ff0b5e6f815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x55921ea40825 in ink_thread_create /home/bneradt/src/trafficserver_10/include/tscore/ink_thread.h:129
    #2 0x55921ea40e45 in Thread::start(char const*, void*, unsigned long, std::function<void ()> const&) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/Thread.cc:85
    #3 0x55921ea4a7e9 in EventProcessor::spawn_event_threads(int, int, unsigned long) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEventProcessor.cc:467
    #4 0x55921ea4b135 in EventProcessor::start(int, unsigned long) /home/bneradt/src/trafficserver_10/src/iocore/eventsystem/UnixEventProcessor.cc:548
    #5 0x55921e0ed961 in main /home/bneradt/src/trafficserver_10/src/traffic_server/traffic_server.cc:2104
    #6 0x7ff0b4ecd082 in __libc_start_main ../csu/libc-start.c:308
SUMMARY: AddressSanitizer: heap-use-after-free /home/bneradt/src/trafficserver_10/include/iocore/net/quic/QUICStreamAdapter.h:37 in QUICStreamAdapter::stream() 
Shadow bytes around the buggy address:
  0x0c267fffe3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fffe3c0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c267fffe3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fffe3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fffe3f0: 00 fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c267fffe400: fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd
  0x0c267fffe410: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c267fffe420: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x0c267fffe430: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c267fffe440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fffe450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==14752==ABORTING

[cmcfarlen]

Please test @maskit 's change and close if fixed!

[bneradt]

I have not seen this recently. I believe it is indeed likely that #1196 has fixed this. Closing.