XpackDynamicTable::_expand_storage_size crash with null _data

[bneradt]

We saw this on a few of our ATS 10.0.x boxes:

(gdb) bt
#0  0x00007f6402be4e92 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#1  0x00000000006c2440 in XpackDynamicTableStorage::write (value_len=0, value=0x0, name_len=23, name=<optimized out>, this=0x7f59da9e8b20) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/hdrs/XPACK.cc:569
#2  ExpandCapacityContext::copy_field (len=23, offset=<optimized out>, this=<synthetic pointer>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/hdrs/XPACK.cc:620
#3  XpackDynamicTable::_expand_storage_size (this=0x7f59da9e8af8, new_storage_size=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/hdrs/XPACK.cc:492
#4  0x00000000006c2681 in XpackDynamicTable::update_maximum_size (this=this@entry=0x7f59da9e8af8, new_max_size=4096) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/hdrs/XPACK.cc:416
#5  0x000000000065bd78 in HpackIndexingTable::update_maximum_size (new_size=<optimized out>, this=0x7f59da9e8af0) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/HPACK.cc:685
#6  update_dynamic_table_size (maximum_table_size=4096, indexing_table=..., buf_end=0x7f58ee6bc79a "t cabp", buf_start=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/HPACK.cc:685
#7  hpack_decode_header_block (indexing_table=..., hdr=hdr@entry=0x7f5a25cc9f80, 
    in_buf=0x7f58ee6bc750 "?\341\037\202ه\004\260b*\210\066\"\250\276R\302\022\064\230\214\022{j\352\206\031d\017.\344o\266ۃ ]i\311\032\033\213\364G\371%t@\211ư\n㮺?\324\322\326\325\302\301\300\277\276\314\313\312\311\310\307\306\305\304t cabp", in_buf_len=<optimized out>, max_header_size=131072, 
    maximum_table_size=maximum_table_size@entry=4096) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/HPACK.cc:737
#8  0x0000000000636c35 in http2_decode_header_blocks (hdr=hdr@entry=0x7f5a25cc9f80, buf_start=<optimized out>, buf_len=<optimized out>, len_read=len_read@entry=0x0, handle=..., is_trailing_header=<optimized out>, maximum_table_size=4096, is_outbound=false)
    at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/HTTP2.cc:443
#9  0x000000000064b10e in Http2Stream::decode_header_blocks (this=this@entry=0x7f5a25cc9e00, hpack_handle=..., maximum_table_size=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2Stream.cc:691
#10 0x000000000064566c in Http2ConnectionState::rcv_headers_frame (this=0x7f5acd6e5320, frame=...) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2ConnectionState.cc:1234
#11 0x0000000000641e91 in Http2ConnectionState::rcv_frame (this=this@entry=0x7f5acd6e5320, frame=frame@entry=0x7f63ff3fc2e0) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2ConnectionState.cc:1464
#12 0x0000000000639569 in Http2CommonSession::do_complete_frame_read (this=this@entry=0x7f5acd6e5318) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2CommonSession.cc:338
#13 0x00000000006399d4 in Http2CommonSession::do_process_frame_read (this=0x7f5acd6e5318, vio=0x7f54bff5ab58, inside_frame=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2CommonSession.cc:400
#14 0x000000000065fbf0 in Http2ClientSession::main_event_handler (this=0x7f5acd6e5000, event=100, edata=0x7f54bff5ab58) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/http2/Http2ClientSession.cc:183
#15 0x000000000080f604 in Continuation::handleEvent (data=0x7f54bff5ab58, event=100, this=0x7f5acd6e5000) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/include/iocore/eventsystem/Continuation.h:228
#16 Continuation::handleEvent (data=0x7f54bff5ab58, event=100, this=0x7f5acd6e5000) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/include/iocore/eventsystem/Continuation.h:224
#17 read_signal_and_update (vc=0x7f54bff5a900, event=100) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/net/UnixNetVConnection.cc:95
#18 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x7f54bff5a900, event=event@entry=100) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/net/UnixNetVConnection.cc:989
#19 0x00000000007d968c in SSLNetVConnection::net_read_io (this=0x7f54bff5a900, nh=0x7f63fdc4df10, lthread=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/net/SSLNetVConnection.cc:731
#20 0x000000000083b088 in NetHandler::process_ready_list (this=this@entry=0x7f63fdc4df10) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/net/NetHandler.cc:286
#21 0x000000000083b38a in NetHandler::waitForActivity (this=0x7f63fdc4df10, timeout=<optimized out>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/net/NetHandler.cc:374
#22 0x0000000000876eac in EThread::execute_regular (this=this@entry=0x7f63fdc4d380) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/include/tsutil/Histogram.h:156
#23 0x0000000000877069 in EThread::execute (this=0x7f63fdc4d380) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/eventsystem/UnixEThread.cc:348
#24 EThread::execute (this=0x7f63fdc4d380) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/eventsystem/UnixEThread.cc:326
#25 0x0000000000873df2 in spawn_thread_internal (a=0x7f6401c90180) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/iocore/eventsystem/Thread.cc:75
#26 0x00007f6402ee31ca in start_thread () from /lib64/libpthread.so.0
#27 0x00007f6402b4fe73 in clone () from /lib64/libc.so.6

(gdb) f 2
#2  ExpandCapacityContext::copy_field (len=23, offset=<optimized out>, this=<synthetic pointer>) at /sd/workspace/src/git.ouryahoo.com/Edge/build/_scm/trafficserver10.0/src/proxy/hdrs/XPACK.cc:620
620       return this->_storage.write(field, len, nullptr, 0);
(gdb) l
615
616     uint32_t
617     ExpandCapacityContext::copy_field(uint32_t offset, uint32_t len)
618     {
619       char const *field = reinterpret_cast<char const *>(this->_storage._old_data + offset);
620       return this->_storage.write(field, len, nullptr, 0);
621     }
(gdb) p this->_storage._old_data
$20 = (uint8_t *) 0x0

We are in the table storage expansion phase, and having _old_data point to nullptr implies that _data was nullptr somehow. The possibilities are:

• It was initialized as such and we're now expanding it from 0. But then why would the logic think there are entries to copy?
• It's a use after free (destruction sets _data to nullptr)
• We incorrectly allowed it to shrink to 0.

The last option seems the most likely.