Segmentation fault (core dumped) with DPD anchoring at the end of payload

[mmguero]

I have attached zeek-dpdcrash.tar.gz that contains a zkg package directory (based off of bbannier/package-template-spicy). The core of the issue seems to be:

signature dpd_crash_message {
  ip-proto == tcp
  payload /^[\xf1-\xfe].+\xf6$/
  enable "spicy_dpdcrash"
}

Steps to reproduce:

1. Download and extract the attached tarball and install with zkg install --skiptest
2. Run zeek -C -r ./traces/trace.pcap zeek-dpdcrash
3. Observe Segmentation fault (core dumped) although conn.log is created.

If I change the DPD payload line to be payload /^[\xf1-\xfe]/ it does not crash.

Platform: Debian Linux 11 on x86_64
Zeek version: 4.2.0
Spicy version: 1.3.0

[timwoj]

I'm not sure why this is triggering via a DPD change, but here's the backtrace for this at least. It's actually crashing on an assert in the hilti code that's part of spicy. I'll look into what causes the DPD change to take a different path here.

* thread #1, name = 'zeek', stop reason = hit program assert
    frame #0: 0x00007ffff5d7d03b libc.so.6`raise + 203
    frame #1: 0x00007ffff5d5c859 libc.so.6`abort + 299
    frame #2: 0x00007ffff5d5c729 libc.so.6`___lldb_unnamed_symbol2384 + 15
    frame #3: 0x00007ffff5d6e006 libc.so.6`__assert_fail + 70
  * frame #4: 0x00007fffefc63fc6 _Zeek-Spicy.linux-x86_64.so`hilti::rt::context::detail::get(allow_missing_context=false) at context.h:93:9
    frame #5: 0x00007fffefc64003 _Zeek-Spicy.linux-x86_64.so`hilti::rt::context::CookieSetter::CookieSetter(this=0x00007fffffffdad0, cookie=0x0000555558ff76b8) at context.h:141:27
    frame #6: 0x00007fffefc95a30 _Zeek-Spicy.linux-x86_64.so`spicy::zeek::rt::ProtocolAnalyzer::Finish(this=0x0000555558ff7600, is_orig=true) at protocol-analyzer.cc:83:59
    frame #7: 0x00007fffefc962d4 _Zeek-Spicy.linux-x86_64.so`spicy::zeek::rt::TCP_Analyzer::EndOfData(this=0x0000555558ff7600, is_orig=true) at protocol-analyzer.cc:165:11
    frame #8: 0x000055555690736a zeek`zeek::analyzer::Analyzer::Done(this=0x0000555558ff7600) at Analyzer.cc:188:12
    frame #9: 0x00007fffefc95f32 _Zeek-Spicy.linux-x86_64.so`spicy::zeek::rt::TCP_Analyzer::Done(this=0x0000555558ff7600) at protocol-analyzer.cc:126:57
    frame #10: 0x0000555556906fe3 zeek`zeek::analyzer::Analyzer::~Analyzer(this=0x000055555b9ac110) at Analyzer.cc:140:14
    frame #11: 0x0000555556c02752 zeek`zeek::packet_analysis::IP::SessionAdapter::~SessionAdapter(this=0x000055555b9ac110) at SessionAdapter.h:20:7
    frame #12: 0x0000555556c05720 zeek`zeek::packet_analysis::TCP::TCPSessionAdapter::~TCPSessionAdapter(this=0x000055555b9ac110) at TCPSessionAdapter.cc:48:39
    frame #13: 0x0000555556c05758 zeek`zeek::packet_analysis::TCP::TCPSessionAdapter::~TCPSessionAdapter(this=0x000055555b9ac110) at TCPSessionAdapter.cc:55:2
    frame #14: 0x000055555634e1b1 zeek`zeek::Connection::~Connection(this=0x000055555bd43ab0) at Conn.cc:88:9
    frame #15: 0x000055555634e236 zeek`zeek::Connection::~Connection(this=0x000055555bd43ab0) at Conn.cc:91:2
    frame #16: 0x00005555561b0121 zeek`zeek::Unref(o=0x000055555bd43ab0) at Obj.h:190:10
    frame #17: 0x000055555681c218 zeek`zeek::session::Manager::Clear(this=0x000055555b9ab4a0) at Manager.cc:216:8
    frame #18: 0x000055555681b9eb zeek`zeek::session::Manager::~Manager(this=0x000055555b9ab4a0) at Manager.cc:96:7
    frame #19: 0x00005555563158e5 zeek`zeek::detail::terminate_zeek() at zeek-setup.cc:352:9
    frame #20: 0x0000555556318f81 zeek`zeek::detail::cleanup(did_run_loop=true) at zeek-setup.cc:984:16
    frame #21: 0x0000555556c9efc4 zeek`main(argc=5, argv=0x00007fffffffe3a8) at main.cc:81:30
    frame #22: 0x00007ffff5d5e0b3 libc.so.6`__libc_start_main + 243
    frame #23: 0x00005555561af4be zeek`_start + 46

[bbannier]

I'll look into the assertion failure on the Spicy side (likely caused by code in zeek/spicy-plugin).

What is interesting is that if I run with -B dpd I hit a different SEGFAULT in the stringification of an analyzer name which probably indicates some state management issue:

thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x0000000109ef4650 zeek`zeek::Tag::operator<(zeek::Tag const&) const(this=0x400007f9834a449d, other=0x00007ffee60e3940) at Tag.h:141:10
    frame #1: 0x0000000109f96ef1 zeek`std::__1::less<zeek::Tag>::operator(this=0x00007f9834a235c8, __x=0x400007f9834a449d, __y=0x00007ffee60e3940)(zeek::Tag const&, zeek::Tag const&) const at __functional_base:54:21
    frame #2: 0x0000000109f96e95 zeek`std::__1::__map_value_compare<zeek::Tag, std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::less<zeek::Tag>, true>::operator(this=0x00007f9834a235c8, __x=0x400007f9834a449d, __y=0x00007ffee60e3940)(std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*> const&, zeek::Tag const&) const at map:516:17
    frame #3: 0x0000000109f96cae zeek`std::__1::__tree_const_iterator<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::__tree_node<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, void*>*, long> std::__1::__tree<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::__map_value_compare<zeek::Tag, std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::less<zeek::Tag>, true>, std::__1::allocator<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*> > >::__lower_bound<zeek::Tag>(this=0x00007f9834a235b8, __v=0x00007ffee60e3940, __root=0x400007f9834a447d, __result=0x00007f9834a235c0) const at __tree:2657:14
    frame #4: 0x0000000109f96b89 zeek`std::__1::__tree_const_iterator<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::__tree_node<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, void*>*, long> std::__1::__tree<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::__map_value_compare<zeek::Tag, std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*>, std::__1::less<zeek::Tag>, true>, std::__1::allocator<std::__1::__value_type<zeek::Tag, zeek::analyzer::Component*> > >::find<zeek::Tag>(this=0x00007f9834a235b8, __v=0x00007ffee60e3940) const at __tree:2577:26
    frame #5: 0x0000000109f96a8d zeek`std::__1::map<zeek::Tag, zeek::analyzer::Component*, std::__1::less<zeek::Tag>, std::__1::allocator<std::__1::pair<zeek::Tag const, zeek::analyzer::Component*> > >::find(this=0x00007f9834a235b8 size=69, __k=0x00007ffee60e3940) const at map:1380:68
    frame #6: 0x0000000109f96a0e zeek`zeek::plugin::ComponentManager<zeek::analyzer::Component>::Lookup(this=0x00007f9834a23560, tag=0x00007ffee60e3940) const at ComponentManager.h:231:73
    frame #7: 0x0000000109f93d5a zeek`zeek::plugin::ComponentManager<zeek::analyzer::Component>::GetComponentName(this=0x00007f9834a23560, tag=Tag @ 0x00007ffee60e3940) const at ComponentManager.h:196:9
    frame #8: 0x000000010a773abf zeek`zeek::analyzer::Analyzer::GetAnalyzerName(this=0x00007f98148048b0) const at Analyzer.cc:75:23
    frame #9: 0x000000010a7760e7 zeek`zeek::analyzer::Analyzer::fmt_analyzer(a=0x00007f98148048b0) at Analyzer.h:669:25
    frame #10: 0x000000010a777a4c zeek`zeek::analyzer::Analyzer::EndOfData(this=0x00007f98148048b0, is_orig=true) at Analyzer.cc:657:2
    frame #11: 0x0000000114d5bab7 _Zeek-Spicy.darwin-x86_64.so`spicy::zeek::rt::TCP_Analyzer::EndOfData(this=0x00007f98148048b0, is_orig=true) at protocol-analyzer.cc:158:53
    frame #12: 0x000000010a774a25 zeek`zeek::analyzer::Analyzer::Done(this=0x00007f98148048b0) at Analyzer.cc:188:3
    frame #13: 0x0000000114d5b726 _Zeek-Spicy.darwin-x86_64.so`spicy::zeek::rt::TCP_Analyzer::Done(this=0x00007f98148048b0) at protocol-analyzer.cc:126:53
    frame #14: 0x000000010a774449 zeek`zeek::analyzer::Analyzer::~Analyzer(this=0x00007f98238add30) at Analyzer.cc:140:10
    frame #15: 0x000000010a7796b8 zeek`zeek::analyzer::SupportAnalyzer::~SupportAnalyzer(this=0x00007f98238add30) at Analyzer.h:846:32
    frame #16: 0x000000010ad6b2f4 zeek`zeek::packet_analysis::TCP::TCPSessionAdapter::~TCPSessionAdapter(this=0x00007f98238add30) at TCPSessionAdapter.cc:55:2
    frame #17: 0x000000010ad6b405 zeek`zeek::packet_analysis::TCP::TCPSessionAdapter::~TCPSessionAdapter(this=0x00007f98238add30) at TCPSessionAdapter.cc:49:2
    frame #18: 0x000000010ad6b42c zeek`zeek::packet_analysis::TCP::TCPSessionAdapter::~TCPSessionAdapter(this=0x00007f98238add30) at TCPSessionAdapter.cc:49:2
    frame #19: 0x0000000109d07a48 zeek`zeek::Connection::~Connection(this=0x00007f98238ae3e0) at Conn.cc:88:2
    frame #20: 0x0000000109d07b15 zeek`zeek::Connection::~Connection(this=0x00007f98238ae3e0) at Conn.cc:79:2
    frame #21: 0x0000000109d07b3c zeek`zeek::Connection::~Connection(this=0x00007f98238ae3e0) at Conn.cc:79:2
    frame #22: 0x0000000109b21c74 zeek`zeek::Unref(o=0x00007f98238ae3e0) at Obj.h:190:3
    frame #23: 0x000000010a532ab9 zeek`zeek::session::Manager::Clear(this=0x00007f98146d6d20) at Manager.cc:216:3
    frame #24: 0x000000010a5329ec zeek`zeek::session::Manager::~Manager(this=0x00007f98146d6d20) at Manager.cc:96:2
    frame #25: 0x000000010a532b25 zeek`zeek::session::Manager::~Manager(this=0x00007f98146d6d20) at Manager.cc:95:2
    frame #26: 0x0000000109cb114e zeek`zeek::detail::terminate_zeek() at zeek-setup.cc:352:2
    frame #27: 0x0000000109cb08f9 zeek`zeek::detail::cleanup(did_run_loop=true) at zeek-setup.cc:963:2
    frame #28: 0x000000010aed0684 zeek`main(argc=7, argv=0x00007ffee60e4308) at main.cc:81:9
    frame #29: 0x00007fff71716cc9 libdyld.dylib`start + 1
    frame #30: 0x00007fff71716cc9 libdyld.dylib`start + 1

[bbannier]

The assertion checks that we still have a HILTI context around. It fails here since it is called via zeek::analyzer::Analyzer::Done after it was previously cleaned up via a call to zeek::plugin::Manager::FinishPlugins. I saw this after setting a breakpoint on hilti::rt::context::detail::set(hilti::rt::Context*) which is the function managing the HILTI context.

I am unsure on the plugin/analyzer state machine, should this work on the plugin side or is this an issue in Zeek?

[timwoj]

I'm not seeing this crash anymore on the latest master. I had to change the build command in the zkg.meta file to get it to install, but it's going through dpd just fine. Also, the included pcap is for port 10001, but the analyzer.evt specifies port 10002 (which doesn't appear in the pcap). I tested with analyzer.evt set to both 10001 and 10002 and the dpd only occurs with it set to 10001. Either way, it's not crashing.

[timwoj]

I'm going to close this one out as not reproducible. Feel free to reopen if it's still a problem.