Segfault in storage::segment_reader_handle::close() from tiered storage

[andrwng]

Version & Environment

Redpanda version: (use rpk version): v23.3.12 arm

We saw in one cluster several crashes that all appear to have similar root causes.

23.3.12-arm$ ~/Repos/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/scripts/seastar-addr2line  -e ~/xfs/23.3.12-arm/libexec/redpanda 0x7b61a43   0x7ba66f3   0x6dd5b6b   0x635ab43   0x2c6645f   0x7b5c36b   0x7b5ea1f   0x7b5cbf3   0x7a81e43   0x7a80997   0x2b56ca7   0x7df9163   0x2b4fd2f
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:824
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:854
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:866
seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4150
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4131
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4127
storage::segment_reader_handle::~segment_reader_handle() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/storage/segment_reader.cc:208
storage::continuous_batch_parser::~continuous_batch_parser() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/storage/parser.h:108
 (inlined by) std::__1::default_delete<storage::continuous_batch_parser>::operator()[abi:v160004](storage::continuous_batch_parser*) const at /vectorized/llvm/bin/../include/c++/v1/__memory/unique_ptr.h:65
 (inlined by) std::__1::unique_ptr<storage::continuous_batch_parser, std::__1::default_delete<storage::continuous_batch_parser> >::reset[abi:v160004](storage::continuous_batch_parser*) at /vectorized/llvm/bin/../include/c++/v1/__memory/unique_ptr.h:297
 (inlined by) cloud_storage::remote_segment_batch_reader::stop() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/cloud_storage/remote_segment.cc:1484

23.3.12-arm$ ~/Repos/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/scripts/seastar-addr2line  -e ~/xfs/23.3.12-arm/libexec/redpanda 0x7b61a43   0x7ba66f3   linux-vdso.so.1+0x84f   0x6dda8bf   0x2c6645f   0x7b5c36b   0x7b5ea1f   0x7ba7d83   0x7afca1b   /opt/redpanda/lib/libc.so.6+0x843b7   /opt/redpanda/lib/libc.so.6
+0xef2db
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:824
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:854
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:866
seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4150
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4131
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4127
addr2line: 'linux-vdso.so.1': No such file
linux-vdso.so.1 0x84f
boost::intrusive::list_node_traits<void*>::get_previous(boost::intrusive::list_node<void*>*) at /vectorized/include/boost/intrusive/detail/list_node.hpp:54
 (inlined by) boost::intrusive::circular_list_algorithms<boost::intrusive::list_node_traits<void*> >::unlink(boost::intrusive::list_node<void*>*) at /vectorized/include/boost/intrusive/circular_list_algorithms.hpp:153
 (inlined by) boost::intrusive::generic_hook<(boost::intrusive::algo_types)0, boost::intrusive::list_node_traits<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)2, (boost::intrusive::base_hook_type)0>::unlink() at /vectorized/include/boost/intrusive/detail/generic_hook.hpp:214
 (inlined by) storage::segment_reader_handle::close() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/storage/segment_reader.cc:218

23.3.12-arm$ ~/Repos/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/scripts/seastar-addr2line  -e ~/xfs/23.3.12-arm/libexec/redpanda 0x6dd385b   0x635aacf   0x2c6645f   0x7b5c36b   0x7b5ea1f   0x7b5cbf3   0x7a81e43   0x7a80997   0x2b56ca7   0x7df9163
[Backtrace #0]
boost::intrusive::list_node_traits<void*>::get_previous(boost::intrusive::list_node<void*>*) at /vectorized/include/boost/intrusive/detail/list_node.hpp:54
 (inlined by) boost::intrusive::circular_list_algorithms<boost::intrusive::list_node_traits<void*> >::unlink(boost::intrusive::list_node<void*>*) at /vectorized/include/boost/intrusive/circular_list_algorithms.hpp:153
 (inlined by) boost::intrusive::generic_hook<(boost::intrusive::algo_types)0, boost::intrusive::list_node_traits<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)2, (boost::intrusive::base_hook_type)0>::unlink() at /vectorized/include/boost/intrusive/detail/generic_hook.hpp:214
 (inlined by) storage::segment_reader_handle::close() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/storage/segment_reader.cc:218
storage::continuous_batch_parser::close() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/storage/parser.h:114
 (inlined by) cloud_storage::remote_segment_batch_reader::stop() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0831432e5e658e55f-1/redpanda/redpanda/src/v/cloud_storage/remote_segment.cc:1483

23.3.12-arm$ ~/Repos/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/scripts/seastar-addr2line  -e ~/xfs/23.3.12-arm/libexec/redpanda 0x7b61a43 0x7ba66f3 0x7e776ef
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:824
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:854
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:866
seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4150
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4131
 (inlined by) seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4127
$d.1462 at fetch.cc:?

From these, we can see:

• the crash is coming from a fetch handler (not timequery, compaction, uploader, or any other users of segment_readers)
• it appears to be caused by unsound lifecycle management of remote_segment_batch_reader
• sometimes the crash happens in the destructor ~segment_reader_handle() rather than in close(), further pointing to a lifecycle bug

One suspicious piece of code, from

redpanda/src/v/cloud_storage/remote_segment.cc

Line 1483 in 50b992b

co_await _parser->close();

it looks like stop() is not thread-safe. Multiple concurrent calls to stop() may call _parser->close() on the same parser and it to null. It's unclear if this is the exact failure path, given reader eviction happens from a single fiber, and set_end_of_stream() appears to be called at most once in do_load_slice().

Another suspicious piece of code from

redpanda/src/v/cloud_storage/remote_partition.cc

Lines 248 to 250 in 50b992b

	if (_seg_reader) {
	_partition->evict_segment_reader(std::move(_seg_reader));
	}

it looks like at any point, _segment_reader may be moved. In most places, we check if _segment_reader is a nullptr before using it, but it seems possible that e.g. here

redpanda/src/v/cloud_storage/remote_partition.cc

Lines 815 to 818 in 50b992b

	ss::future<> set_end_of_stream() {
	co_await _seg_reader->stop();
	_seg_reader = {};
	}

the scheduling point in the coroutine allows us to continue after the abort has wiped away _segment_reader.

JIRA Link: CORE-2750