mutation_reader_test.combined_mutation_reader_galloping_over_multiple_partitions_test exited with code -11(SIGSEGV)

[denesb]

Seen in https://jenkins.scylladb.com/job/scylla-master/job/next/5374/.

19:36:07  The following test(s) have failed: boost/mutation_reader_test
19:36:07  Output of build/release/test/boost/mutation_reader_test --run_test=combined_mutation_reader_galloping_over_multiple_partitions_test --report_level=no --logger=HRF,test_suite:XML,test_suite,/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/release/xml/mutation_reader_test.combined_mutation_reader_galloping_over_multiple_partitions_test.486.xunit.xml --catch_system_errors=no --color_output=false -- -c3 -m2G --overprovisioned --unsafe-bypass-fsync 1 --kernel-page-cache 1 --blocked-reactor-notify-ms 2000000 --collectd 0 --max-networking-io-control-blocks=100:
19:36:07  === TEST.PY STARTING TEST mutation_reader_test.combined_mutation_reader_galloping_over_multiple_partitions_test.486 ===
19:36:07  export UBSAN_OPTIONS='halt_on_error=1:abort_on_error=1:suppressions=/jenkins/workspace/scylla-master/next/scylla/ubsan-suppressions.supp'
19:36:07  export ASAN_OPTIONS='disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1'
19:36:07  build/release/test/boost/mutation_reader_test --run_test=combined_mutation_reader_galloping_over_multiple_partitions_test --report_level=no --logger=HRF,test_suite:XML,test_suite,/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/release/xml/mutation_reader_test.combined_mutation_reader_galloping_over_multiple_partitions_test.486.xunit.xml --catch_system_errors=no --color_output=false -- -c3 -m2G --overprovisioned --unsafe-bypass-fsync 1 --kernel-page-cache 1 --blocked-reactor-notify-ms 2000000 --collectd 0 --max-networking-io-control-blocks=100
19:36:07  === TEST.PY TEST mutation_reader_test.combined_mutation_reader_galloping_over_multiple_partitions_test.486 OUTPUT ===
19:36:07  Running 1 test case...
19:36:07  Entering test module "test/boost/mutation_reader_test.cc"
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_two_readers_with_the_same_row" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_two_non_overlapping_readers" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_two_partially_overlapping_readers" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_one_reader_with_many_partitions" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "combined_reader_galloping_within_partition_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Entering test case "combined_mutation_reader_galloping_over_multiple_partitions_test"
19:36:07  WARN  2022-10-25 18:40:37,357 [shard 0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
19:36:07  INFO  2022-10-25 18:40:37,379 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
19:36:07  INFO  2022-10-25 18:40:37,379 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0
19:36:07  INFO  2022-10-25 18:40:37,379 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
19:36:07  INFO  2022-10-25 18:40:37,379 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000
19:36:07  WARN  2022-10-25 18:40:37,381 [shard 1] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
19:36:07  WARN  2022-10-25 18:40:37,387 [shard 2] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
19:36:07  INFO  2022-10-25 18:40:37,394 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
19:36:07  INFO  2022-10-25 18:40:37,394 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0
19:36:07  INFO  2022-10-25 18:40:37,394 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
19:36:07  random-seed=776874329
19:36:07  test/boost/mutation_reader_test.cc(0): Leaving test case "combined_mutation_reader_galloping_over_multiple_partitions_test"; testing time: 256241us
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "combined_reader_galloping_changing_multiple_partitions_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combined_reader_range_tombstone_change_merging" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_filtering" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_two_readers_with_one_reader_empty" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_two_empty_readers" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combining_one_empty_reader" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_fast_forwarding_combining_reader" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_fast_forwarding_combining_reader_with_galloping" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_sm_fast_forwarding_combining_reader" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_sm_fast_forwarding_combining_reader_with_galloping" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "combined_mutation_reader_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "reader_selector_gap_between_readers_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "reader_selector_overlapping_readers_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "reader_selector_fast_forwarding_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_fast_forwarding_combined_reader_is_consistent_with_slicing" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combined_reader_slicing_with_overlapping_range_tombstones" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_combined_mutation_source_is_a_mutation_source" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_foreign_reader_as_mutation_source" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_trim_clustering_row_ranges_to" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_reading_empty_table" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_stopping_reader_with_pending_read_ahead" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_custom_shard_number" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_only_reads_from_needed_shards" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_destroyed_with_pending_read_ahead" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_fast_forwarded_with_pending_read_ahead" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_combining_reader_next_partition" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_multishard_streaming_reader" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_queue_reader" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_compacting_reader_as_mutation_source" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_compacting_reader_next_partition" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_compacting_reader_is_consistent_with_compaction" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_auto_paused_evictable_reader_is_mutation_source" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_manual_paused_evictable_reader_is_mutation_source" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_evictable_reader_self_validation" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_evictable_reader_recreate_before_fast_forward_to" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_evictable_reader_drop_flags" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_evictable_reader_non_monotonic_positions" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_evictable_reader_clear_tombstone_in_discontinued_partition" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_clustering_order_merger_in_memory" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_clustering_order_merger_sstable_set" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_clustering_order_merger_sstable_set_reversed" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "clustering_combined_reader_mutation_source_test" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_clustering_combining_of_empty_readers" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_generating_reader_v1" is skipped because disabled
19:36:07  test/boost/mutation_reader_test.cc(0): Test case "test_generating_reader_v2" is skipped because disabled
19:36:07  Leaving test module "test/boost/mutation_reader_test.cc"; testing time: 256903us
19:36:07  === TEST.PY SUMMARY START ===
19:36:07  Test exited with code -11
19:36:07  
19:36:07  === TEST.PY SUMMARY END ===
19:36:07  
19:36:07  ------------------------------------------------------------------------------
19:36:07  Summary: 1 of the total 7581 tests failed

There is no details whatsoever here. Also, it looks like the crash happened in infrastructure code (most likely boost::test), the test successfully finishes and the crash happens after boost have iterated over and skipped the remaining test cases in the file.

[denesb]

The crash didn't reproduce since. I don't think we have anything to go on here. Closing, will reopen if we see it again.

[denesb]

Seen again in https://jenkins.scylladb.com/job/releng/job/Scylla-CI/3687/artifact/testlog/x86_64/dev/boost.mutation_reader_test.combined_reader_galloping_within_partition_test.1.log.