ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes #9933

yarongilor · 2022-01-17T09:23:25Z

Installation details
Kernel version: 5.4.0-1035-aws
Scylla version (or git commit hash): 4.6.rc2-0.20220102.e8a1cfb6f with build-id 5d7b96e39c909424e8224207a162fc2c82b67214
Cluster size: 4 nodes (i3en.3xlarge)
Scylla running with shards number (live nodes):
longevity-large-partitions-4d-4-6-db-node-3bce3643-1 (13.51.176.199 | 10.0.1.153): 12 shards
longevity-large-partitions-4d-4-6-db-node-3bce3643-2 (13.53.118.234 | 10.0.2.180): 12 shards
longevity-large-partitions-4d-4-6-db-node-3bce3643-3 (13.49.69.157 | 10.0.2.208): 12 shards
longevity-large-partitions-4d-4-6-db-node-3bce3643-4 (13.49.80.203 | 10.0.2.19): 12 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-04034a8a446f92efc (aws: eu-north-1)

Test: longevity-large-partition-4days-test
Test name: longevity_large_partition_test.LargePartitionLongevityTest.test_large_partition_longevity
Test config file(s):

longevity-large-partition-4days.yaml

Issue description

====================================

cluster had scylla-bench stress of large partitions running.
A modify_table SCT nemesis issue a command of: < t:2022-01-16 19:53:24,347 f:nemesis.py l:1352 c:sdcm.nemesis p:DEBUG > sdcm.nemesis.SisyphusMonkey: _modify_table_property: ALTER TABLE scylla_bench.test WITH compression = {'sstable_compression': 'ZstdCompressor', 'chunk_length_kb': '1M', 'crc_check_chance': 0.042447059711723134};
==> 3 minutes afterwards, 3 of the nodes started getting core dumps:

< t:2022-01-16 19:53:15,004 f:nemesis.py      l:1209 c:sdcm.nemesis         p:INFO  > sdcm.nemesis.SisyphusMonkey: >>>>>>>>>>>>>Started random_disrupt_method modify_table
< t:2022-01-16 19:53:23,838 f:nemesis.py      l:791  c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: Set current_disruption -> ModifyTable Node longevity-large-partitions-4d-4-6-db-node-3bce3643-3 [13.49.69.157 | 10.0.2.208] (seed: False)
< t:2022-01-16 19:59:09,299 f:nemesis.py      l:1219 c:sdcm.nemesis         p:INFO  > sdcm.nemesis.SisyphusMonkey: <<<<<<<<<<<<<Finished random_disrupt_method modify_table
yarongilor@yaron-pc:~/Downloads/logs/sct-runner-3bce3643$ grep CoreDumpEvent error.log 
2022-01-16 19:56:20.051: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=57b8f322-71dd-475e-9dcc-859efad460d7 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-1 [13.51.176.199 | 10.0.1.153] (seed: True)
2022-01-16 19:56:37.801: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=911a21a3-6403-4a7f-a089-7a7bf682d39b node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-2 [13.53.118.234 | 10.0.2.180] (seed: False)
2022-01-16 19:58:17.113: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=fe616ec2-748d-4a18-9ec0-b3d77815c14e node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-4 [13.49.80.203 | 10.0.2.19] (seed: False)
2022-01-16 20:01:22.111: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=b8636693-409c-4c09-b9d7-0b43d1673644 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-1 [13.51.176.199 | 10.0.1.153] (seed: True)
2022-01-16 20:04:13.578: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=c7097b79-0afe-4543-b1dd-e38a411095c8 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-4 [13.49.80.203 | 10.0.2.19] (seed: False)
2022-01-16 20:04:45.158: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=1b1cb57b-6848-4fe3-93b5-882b254deef7 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-2 [13.53.118.234 | 10.0.2.180] (seed: False)

core dump details:

(CoreDumpEvent Severity.ERROR) period_type=one-time event_id=1b1cb57b-6848-4fe3-93b5-882b254deef7 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-2 [13.53.118.234 | 10.0.2.180] (seed: False) corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000.gz backtrace= PID: 12604 (scylla) UID: 113 (scylla) GID: 119 (scylla) Signal: 6 (ABRT) Timestamp: Sun 2022-01-16 19:53:33 UTC (3min 34s ago) Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-11 --lock-memory=1 Executable: /opt/scylladb/libexec/scylla Control Group: /scylla.slice/scylla-server.slice/scylla-server.service Unit: scylla-server.service Slice: scylla-server.slice Boot ID: 8e1bf2c1618b4fc28b1b454a71f10016 Machine ID: 6fa2171583af4b13bec1e274475e7467 Hostname: longevity-large-partitions-4d-4-6-db-node-3bce3643-2 Storage: /var/lib/systemd/coredump/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000 Message: Process 12604 (scylla) of user 113 dumped core. Stack trace of thread 12615: #0 0x00007f3941bb52a2 raise (libc.so.6 + 0x3d2a2) #1 0x00007f3941b9e950 abort (libc.so.6 + 0x26950) #2 0x00000000013f523f _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f523f) #3 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #4 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12622: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12621: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12616: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12617: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12618: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12620: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12610: #0 0x00000000012cfc84 _ZNK10bound_view11tri_compareclERK21clustering_key_prefixiS3_i (scylla + 0x10cfc84) #1 0x0000000001473bac _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bac) #2 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #3 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #4 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #5 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #6 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #7 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #8 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #9 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #10 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #11 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #12 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #13 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #14 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #15 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #16 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #17 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #18 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12626: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12619: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12624: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12604: #0 0x0000000001473bdb _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bdb) #1 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #2 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #3 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #4 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003dc74fd _ZN7seastar7reactor3runEv (scylla + 0x3bc74fd) #15 0x0000000003d70d8f _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3b70d8f) #16 0x0000000003d70107 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3b70107) #17 0x0000000000f406b6 main (scylla + 0xd406b6) #18 0x00007f3941b9fb75 __libc_start_main (libc.so.6 + 0x27b75) #19 0x0000000000f3d86e _start (scylla + 0xd3d86e) Stack trace of thread 12605: #0 0x00007f3941cdb9d1 __memmove_avx_unaligned_erms (libc.so.6 + 0x1639d1) #1 0x000000000124bbf9 _ZNK25atomic_cell_or_collection4copyERK13abstract_type (scylla + 0x104bbf9) #2 0x00000000014dc96d _ZNK18compact_radix_tree4treeI13cell_and_hashjE9node_head5cloneIRZN3rowC1ERK6schema11column_kindRKS5_E4$_14EESt4pairIPS3_NSt15__exception_ptr13exception_ptrEEOT_j (scylla + 0x12dc96d) #3 0x00000000014624a4 _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x12624a4) #4 0x00000000014b9c7b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9c7b) #5 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #6 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #7 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #8 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #9 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #10 0x0000000001461571 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x1261571) #11 0x00000000012ed11f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed11f) #12 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #13 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #14 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #15 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #16 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #17 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #18 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #19 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #20 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #21 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #22 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #23 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #24 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #25 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12625: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12623: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12611: #0 0x0000000001473b89 _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273b89) #1 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #2 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #3 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #4 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12606: #0 0x0000000001473b94 _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273b94) #1 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #2 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #3 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #4 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12607: #0 0x00000000020583e1 _ZL5visitIN12_GLOBAL__N_115compare_visitorEENSt13invoke_resultIT_JRK15ascii_type_implEE4typeERK13abstract_typeOS3_ (scylla + 0x1e583e1) #1 0x000000000202e32d _ZNK13abstract_type7compareE24managed_bytes_basic_viewIL12mutable_view0EES2_ (scylla + 0x1e2e32d) #2 0x00000000012cfcb1 _ZNK10bound_view11tri_compareclERK21clustering_key_prefixiS3_i (scylla + 0x10cfcb1) #3 0x0000000001473bac _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bac) #4 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #5 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #6 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #7 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #8 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #9 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #10 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #11 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #12 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #13 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #14 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #15 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #16 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #17 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #18 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #19 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #20 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12614: #0 0x00000000014e1cc1 _ZN18compact_radix_tree4treeI13cell_and_hashjE14weed_from_slotIRZN3row9remove_ifIZNS4_18compact_and_expireERK6schema11column_kind13row_tombstoneNSt6chrono10time_pointI8gc_clockNSB_8durationIlSt5ratioILl1ELl1EEEEEERSt8functionIFb9tombstoneEESI_RK10row_markerP28compaction_garbage_collectorE4$_18EEvOT_EUljRS1_E_EEbPNS2_13node_head_ptrESV_j (scylla + 0x12e1cc1) #1 0x0000000001486406 _ZN3row18compact_and_expireERK6schema11column_kind13row_tombstoneNSt6chrono10time_pointI8gc_clockNS5_8durationIlSt5ratioILl1ELl1EEEEEERSt8functionIFb9tombstoneEESC_RK10row_markerP28compaction_garbage_collector (scylla + 0x1286406) #2 0x000000000198f64c _ZN22compact_mutation_stateIL19emit_only_live_rows0EL20compact_for_sstables1EE7consumeIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEEN7seastar10bool_classINS7_18stop_iteration_tagEEEO14clustering_rowRT_RT0_ (scylla + 0x178f64c) #3 0x000000000198d805 _ZNO17mutation_fragment7consumeIN20flat_mutation_reader4impl16consumer_adapterI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEEEEEEDcRT_ (scylla + 0x178d805) #4 0x000000000198b99d _ZZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEEN7seastar6futureIvEET_ENUl20flat_mutation_readerE_clES7_ENUlvE_clEv (scylla + 0x178b99d) #5 0x000000000198b5b4 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIvEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE_E4callEPKS2_ (scylla + 0x178b5b4) #6 0x0000000004037d22 _ZN7seastar14thread_context4mainEv (scylla + 0x3e37d22) Stack trace of thread 12613: #0 0x00007f3941cdb951 __memmove_avx_unaligned_erms (libc.so.6 + 0x163951) #1 0x000000000124bbf9 _ZNK25atomic_cell_or_collection4copyERK13abstract_type (scylla + 0x104bbf9) #2 0x00000000014dc96d _ZNK18compact_radix_tree4treeI13cell_and_hashjE9node_head5cloneIRZN3rowC1ERK6schema11column_kindRKS5_E4$_14EESt4pairIPS3_NSt15__exception_ptr13exception_ptrEEOT_j (scylla + 0x12dc96d) #3 0x00000000014624a4 _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x12624a4) #4 0x00000000014b9c7b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9c7b) #5 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #6 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #7 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #8 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #9 0x0000000001461571 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x1261571) #10 0x00000000012ed11f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed11f) #11 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #12 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #13 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #14 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #15 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #16 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #17 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #18 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #19 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #20 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #21 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #22 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #23 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #24 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12612: #0 0x0000000001473bdf _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bdf) #1 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #2 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #3 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #4 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12608: #0 0x00000000014bad5f _ZN11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5clearIRZNKS5_5cloneIRZN18mutation_partitionC1ERK6schemaRKS8_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSK_SL_OT_OT0_EUlPNS_11member_hookEE_EEvSN_ (scylla + 0x12bad5f) #1 0x00000000014bac9d _ZN11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5clearIRZNKS5_5cloneIRZN18mutation_partitionC1ERK6schemaRKS8_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSK_SL_OT_OT0_EUlPNS_11member_hookEE_EEvSN_ (scylla + 0x12bac9d) #2 0x00000000014bac9d _ZN11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5clearIRZNKS5_5cloneIRZN18mutation_partitionC1ERK6schemaRKS8_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSK_SL_OT_OT0_EUlPNS_11member_hookEE_EEvSN_ (scylla + 0x12bac9d) #3 0x00000000014bac9d _ZN11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5clearIRZNKS5_5cloneIRZN18mutation_partitionC1ERK6schemaRKS8_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSK_SL_OT_OT0_EUlPNS_11member_hookEE_EEvSN_ (scylla + 0x12bac9d) #4 0x00000000014ba2e0 _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12ba2e0) #5 0x0000000001461571 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x1261571) #6 0x00000000012ed11f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed11f) #7 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #8 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #9 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #10 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #11 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #12 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #13 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #14 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #15 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #16 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #17 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #18 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #19 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #20 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12627: #0 0x00007f394276f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) Stack trace of thread 12609: #0 0x00007f3941c72e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f3942766299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f3941c78353 __clone (libc.so.6 + 0x100353) download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000.gz . gunzip /var/lib/systemd/coredump/core.scylla.113.8e1bf2c1618b4fc28b1b454a71f10016.12604.1642362813000000000000.gz 2022-01-16 20:04:45
CoreDumpEvent
ERROR
events
(CoreDumpEvent Severity.ERROR) period_type=one-time event_id=c7097b79-0afe-4543-b1dd-e38a411095c8 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-4 [13.49.80.203 | 10.0.2.19] (seed: False) corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000.gz backtrace= PID: 17520 (scylla) UID: 113 (scylla) GID: 119 (scylla) Signal: 6 (ABRT) Timestamp: Sun 2022-01-16 19:56:00 UTC (2min 47s ago) Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-11 --lock-memory=1 Executable: /opt/scylladb/libexec/scylla Control Group: /scylla.slice/scylla-server.slice/scylla-server.service Unit: scylla-server.service Slice: scylla-server.slice Boot ID: 9fef050e71114af7aa4c9e0d55c24f15 Machine ID: 6fa2171583af4b13bec1e274475e7467 Hostname: longevity-large-partitions-4d-4-6-db-node-3bce3643-4 Storage: /var/lib/systemd/coredump/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000 Message: Process 17520 (scylla) of user 113 dumped core. Stack trace of thread 17531: #0 0x00007fb1f6b552a2 raise (libc.so.6 + 0x3d2a2) #1 0x00007fb1f6b3e950 abort (libc.so.6 + 0x26950) #2 0x00000000013f523f _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f523f) #3 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #4 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17532: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17535: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17534: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17538: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17536: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17530: #0 0x000000000160d250 _ZN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEvEUlvE_ZNS_6futureIvE14then_impl_nrvoIS6_S8_EET0_OT_EUlOS3_RS6_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x140d250) #1 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #2 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #3 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #4 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #5 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #6 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17537: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17542: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17543: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17540: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17525: #0 0x00000000012cfc84 _ZNK10bound_view11tri_compareclERK21clustering_key_prefixiS3_i (scylla + 0x10cfc84) #1 0x0000000001473bac _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bac) #2 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #3 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #4 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #5 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #6 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #7 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #8 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #9 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #10 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #11 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #12 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #13 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #14 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #15 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #16 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #17 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #18 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17524: #0 0x00000000012cfc89 _ZNK10bound_view11tri_compareclERK21clustering_key_prefixiS3_i (scylla + 0x10cfc89) #1 0x0000000001473bac _ZN18mutation_partition13clustered_rowERK6schema26position_in_partition_viewN7seastar10bool_classI9dummy_tagEENS5_I14continuous_tagEE (scylla + 0x1273bac) #2 0x0000000001549b50 _ZN37converting_mutation_partition_applier10accept_rowE26position_in_partition_viewRK13row_tombstoneRK10row_markerN7seastar10bool_classI9dummy_tagEENS8_I14continuous_tagEE (scylla + 0x1349b50) #3 0x000000000146c88b _ZN18mutation_partition7upgradeERK6schemaS2_ (scylla + 0x126c88b) #4 0x00000000012ed14d _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed14d) #5 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #6 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #7 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #8 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #9 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #10 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #11 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #12 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #13 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #14 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #15 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #16 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #17 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #18 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17523: #0 0x0000000003d84076 _ZN7seastar15basic_semaphoreINS_35semaphore_default_exception_factoryENSt6chrono3_V212steady_clockEE6signalEm (scylla + 0x3b84076) #1 0x0000000003dc9bd2 _ZN7seastar18syscall_work_queue8completeEv (scylla + 0x3bc9bd2) #2 0x0000000003e018d5 _ZN7seastar7reactor14syscall_pollfn4pollEv (scylla + 0x3c018d5) #3 0x0000000003de49c9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3be49c9) #4 0x0000000003dc82e6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82e6) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17527: #0 0x0000000003014c66 _ZN7service5pager11query_pager20fetch_page_generatorEjNSt6chrono10time_pointI8gc_clockNS2_8durationIlSt5ratioILl1ELl1EEEEEENS3_IN7seastar12lowres_clockENS5_IlS6_ILl1ELl1000EEEEEERN4cql39cql_statsE (scylla + 0x2e14c66) #1 0x0000000001df59a5 _ZNK4cql310statements16select_statement10do_executeERN7service13storage_proxyERNS2_11query_stateERKNS_13query_optionsE (scylla + 0x1bf59a5) #2 0x0000000001e950e8 _ZN7seastar20noncopyable_functionIFNS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE17direct_vtable_forISt7_Mem_fnIMSA_KFS7_SF_SH_SK_EEE4callEPKSM_SC_SF_SH_SK_ (scylla + 0x1c950e8) #3 0x0000000001e9512d _ZN7seastar20noncopyable_functionIFNS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS7_JSC_SF_SH_SK_EE20make_stage_for_groupENS_16scheduling_groupEEUlSC_SF_SH_SK_E_E4callEPKSM_SC_SF_SH_SK_ (scylla + 0x1c9512d) #4 0x0000000001e70f05 _ZN7seastar24concrete_execution_stageINS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEJPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE8do_flushEv (scylla + 0x1c70f05) #5 0x0000000003d7c8e6 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_5E15run_and_disposeEv (scylla + 0x3b7c8e6) #6 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #7 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #8 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #9 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #10 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #11 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17533: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17520: #0 0x00000000023ea103 _ZNK4cql38metadata5flagsEv (scylla + 0x21ea103) #1 0x0000000001abe5c2 _ZN13cql_transport8response5writeERKN4cql38metadataEb (scylla + 0x18be5c2) #2 0x0000000001ad5654 _ZN13cql_transport10cql_server11fmt_visitor5visitERKNS_8messages14result_message4rowsE (scylla + 0x18d5654) #3 0x0000000001aba318 _ZN13cql_transport11make_resultEsRNS_8messages14result_messageERKN7tracing15trace_state_ptrEhb (scylla + 0x18ba318) #4 0x0000000001ad0164 _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt7variantIJNS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS7_EEEENS_10shared_ptrINS6_8messages14result_message15bounce_to_shardEEEEEEEZNS6_L24process_execute_internalERN7service12client_stateERNS_7shardedIN4cql315query_processorEEENS6_14request_readerEth24cql_serialization_format14service_permitN7tracing15trace_state_ptrEbSt13unordered_mapIhSt8optionalINS_13basic_sstringIajLj31ELb0EEEESt4hashIhESt8equal_toIhESaISt4pairIKhS10_EEEE4$_14ZNS_6futureINSC_ISE_EEE14then_impl_nrvoIS1A_NS1B_ISH_EEEET0_OT_EUlOSI_RS1A_ONS_12future_stateIS1C_EEE_S1C_E15run_and_disposeEv (scylla + 0x18d0164) #5 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #6 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #7 0x0000000003dc74fd _ZN7seastar7reactor3runEv (scylla + 0x3bc74fd) #8 0x0000000003d70d8f _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3b70d8f) #9 0x0000000003d70107 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3b70107) #10 0x0000000000f406b6 main (scylla + 0xd406b6) #11 0x00007fb1f6b3fb75 __libc_start_main (libc.so.6 + 0x27b75) #12 0x0000000000f3d86e _start (scylla + 0xd3d86e) Stack trace of thread 17529: #0 0x00007ffe9bd17794 n/a (linux-vdso.so.1 + 0x794) #1 0x00007ffe9bd17918 __vdso_clock_gettime (linux-vdso.so.1 + 0x918) #2 0x00007fb1f6bdfa25 __clock_gettime (libc.so.6 + 0xc7a25) #3 0x00007fb1f6dd2de5 _ZNSt6chrono3_V212steady_clock3nowEv (libstdc++.so.6 + 0xd0de5) #4 0x0000000003dc8311 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc8311) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17528: #0 0x00007fb1f6c12e0d syscall (libc.so.6 + 0xfae0d) #1 0x000000000403095d _ZN7seastar29try_systemwide_memory_barrierEv (scylla + 0x3e3095d) #2 0x0000000003e01436 _ZN7seastar7reactor10smp_pollfn24try_enter_interrupt_modeEv (scylla + 0x3c01436) #3 0x0000000003dc85da _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85da) #4 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #5 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #6 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #7 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17526: #0 0x00000000011b53b2 _ZN13bytes_ostream9alloc_newEj (scylla + 0xfb53b2) #1 0x0000000002a60377 _ZN5query6resultC1Ev (scylla + 0x2860377) #2 0x00000000021ddd8b _ZN7service13storage_proxy8do_queryEN7seastar13lw_shared_ptrIK6schemaEENS2_IN5query12read_commandEEEOSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISD_EEN2db17consistency_levelENS0_25coordinator_query_optionsE (scylla + 0x1fddd8b) #3 0x00000000021dc7c9 _ZN7service13storage_proxy5queryEN7seastar13lw_shared_ptrIK6schemaEENS2_IN5query12read_commandEEEOSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISD_EEN2db17consistency_levelENS0_25coordinator_query_optionsE (scylla + 0x1fdc7c9) #4 0x000000000300ec56 _ZN7service5pager11query_pager13do_fetch_pageEjNSt6chrono10time_pointI8gc_clockNS2_8durationIlSt5ratioILl1ELl1EEEEEENS3_IN7seastar12lowres_clockENS5_IlS6_ILl1ELl1000EEEEEE (scylla + 0x2e0ec56) #5 0x00000000030148ca _ZN7service5pager11query_pager20fetch_page_generatorEjNSt6chrono10time_pointI8gc_clockNS2_8durationIlSt5ratioILl1ELl1EEEEEENS3_IN7seastar12lowres_clockENS5_IlS6_ILl1ELl1000EEEEEERN4cql39cql_statsE (scylla + 0x2e148ca) #6 0x0000000001df59a5 _ZNK4cql310statements16select_statement10do_executeERN7service13storage_proxyERNS2_11query_stateERKNS_13query_optionsE (scylla + 0x1bf59a5) #7 0x0000000001e950e8 _ZN7seastar20noncopyable_functionIFNS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE17direct_vtable_forISt7_Mem_fnIMSA_KFS7_SF_SH_SK_EEE4callEPKSM_SC_SF_SH_SK_ (scylla + 0x1c950e8) #8 0x0000000001e9512d _ZN7seastar20noncopyable_functionIFNS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS7_JSC_SF_SH_SK_EE20make_stage_for_groupENS_16scheduling_groupEEUlSC_SF_SH_SK_E_E4callEPKSM_SC_SF_SH_SK_ (scylla + 0x1c9512d) #9 0x0000000001e70f05 _ZN7seastar24concrete_execution_stageINS_6futureINS_10shared_ptrIN13cql_transport8messages14result_messageEEEEEJPKN4cql310statements16select_statementERN7service13storage_proxyERNSD_11query_stateERKNS8_13query_optionsEEE8do_flushEv (scylla + 0x1c70f05) #10 0x0000000003d7c8e6 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_5E15run_and_disposeEv (scylla + 0x3b7c8e6) #11 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #12 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #13 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #14 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #15 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #16 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17539: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17522: #0 0x00000000014dc936 _ZNK18compact_radix_tree4treeI13cell_and_hashjE9node_head5cloneIRZN3rowC1ERK6schema11column_kindRKS5_E4$_14EESt4pairIPS3_NSt15__exception_ptr13exception_ptrEEOT_j (scylla + 0x12dc936) #1 0x00000000014624a4 _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x12624a4) #2 0x00000000014b9c7b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9c7b) #3 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #4 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #5 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #6 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #7 0x0000000001461571 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x1261571) #8 0x00000000012ed11f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed11f) #9 0x0000000001387109 _ZN8memtable16make_flat_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x1187109) #10 0x0000000001130a1f _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30a1f) #11 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #12 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #13 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #14 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #15 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #16 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #17 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #18 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #19 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #20 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #21 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17541: #0 0x00007fb1f770f94c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) Stack trace of thread 17521: #0 0x0000000001aa17a4 _ZN13cql_transport10cql_server10connection7processIPFN7seastar6futureISt7variantIJNS3_11foreign_ptrISt10unique_ptrINS_8responseESt14default_deleteIS8_EEEENS3_10shared_ptrINS_8messages14result_message15bounce_to_shardEEEEEEERN7service12client_stateERNS3_7shardedIN4cql315query_processorEEENS_14request_readerEth24cql_serialization_format14service_permitN7tracing15trace_state_ptrEbSt13unordered_mapIhSt8optionalINS3_13basic_sstringIajLj31ELb0EEEESt4hashIhESt8equal_toIhESaISt4pairIKhS11_EEEEEENS4_ISC_EEtSS_SM_SU_SW_T_ (scylla + 0x18a17a4) #1 0x0000000001aa54fb _ZN13cql_transport10cql_server10connection15process_executeEtNS_14request_readerERN7service12client_stateE14service_permitN7tracing15trace_state_ptrE (scylla + 0x18a54fb) #2 0x0000000001a91877 _ZN13cql_transport10cql_server10connection19process_request_oneEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENS1_20tracing_request_typeE14service_permit (scylla + 0x1891877) #3 0x0000000001ae7374 _ZSt13__invoke_implIN7seastar6futureINS0_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEERKMNS4_10cql_server10connectionEFSA_N27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEPSC_JSE_htSH_SI_SJ_EET_St21__invoke_memfun_derefOT0_OT1_DpOT2_ (scylla + 0x18e7374) #4 0x0000000001ae72c6 _ZN7seastar20noncopyable_functionIFNS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEE17direct_vtable_forISt7_Mem_fnIMSC_FSA_SF_htSI_SJ_SK_EEE4callEPKSM_SD_SF_htSI_SJ_SK_ (scylla + 0x18e72c6) #5 0x0000000001ac5251 _ZNK7seastar20noncopyable_functionIFNS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEEclESD_SF_htSI_SJ_SK_ (scylla + 0x18c5251) #6 0x0000000001af4abf _ZZN7seastar35inheriting_concrete_execution_stageINS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEJPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEE20make_stage_for_groupENS_16scheduling_groupEENKUlSD_SF_htSI_SJ_SK_E_clESD_SF_htSI_SJ_SK_ (scylla + 0x18f4abf) #7 0x0000000001af43bc _ZN7seastar20noncopyable_functionIFNS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageISA_JSD_SF_htSI_SJ_SK_EE20make_stage_for_groupENS_16scheduling_groupEEUlSD_SF_htSI_SJ_SK_E_E4callEPKSM_SD_SF_htSI_SJ_SK_ (scylla + 0x18f43bc) #8 0x0000000001ac5251 _ZNK7seastar20noncopyable_functionIFNS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEEclESD_SF_htSI_SJ_SK_ (scylla + 0x18c5251) #9 0x0000000001ac4b44 _ZSt13__invoke_implIN7seastar6futureINS0_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEERNS0_20noncopyable_functionIFSA_PNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSD_20tracing_request_typeE14service_permitEEEJSE_SG_htSJ_SK_SL_EET_St14__invoke_otherOT0_DpOT1_ (scylla + 0x18c4b44) #10 0x0000000001ac38e5 _ZN7seastar24concrete_execution_stageINS_6futureINS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS5_EEEEEEJPNS4_10cql_server10connectionEN27fragmented_temporary_buffer7istreamEhtRN7service12client_stateENSC_20tracing_request_typeE14service_permitEE8do_flushEv (scylla + 0x18c38e5) #11 0x0000000003d7c8e6 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_5E15run_and_disposeEv (scylla + 0x3b7c8e6) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007fb1f7706299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007fb1f6c18353 __clone (libc.so.6 + 0x100353) download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000.gz . gunzip /var/lib/systemd/coredump/core.scylla.113.9fef050e71114af7aa4c9e0d55c24f15.17520.1642362960000000000000.gz 2022-01-16 20:04:13
CoreDumpEvent
ERROR
events
(CoreDumpEvent Severity.ERROR) period_type=one-time event_id=b8636693-409c-4c09-b9d7-0b43d1673644 node=Node longevity-large-partitions-4d-4-6-db-node-3bce3643-1 [13.51.176.199 | 10.0.1.153] (seed: True) corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000.gz backtrace= PID: 11544 (scylla) UID: 113 (scylla) GID: 119 (scylla) Signal: 6 (ABRT) Timestamp: Sun 2022-01-16 19:53:32 UTC (3min 18s ago) Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-11 --lock-memory=1 Executable: /opt/scylladb/libexec/scylla Control Group: /scylla.slice/scylla-server.slice/scylla-server.service Unit: scylla-server.service Slice: scylla-server.slice Boot ID: a4f164f79f2249cb989230903e6e273a Machine ID: 6fa2171583af4b13bec1e274475e7467 Hostname: longevity-large-partitions-4d-4-6-db-node-3bce3643-1 Storage: /var/lib/systemd/coredump/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000 Message: Process 11544 (scylla) of user 113 dumped core. Stack trace of thread 11555: #0 0x00007f1e1011a2a2 raise (libc.so.6 + 0x3d2a2) #1 0x00007f1e10103950 abort (libc.so.6 + 0x26950) #2 0x00000000013f523f _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f523f) #3 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #4 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #5 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #6 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #7 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #8 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #9 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #10 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #11 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #12 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #13 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #14 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #15 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #16 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #17 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11556: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11563: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11560: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11565: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11561: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11564: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11562: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11559: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11547: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11546: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11552: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11553: #0 0x00007f1e1024089d __memmove_avx_unaligned_erms (libc.so.6 + 0x16389d) #1 0x000000000124bbf9 _ZNK25atomic_cell_or_collection4copyERK13abstract_type (scylla + 0x104bbf9) #2 0x00000000014dc96d _ZNK18compact_radix_tree4treeI13cell_and_hashjE9node_head5cloneIRZN3rowC1ERK6schema11column_kindRKS5_E4$_14EESt4pairIPS3_NSt15__exception_ptr13exception_ptrEEOT_j (scylla + 0x12dc96d) #3 0x00000000014624a4 _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x12624a4) #4 0x00000000014b9c7b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9c7b) #5 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #6 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #7 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #8 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #9 0x00000000014b9cca _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x12b9cca) #10 0x0000000001461571 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x1261571) #11 0x00000000012ed11f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x10ed11f) #12 0x000000000130ad5e _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110ad5e) #13 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #14 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #15 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #16 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #17 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #18 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #19 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #20 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #21 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #22 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #23 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #24 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #25 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11554: #0 0x0000000001314ee3 _ZN10rows_entry10on_evictedER13cache_tracker (scylla + 0x1114ee3) #1 0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473) #2 0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de) #3 0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539) #4 0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4) #5 0x00000000013d49bc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x11d49bc) #6 0x00000000013f3158 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f3158) #7 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #8 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #9 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #10 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #11 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #12 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #13 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #14 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #15 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #16 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #17 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #18 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #19 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #20 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #21 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11557: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11545: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11567: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11550: #0 0x000000000140a060 _ZN8logalloc11region_impl4freeEPvm (scylla + 0x120a060) #1 0x000000000148558d _ZN3rowD1Ev (scylla + 0x128558d) #2 0x0000000001314e7c _ZN10rows_entry10on_evictedER13cache_tracker (scylla + 0x1114e7c) #3 0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473) #4 0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de) #5 0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539) #6 0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4) #7 0x00000000013d49bc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x11d49bc) #8 0x00000000013f3158 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f3158) #9 0x00000000012fac14 _ZZN5cache26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEv (scylla + 0x10fac14) #10 0x00000000012f9c61 _ZN5cache26cache_flat_mutation_reader11fill_bufferEv (scylla + 0x10f9c61) #11 0x000000000154ba42 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x134ba42) #12 0x000000000154ca4f _ZN22mutation_reader_mergerclEv (scylla + 0x134ca4f) #13 0x0000000001584341 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x1384341) #14 0x00000000015854c1 _ZN7seastar8internal8repeaterIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_E15run_and_disposeEv (scylla + 0x13854c1) #15 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #16 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #17 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #18 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #19 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #20 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11544: #0 0x0000000001409ef7 _ZN8logalloc11region_impl4freeEPvm (scylla + 0x1209ef7) #1 0x00000000012f8fcc _ZN11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE10remove_keyEm (scylla + 0x10f8fcc) #2 0x0000000001314e49 _ZN10rows_entry10on_evictedER13cache_tracker (scylla + 0x1114e49) #3 0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473) #4 0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de) #5 0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539) #6 0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4) #7 0x00000000013d49bc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x11d49bc) #8 0x00000000013f3158 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f3158) #9 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #10 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #11 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #12 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #13 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #14 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #15 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #16 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #17 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #18 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #19 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #20 0x0000000003dc74fd _ZN7seastar7reactor3runEv (scylla + 0x3bc74fd) #21 0x0000000003d70d8f _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3b70d8f) #22 0x0000000003d70107 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3b70107) #23 0x0000000000f406b6 main (scylla + 0xd406b6) #24 0x00007f1e10104b75 __libc_start_main (libc.so.6 + 0x27b75) #25 0x0000000000f3d86e _start (scylla + 0xd3d86e) Stack trace of thread 11549: #0 0x000000000148557d _ZN3rowD1Ev (scylla + 0x128557d) #1 0x0000000001314e7c _ZN10rows_entry10on_evictedER13cache_tracker (scylla + 0x1114e7c) #2 0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473) #3 0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de) #4 0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539) #5 0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4) #6 0x00000000013d49bc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x11d49bc) #7 0x00000000013f3158 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f3158) #8 0x00000000013f5aba _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x11f5aba) #9 0x000000000130b918 _ZN9row_cache14do_make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110b918) #10 0x000000000130cf80 _ZN9row_cache11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x110cf80) #11 0x0000000001130ffa _ZNK5table11make_readerEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN20flat_mutation_reader30partition_range_forwarding_tagEEE (scylla + 0xf30ffa) #12 0x0000000001217273 _ZNSt17_Function_handlerIF20flat_mutation_readerN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_INS0_30partition_range_forwarding_tagEEEEZNK5table18as_mutation_sourceEvE4$_55E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSR_ (scylla + 0x1017273) #13 0x0000000001215400 _ZN5query12querier_baseC2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERK15mutation_sourceRKNS1_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x1015400) #14 0x000000000117a09b _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a09b) #15 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac) #16 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594) #17 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5) #18 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8) #19 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #20 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #21 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #22 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11566: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11551: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11558: #0 0x00007f1e10cd494c read (libpthread.so.0 + 0x1294c) #1 0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45) #2 0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0) #3 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #4 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #5 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) Stack trace of thread 11548: #0 0x00007f1e101d7e0d syscall (libc.so.6 + 0xfae0d) #1 0x0000000003e0d8b1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x3c0d8b1) #2 0x0000000003e0a275 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x3c0a275) #3 0x0000000003e0aa84 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x3c0aa84) #4 0x0000000003dc85fd _ZN7seastar7reactor6do_runEv (scylla + 0x3bc85fd) #5 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6) #6 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb) #7 0x00007f1e10ccb299 start_thread (libpthread.so.0 + 0x9299) #8 0x00007f1e101dd353 __clone (libc.so.6 + 0x100353) download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000.gz . gunzip /var/lib/systemd/coredump/core.scylla.113.a4f164f79f2249cb989230903e6e273a.11544.1642362812000000000000.gz 2022-01-16 20:01:22

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor 3bce3643-481e-40e1-9c3c-c4cf39e4bae8
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 3bce3643-481e-40e1-9c3c-c4cf39e4bae8

Test id: 3bce3643-481e-40e1-9c3c-c4cf39e4bae8

Logs:
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_200857/grafana-screenshot-longevity-large-partition-4days-test-scylla-per-server-metrics-nemesis-20220116_201122-longevity-large-partitions-4d-4-6-monitor-node-3bce3643-1.png
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_200857/grafana-screenshot-overview-20220116_200857-longevity-large-partitions-4d-4-6-monitor-node-3bce3643-1.png
db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_201642/db-cluster-3bce3643.tar.gz
loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_201642/loader-set-3bce3643.tar.gz
monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_201642/monitor-set-3bce3643.tar.gz
sct - https://cloudius-jenkins-test.s3.amazonaws.com/3bce3643-481e-40e1-9c3c-c4cf39e4bae8/20220116_201642/sct-runner-3bce3643.tar.gz

Jenkins job URL

The text was updated successfully, but these errors were encountered:

slivne · 2022-01-17T17:05:16Z

where is the coredump ?

The backtrace is not clear please extract the stacktrace and backtrace in a clean format not from graphana - its hard to parse (maybe you have a trick todo that)

slivne · 2022-01-17T20:24:19Z

OK

2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! WARNING |  [shard 11] lsa-timing - Reclamation cycle took 732002758 ms, trying to release 0.125 MiB non-preemptibly
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |  [shard 11] lsa-timing - - reserved segments: 65536
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO | Aborting on shard 11.
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO | Backtrace:
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3db44f8
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3de5352
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x7fb1f7710a1f
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x3d2a1
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x268a3
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x13f523e
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x13f5ab9
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x130b917
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x130cf7f
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x1130ff9
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x1217272
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x12153ff
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x117a09a
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x10f89ab
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x33e7593
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3dc6ec4
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3dc82b7
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3de58e5
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   0x3d9f4fa
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   /opt/scylladb/libreloc/libpthread.so.0+0x9298
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x100352
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |  [shard 11] lsa-timing - - reclamation rate = 4313.191 MiB/s
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !    INFO |  [shard 11] lsa-timing - - evicted memory: 2814.451 MiB
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 !     ERR |  [shard 11] lsa - Aborting due to allocation failure

The issue is memory allocation I think we may have an issue with the alter done

 {'sstable_compression': 'ZstdCompressor', 'chunk_length_kb': '1M', 'crc_check_chance': 0.042447059711723134};

if the chunk_length_kb 1M is actually translated to 1*10^6then its clear we will have memory pressure.

Is this a new nemesis ?

Please change this value and have it as 64 and lets see if this reproduces (I doubt it).

slivne · 2022-01-17T20:24:59Z

@roydahan / @yarongilor ^^

yarongilor · 2022-01-18T08:58:17Z

@slivne , it is not a new code ( > 3 years ).
it is already randomises and covers the value of '64':

prop_val["chunk_length_kb"] = random.choice(["4K", "64KB", "1M"])

slivne · 2022-02-08T23:22:40Z

I am not sure we should test this config combination -not sure where we picked the chunk 1M size - is there a reference to something ?

roydahan · 2022-02-09T13:47:43Z

I am not sure we should test this config combination -not sure where we picked the chunk 1M size - is there a reference to something ?

It was randomly picked by the test.
@bentsi how did you decide on the values?

bhalevy · 2022-02-13T10:30:00Z

2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3db44f8
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3de5352
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x7fb1f7710a1f
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | /opt/scylladb/libreloc/libc.so.6+0x3d2a1
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | /opt/scylladb/libreloc/libc.so.6+0x268a3
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x13f523e
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x13f5ab9
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x130b917
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x130cf7f
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x1130ff9
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x1217272
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x12153ff
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x117a09a
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x10f89ab
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x33e7593
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3dc6ec4
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3dc82b7
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3de58e5
2022-01-16T19:56:00+00:00 longevity-large-partitions-4d-4-6-db-node-3bce3643-4 ! INFO | 0x3d9f4fa

Decodes (using build-id 5d7b96e39c909424e8224207a162fc2c82b67214):

void seastar::backtrace(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:756
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:786
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:798
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3716
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3698
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3694
?? ??:0
/data/scylla-s3-reloc.cache/by-build-id/5d7b96e39c909424e8224207a162fc2c82b67214/extracted/scylla/libexec/scylladb/libreloc/libc.so.6 0x3d2a1 
/data/scylla-s3-reloc.cache/by-build-id/5d7b96e39c909424e8224207a162fc2c82b67214/extracted/scylla/libexec/scylladb/libreloc/libc.so.6 0x268a3 
logalloc::allocating_section::reserve() at ./utils/logalloc.cc:2674
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ./utils/logalloc.cc:2689
decltype(auto) logalloc::allocating_section::with_reclaiming_disabled, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6&>(logalloc::region&, row_cache::do_make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6&) at ././utils/logalloc.hh:812
 (inlined by) operator() at ././utils/logalloc.hh:832
 (inlined by) decltype(auto) logalloc::allocating_section::with_reserve, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6>(logalloc::region&, row_cache::do_make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6&&)::{lambda()#1}>(row_cache::do_make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6&&) at ././utils/logalloc.hh:781
 (inlined by) decltype(auto) logalloc::allocating_section::operator(), reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6>(logalloc::region&, row_cache::do_make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)::$_6&&) at ././utils/logalloc.hh:831
 (inlined by) row_cache::do_make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class) at ./row_cache.cc:744
row_cache::make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class) at ./row_cache.cc:785
table::make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class) const at ./table.cc:199
operator() at ./table.cc:2116
 (inlined by) flat_mutation_reader std::__invoke_impl, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class >(std::__invoke_other, table::as_mutation_source() const::$_55&, seastar::lw_shared_ptr&&, reader_permit&&, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class&&, seastar::bool_class&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class >, flat_mutation_reader>::type std::__invoke_r, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class >(table::as_mutation_source() const::$_55&, seastar::lw_shared_ptr&&, reader_permit&&, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class&&, seastar::bool_class&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:114
 (inlined by) std::_Function_handler, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class), table::as_mutation_source() const::$_55>::_M_invoke(std::_Any_data const&, seastar::lw_shared_ptr&&, reader_permit&&, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class&&, seastar::bool_class&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class)>::operator()(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) mutation_source::make_reader(seastar::lw_shared_ptr, reader_permit, nonwrapping_interval const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class, seastar::bool_class) const at ././mutation_reader.hh:281
 (inlined by) querier_base at ././querier.hh:137
querier at ././querier.hh:205
 (inlined by) table::query(seastar::lw_shared_ptr, reader_permit, query::read_command const&, query::result_options, std::vector, std::allocator > > const&, tracing::trace_state_ptr, query::result_memory_limiter&, std::chrono::time_point > >, std::optional >*) at ./table.cc:2028
operator() at ./database.cc:1412
 (inlined by) seastar::noncopyable_function (reader_permit)>::indirect_vtable_for, query::read_command const&, query::result_options, std::vector, std::allocator > > const&, tracing::trace_state_ptr, std::chrono::time_point > >)::$_38>::call(seastar::noncopyable_function (reader_permit)> const*, reader_permit) at ././seastar/include/seastar/util/noncopyable_function.hh:153
seastar::noncopyable_function (reader_permit)>::operator()(reader_permit) const at ././seastar/include/seastar/util/noncopyable_function.hh:209
 (inlined by) reader_concurrency_semaphore::execution_loop() at ./reader_concurrency_semaphore.cc:616
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2374
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2783
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2952
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4166
 (inlined by) void std::__invoke_impl(std::__invoke_other, seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if, void>::type std::__invoke_r(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60

avikivity · 2022-02-15T09:08:05Z

1 MiB chunk sizes is stressing the allocator too much.

We have two choices:

work hard to use fragmented chunks (8X 128 kiB)
Declare that this is an edge case that no one uses and ignore it (maybe add a warning)

slivne · 2022-04-13T22:09:14Z

pushing out edge case

ShlomiBalalis · 2022-08-16T00:10:01Z

Installation details

Kernel Version: 5.15.0-1017-aws
Scylla version (or git commit hash): 5.2.0~dev-20220809.3ffc978166ab with build-id 7da4f902c8209ca890ba8338dbf376529fb6c2fe
Cluster size: 4 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

longevity-twcs-48h-master-db-node-02b6a6a3-7 (13.50.5.249 | 10.0.0.119) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-6 (13.48.135.168 | 10.0.3.208) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-5 (13.51.178.28 | 10.0.2.93) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-4 (13.51.170.89 | 10.0.2.187) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-3 (13.51.174.154 | 10.0.0.60) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-2 (13.49.0.236 | 10.0.2.107) (shards: 7)
longevity-twcs-48h-master-db-node-02b6a6a3-1 (16.170.141.204 | 10.0.3.109) (shards: 7)

OS / Image: ami-0786ef2aeb8f93c28 (aws: eu-north-1)

Test: longevity-twcs-48h-test
Test id: 02b6a6a3-4530-4e1b-af60-ce437fbec08a
Test name: scylla-master/longevity/longevity-twcs-48h-test
Test config file(s):

longevity-twcs-48h.yaml

Issue description

At 2022-08-13 06:56:14.229 a ModifyTable nemesis ran, which altered the scylla_bench.test like so:
< t:2022-08-12 22:37:17,030 f:nemesis.py l:1723 c:sdcm.nemesis p:DEBUG > sdcm.nemesis.SisyphusMonkey: _modify_table_property: ALTER TABLE scylla_bench.test WITH compression = {'sstable_compression': 'SnappyCompressor', 'chunk_length_kb': '1M', 'crc_check_chance': 0.7258981385767647};

Shortly afterwards, both nodes 2 and 6 aborted with codedumps:

2022-08-13T06:56:53+00:00 longevity-twcs-48h-master-db-node-02b6a6a3-2      !ERR | scylla[8177]:  [shard 2] lsa - Aborting due to allocation failure
Aborting on shard 2.
Backtrace:
  0x4c91418
  0x4cc0b22
  0x7f2df86fca1f
  /opt/scylladb/libreloc/libc.so.6+0x3d2a1
  /opt/scylladb/libreloc/libc.so.6+0x268a3
  0x1644e3a
  0x16456cf
  0x158c001
  0x12de3f8
  0x13b3ec2
  0x13317be
  0x132d330
  0x127703c
  0x3a62700
  0x4ca0ba4
  0x4ca1f87
  0x4cc0fd5
  0x4c7494a
  /opt/scylladb/libreloc/libpthread.so.0+0x92a4
  /opt/scylladb/libreloc/libc.so.6+0x100322

Translated:

2022-08-13T06:56:53+00:00 longevity-twcs-48h-master-db-node-02b6a6a3-2     !INFO | scylla[8177]: Aborting on shard 2.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:790
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3672
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3648
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3644
?? ??:0
?? ??:0
?? ??:0
logalloc::allocating_section::reserve() at ./utils/logalloc.cc:2703
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ./utils/logalloc.cc:2718
_ZN8logalloc18allocating_section24with_reclaiming_disabledIRZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS3_17io_priority_classEN7tracing15trace_state_ptrENS3_10bool_classIN17streamed_mutation14forwarding_tagEEENSO_IN15mutation_reader30partition_range_forwarding_tagEEEE3$_7EEDcRNS_6regionEOT_ at ././utils/logalloc.hh:447
 (inlined by) operator() at ././utils/logalloc.hh:467
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS4_17io_priority_classEN7tracing15trace_state_ptrENS4_10bool_classIN17streamed_mutation14forwarding_tagEEENSP_IN15mutation_reader30partition_range_forwarding_tagEEEE3$_7EEDcRNS_6regionEOT_EUlvE_EEDcS10_ at ././utils/logalloc.hh:416
 (inlined by) _ZN8logalloc18allocating_sectionclIZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS3_17io_priority_classEN7tracing15trace_state_ptrENS3_10bool_classIN17streamed_mutation14forwarding_tagEEENSO_IN15mutation_reader30partition_range_forwarding_tagEEEE3$_7EEDcRNS_6regionEOT_ at ././utils/logalloc.hh:466
 (inlined by) row_cache::make_reader_opt(seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) at ./row_cache.cc:749
replica::table::make_reader_v2(seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at ./replica/table.cc:199
operator() at ./replica/table.cc:2184
 (inlined by) flat_mutation_reader_v2 std::__invoke_impl<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_36&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >(std::__invoke_other, replica::table::as_mutation_source() const::$_36&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_36&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >, flat_mutation_reader_v2>::type std::__invoke_r<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_36&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >(replica::table::as_mutation_source() const::$_36&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:114
 (inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::as_mutation_source() const::$_36>::_M_invoke(std::_Any_data const&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:290
std::function<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)>::operator()(seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) mutation_source::make_reader_v2(seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at ././readers/mutation_source.hh:163
 (inlined by) querier_base at ././querier.hh:79
 (inlined by) querier at ././querier.hh:145
replica::table::query(seastar::lw_shared_ptr<schema const>, reader_permit, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, query::result_memory_limiter&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::optional<query::querier>*) at ./replica/table.cc:2091
operator() at ./replica/database.cc:1490
 (inlined by) seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_20>::call(seastar::noncopyable_function<seastar::future<void> (reader_permit)> const*, reader_permit) at ././seastar/include/seastar/util/noncopyable_function.hh:153
seastar::noncopyable_function<seastar::future<void> (reader_permit)>::operator()(reader_permit) const at ././seastar/include/seastar/util/noncopyable_function.hh:209
 (inlined by) reader_concurrency_semaphore::execution_loop() at ./reader_concurrency_semaphore.cc:620
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2345
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2752
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2921
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4145
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Coredump:

2022-08-13 06:59:35.472: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=957e3b1b-8aa4-4aed-a047-9a278fc8f447 node=Node longevity-twcs-48h-master-db-node-02b6a6a3-2 [13.49.0.236 | 10.0.2.107] (seed: False)
2022-08-13 07:12:10.820 <2022-08-13 06:56:53.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=fe17f42d-cece-4f50-899e-a49c0be88bf0 node=Node longevity-twcs-48h-master-db-node-02b6a6a3-2 [13.49.0.236 | 10.0.2.107] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000.gz
backtrace=           PID: 8177 (scylla)
           UID: 113 (scylla)
           GID: 119 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Sat 2022-08-13 06:56:53 UTC (3min 12s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: 7cdd369b7c08484c9977c0d581acbabe
    Machine ID: 7869118a6c424c3da38d97ceed1f9fc2
      Hostname: longevity-twcs-48h-master-db-node-02b6a6a3-2
       Storage: /var/lib/systemd/coredump/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000
       Message: Process 8177 (scylla) of user 113 dumped core.
                
                Stack trace of thread 8179:
                #0  0x00007f2df7b542a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007f2df7b3d8a4 abort (libc.so.6 + 0x268a4)
                #2  0x0000000001644e3b _ZN8logalloc18allocating_section7reserveEv (scylla + 0x1444e3b)
                #3  0x00000000016456d0 _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x14456d0)
                #4  0x000000000158c002 _ZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x138c002)
                #5  0x00000000012de3f9 _ZNK7replica5table14make_reader_v2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x10de3f9)
                #6  0x00000000013b3ec3 _ZNSt17_Function_handlerIF23flat_mutation_reader_v2N7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEEEZNK7replica5table18as_mutation_sourceEvE4$_36E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSS_ (scylla + 0x11b3ec3)
                #7  0x00000000013317bf _ZN5query7querierC2ERK15mutation_sourceN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERKNS4_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x11317bf)
                #8  0x000000000132d331 _ZN7replica5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS7_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISG_EEN7tracing15trace_state_ptrERNS7_21result_memory_limiterENSt6chrono10time_pointINS1_12lowres_clockENSP_8durationIlSt5ratioILl1ELl1000000000EEEEEEPSt8optionalINS7_7querierEE (scylla + 0x112d331)
                #9  0x000000000127703d _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN7replica8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSD_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISM_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENST_8durationIlSt5ratioILl1ELl1000000000EEEEEESt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS14_19account_and_enforceEEEE4$_20E4callEPKS5_S3_ (scylla + 0x107703d)
                #10 0x0000000003a62701 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x3862701)
                #11 0x0000000004ca0ba5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aa0ba5)
                #12 0x0000000004ca1f88 _ZN7seastar7reactor6do_runEv (scylla + 0x4aa1f88)
                #13 0x0000000004cc0fd6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4ac0fd6)
                #14 0x0000000004c7494b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7494b)
                #15 0x00007f2df86f22a5 start_thread (libpthread.so.0 + 0x92a5)
                #16 0x00007f2df7c17323 __clone (libc.so.6 + 0x100323)
...
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.7cdd369b7c08484c9977c0d581acbabe.8177.1660373813000000000000.gz

2022-08-13T06:57:35+00:00 longevity-twcs-48h-master-db-node-02b6a6a3-6      !ERR | scylla[48787]:  [shard 3] lsa - Aborting due to allocation failure
Aborting on shard 3.
Backtrace:
  0x4c91418
  0x4cc0b22
  0x7f1f7603aa1f
  /opt/scylladb/libreloc/libc.so.6+0x3d2a1
  /opt/scylladb/libreloc/libc.so.6+0x268a3
  0x1644e3a
  0x16456cf
  0x15ee001
  0x4f0ff31

Translated:

2022-08-13T06:57:35+00:00 longevity-twcs-48h-master-db-node-02b6a6a3-6     !INFO | scylla[48787]: Aborting on shard 3.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:790
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3672
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3648
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3644
?? ??:0
?? ??:0
?? ??:0
logalloc::allocating_section::reserve() at ./utils/logalloc.cc:2703
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ./utils/logalloc.cc:2718
_ZN8logalloc18allocating_section24with_reclaiming_disabledIRZZZZN9row_cache9do_updateIZNS2_6updateENS2_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES4_S7_T_ENKUlvE_clEvENUlvE0_clEvENKUlvE0_clEvEUlvE_EEDcRNS_6regionEOSC_ at ././utils/logalloc.hh:447
 (inlined by) operator() at ././utils/logalloc.hh:467
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZZZZN9row_cache9do_updateIZNS3_6updateENS3_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES5_S8_T_ENKUlvE_clEvENUlvE0_clEvENKUlvE0_clEvEUlvE_EEDcRNS_6regionEOSD_EUlvE_EEDcSK_ at ././utils/logalloc.hh:416
 (inlined by) _ZN8logalloc18allocating_sectionclIZZZZN9row_cache9do_updateIZNS2_6updateENS2_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES4_S7_T_ENKUlvE_clEvENUlvE0_clEvENKUlvE0_clEvEUlvE_EEDcRNS_6regionEOSC_ at ././utils/logalloc.hh:466
 (inlined by) operator() at ./row_cache.cc:978
 (inlined by) _Z14with_allocatorIZZZN9row_cache9do_updateIZNS0_6updateENS0_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES2_S5_T_ENKUlvE_clEvENUlvE0_clEvEUlvE0_EDcR19allocation_strategyOSA_ at ././utils/allocation_strategy.hh:313
 (inlined by) operator() at ./row_cache.cc:965
 (inlined by) void std::__invoke_impl<void, seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>(std::__invoke_other, seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::__invoke_result<seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>::type std::__invoke<seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>(seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}&&, (seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}&&)...) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIZZN9row_cache9do_updateIZNS0_6updateENS0_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES2_S5_T_ENKUlvE_clEvEUlvE0_St5tupleIJEEJEEDcOSA_OT0_St16integer_sequenceImJXspT1_EEE at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/tuple:1858
 (inlined by) _ZSt5applyIZZN9row_cache9do_updateIZNS0_6updateENS0_16external_updaterERN7replica8memtableEE4$_21EEN7seastar6futureIvEES2_S5_T_ENKUlvE_clEvEUlvE0_St5tupleIJEEEDcOSA_OT0_ at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/tuple:1869
 (inlined by) seastar::future<void> seastar::futurize<void>::apply<seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>(seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2113
 (inlined by) operator() at ././seastar/include/seastar/core/thread.hh:258
 (inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>::type>::type seastar::async<seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}>(seastar::thread_attributes, seastar::future<void> row_cache::do_update<row_cache::update(row_cache::external_updater, replica::memtable&)::$_21>(row_cache::external_updater, replica::memtable&, row_cache::update(row_cache::external_updater, replica::memtable&)::$_21)::{lambda()#1}::operator()() const::{lambda()#2}&&, (seastar::futurize&&)...)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124
seastar::noncopyable_function<void ()>::operator()() const at ./build/release/seastar/./seastar/include/seastar/util/noncopyable_function.hh:209
 (inlined by) seastar::thread_context::main() at ./build/release/seastar/./seastar/src/core/thread.cc:299

Coredump:

2022-08-13 06:59:25.589: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=5dccba27-f0af-4ce2-b271-ad0639822431 node=Node longevity-twcs-48h-master-db-node-02b6a6a3-6 [13.48.135.168 | 10.0.3.208] (seed: False)
2022-08-13 07:11:59.372 <2022-08-13 06:57:35.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=ebcb753b-c79d-4c00-b3cc-8cff37e22a2f node=Node longevity-twcs-48h-master-db-node-02b6a6a3-6 [13.48.135.168 | 10.0.3.208] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000.gz
backtrace=           PID: 48787 (scylla)
           UID: 113 (scylla)
           GID: 119 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Sat 2022-08-13 06:57:35 UTC (2min 20s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: 8654ea0c21de4724b52a718fad013c16
    Machine ID: 7869118a6c424c3da38d97ceed1f9fc2
      Hostname: longevity-twcs-48h-master-db-node-02b6a6a3-6
       Storage: /var/lib/systemd/coredump/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000
       Message: Process 48787 (scylla) of user 113 dumped core.
                
                Stack trace of thread 48790:
                #0  0x00007f1f754922a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007f1f7547b8a4 abort (libc.so.6 + 0x268a4)
                #2  0x0000000001644e3b _ZN8logalloc18allocating_section7reserveEv (scylla + 0x1444e3b)
                #3  0x00000000016456d0 _ZN8logalloc18allocating_section16on_alloc_failureERNS_6regionE (scylla + 0x14456d0)
                #4  0x00000000015ee002 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN9row_cache9do_updateIZNS5_6updateENS5_16external_updaterERN7replica8memtableEE4$_21EENS_6futureIvEES7_SA_T_ENKUlvE_clEvEUlvE0_JEEENS_8futurizeINSt13invoke_resultISE_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSE_DpOSJ_EUlvE_E4callEPKS2_ (scylla + 0x13ee002)
                #5  0x0000000004f0ff32 _ZN7seastar14thread_context4mainEv (scylla + 0x4d0ff32)
...
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.8654ea0c21de4724b52a718fad013c16.48787.1660373855000000000000.gz

Restore Monitor Stack command: $ hydra investigate show-monitor 02b6a6a3-4530-4e1b-af60-ce437fbec08a
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 02b6a6a3-4530-4e1b-af60-ce437fbec08a

Logs:

db-cluster-02b6a6a3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/02b6a6a3-4530-4e1b-af60-ce437fbec08a/20220813_085133/db-cluster-02b6a6a3.tar.gz
monitor-set-02b6a6a3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/02b6a6a3-4530-4e1b-af60-ce437fbec08a/20220813_085133/monitor-set-02b6a6a3.tar.gz
loader-set-02b6a6a3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/02b6a6a3-4530-4e1b-af60-ce437fbec08a/20220813_085133/loader-set-02b6a6a3.tar.gz
sct-runner-02b6a6a3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/02b6a6a3-4530-4e1b-af60-ce437fbec08a/20220813_085133/sct-runner-02b6a6a3.tar.gz

Jenkins job URL

KnifeyMoloko · 2022-08-30T13:20:23Z

Reproduced it without the chunk_length_kb attribute being changed:

Installation details

Kernel Version: 5.15.0-1017-aws
Scylla version (or git commit hash): 5.2.0~dev-20220824.6ce5e9079c1e with build-id d7559bfdcb8ca68f5a8b7568ee81e1cc505e8498
Cluster size: 4 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

longevity-twcs-48h-master-db-node-215295f3-9 (13.49.77.251 | 10.0.0.243) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-8 (13.53.125.25 | 10.0.2.152) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-7 (13.51.235.9 | 10.0.3.165) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-6 (16.170.230.115 | 10.0.0.208) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-5 (16.170.214.16 | 10.0.2.221) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-4 (13.51.64.169 | 10.0.2.203) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-3 (13.53.124.5 | 10.0.1.134) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-2 (13.51.106.129 | 10.0.3.58) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-13 (16.16.58.122 | 10.0.1.192) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-12 (13.49.224.103 | 10.0.1.49) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-11 (16.170.146.146 | 10.0.0.171) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-10 (16.170.98.172 | 10.0.2.131) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-1 (13.51.156.30 | 10.0.1.177) (shards: 7)

OS / Image: ami-03b21ba37b2c1398a (aws: eu-north-1)

Test: longevity-twcs-48h-test
Test id: 215295f3-6048-40c2-818a-47e82e11df1f
Test name: scylla-master/longevity/longevity-twcs-48h-test
Test config file(s):

longevity-twcs-48h.yaml

Issue description

>>>>>>>
During ModifyTable nemesis:

We alter a table:

< t:2022-08-26 14:26:22,364 f:nemesis.py      l:1746 c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: _modify_table_property: ALTER TABLE scylla_bench.test WITH comment = 'VXG9LBNHQI28XYBZ3Q6GC2OB';

Which leads the changes being processed and Reactor stalls being logged:

2022-08-26T14:26:22+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:  [shard 0] schema_tables - Altering scylla_bench.test id=d5691210-24ef-11ed-9ed9-0355a72b9b92 version=2d887f33-a131-3202-8db2-96cbcd6bda4a
2022-08-26T14:26:22+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:  [shard 0] query_processor - Column definitions for scylla_bench.test changed, invalidating related prepared statements
2022-08-26T14:26:27+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Reactor stalled for 265 ms on shard 4. Backtrace: 0x4c9bba2 0x4c9a800 0x4c9bab0 0x7eff14fbda1f 0x1894292 0x1adde0b 0x1bebbaf 0x1ad6498 0x19959ac 0x18824be 0x1881de8 0x1887d54 0x175bcb3 0x15f551f 0x169302f 0x16b7437 0x16782b6 0x4c5d415 0x4cab394 0x4cac777 0x4ccb7c5 0x4c7f13a 0x92a4 0x100322
2022-08-26T14:26:27+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Reactor stalled for 497 ms on shard 4. Backtrace: 0x4c9bba2 0x4c9a800 0x4c9bab0 0x7eff14fbda1f 0x18943b0 0x1adde0b 0x1bebbaf 0x1ad6498 0x19959ac 0x18824be 0x1881de8 0x1887d54 0x175bcb3 0x15f551f 0x169302f 0x16b7437 0x16782b6 0x4c5d415 0x4cab394 0x4cac777 0x4ccb7c5 0x4c7f13a 0x92a4 0x100322

Then to a abort with coredump on 2 db nodes (attached below logs from db-node-1):

2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Aborting on shard 6.
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Backtrace:
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c9bc08
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4ccb312
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x7eff14fbda1f
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x3d2a1
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x268a3
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c6e635
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c71195
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x853dd
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x83cf0
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x6d922
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0xfa5d2
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0xfaa69
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x503f8d7
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a52192
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a59e2b
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a62c48
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x19a573a
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x19a4810
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1beda18
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1beea2e
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1bfa6ca
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1bfca10
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4cab394
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4cac777
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4ccb7c5
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c7f13a
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libpthread.so.0+0x92a4
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x100322
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | systemd[1]: Created slice system-systemd\x2dcoredump.slice.
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | systemd[1]: Started Process Core Dump (PID 30241/UID 0).

2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Aborting on shard 6.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:790
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3672
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3648
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3644
?? ??:0
?? ??:0
?? ??:0
seastar::memory::on_allocation_failure(unsigned long) at ./build/release/seastar/./seastar/src/core/memory.cc:1801
 (inlined by) seastar::memory::allocate(unsigned long) at ./build/release/seastar/./seastar/src/core/memory.cc:1402
malloc at ./build/release/seastar/./seastar/src/core/memory.cc:1838
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) at ./build/release/seastar/./seastar/src/util/log.cc:319
void seastar::logger::log<char const* const&, long, float, char const*, unsigned long const&, unsigned long const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(seastar::log_level, seastar::logger::format_info, char const* const&, long&&, float&&, char const*&&, unsigned long const&, unsigned long const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ././seastar/include/seastar/util/log.hh:227
 (inlined by) logalloc::reclaim_timer::report() const at ./utils/logalloc.cc:1303
~reclaim_timer at ./utils/logalloc.cc:1286
 (inlined by) logalloc::tracker::impl::compact_and_evict_locked(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2501
logalloc::tracker::impl::compact_and_evict(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2416
 (inlined by) logalloc::segment_pool::allocate_segment(unsigned long) at ./utils/logalloc.cc:1117
 (inlined by) logalloc::segment_pool::refill_emergency_reserve() at ./utils/logalloc.cc:1130
 (inlined by) logalloc::allocating_section::reserve(logalloc::tracker::impl&) at ./utils/logalloc.cc:2669
_ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN5cache11lsa_manager19run_in_read_sectionIZZNS3_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_EUlvE_EEDcRNS_6regionEOS9_EUlvE_EEDcSE_SF_ at ././utils/logalloc.hh:469
 (inlined by) _ZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZZNS2_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_EUlvE_EEDcRNS_6regionEOS8_ at ././utils/logalloc.hh:520
 (inlined by) _ZN5cache11lsa_manager19run_in_read_sectionIZZNS_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_ at ././row_cache.hh:480
 (inlined by) operator() at ././cache_flat_mutation_reader.hh:332
cache::cache_flat_mutation_reader::fill_buffer() at ././cache_flat_mutation_reader.hh:340
flat_mutation_reader_v2::impl::operator()() at ././readers/flat_mutation_reader_v2.hh:195
 (inlined by) flat_mutation_reader_v2::operator()() at ././readers/flat_mutation_reader_v2.hh:415
 (inlined by) mutation_reader_merger::prepare_one(mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at ./readers/combined.cc:457
 (inlined by) operator() at ./readers/combined.cc:448
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&>(mutation_reader_merger::prepare_next()::$_2&&, mutation_reader_merger::reader_and_last_fragment_kind&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&>(mutation_reader_merger::prepare_next()::$_2&&, mutation_reader_merger::reader_and_last_fragment_kind&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next()::$_2>(mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:566
 (inlined by) seastar::future<void> seastar::internal::parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:616
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:631
 (inlined by) mutation_reader_merger::prepare_next() at ./readers/combined.cc:447
mutation_reader_merger::operator()() at ./readers/combined.cc:578
operator() at ./readers/combined.cc:171
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&>(mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::repeat<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}>(mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&&) at ././seastar/include/seastar/core/loop.hh:120
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()() at ./readers/combined.cc:170
 (inlined by) operator() at ./readers/combined.cc:692
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&>(merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&>(merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::internal::repeater<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}>::run_and_dispose() at ././seastar/include/seastar/core/loop.hh:73
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2345
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2752
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2921
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4145
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Coredump:

2022-08-26 14:43:26.912 <2022-08-26 14:26:31.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=4700a498-1c46-4066-a970-efe77c42a3e6 node=Node longevity-twcs-48h-master-db-node-215295f3-1 [13.51.156.30 | 10.0.1.177] (seed: True)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz
backtrace=           PID: 26414 (scylla)
           UID: 113 (scylla)
           GID: 119 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Fri 2022-08-26 14:26:31 UTC (3min 43s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: f62071f23c364e6bbc68a0642ff2577b
    Machine ID: 4c635b1baff44156b320ad8682a8f549
      Hostname: longevity-twcs-48h-master-db-node-215295f3-1
       Storage: /var/lib/systemd/coredump/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000
       Message: Process 26414 (scylla) of user 113 dumped core.

                Stack trace of thread 26420:
                #0  0x00007eff144152a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007eff143fe8a4 abort (libc.so.6 + 0x268a4)
                #2  0x0000000004c6e636 _ZN7seastar6memory8allocateEm (scylla + 0x4a6e636)
                #3  0x0000000004c71196 malloc (scylla + 0x4a71196)
                #4  0x00007eff1445d3de _IO_str_overflow (libc.so.6 + 0x853de)
                #5  0x00007eff1445bcf1 _IO_default_xsputn (libc.so.6 + 0x83cf1)
                #6  0x00007eff14445923 __vfprintf_internal (libc.so.6 + 0x6d923)
                #7  0x00007eff144d25d3 __vsyslog_internal (libc.so.6 + 0xfa5d3)
                #8  0x00007eff144d2a6a syslog (libc.so.6 + 0xfaa6a)
                #9  0x000000000503f8d8 _ZN7seastar6logger6do_logENS_9log_levelERNS0_10log_writerE (scylla + 0x4e3f8d8)
                #10 0x0000000001a52193 _ZNK8logalloc13reclaim_timer6reportEv (scylla + 0x1852193)
                #11 0x0000000001a59e2c _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x1859e2c)
                #12 0x0000000001a62c49 _ZN8logalloc18allocating_section7reserveERNS_7tracker4implE (scylla + 0x1862c49)
                #13 0x00000000019a573b _ZZN5cache26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEv (scylla + 0x17a573b)
                #14 0x00000000019a4811 _ZN5cache26cache_flat_mutation_reader11fill_bufferEv (scylla + 0x17a4811)
                #15 0x0000000001beda19 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x19eda19)
                #16 0x0000000001beea2f _ZN22mutation_reader_mergerclEv (scylla + 0x19eea2f)
                #17 0x0000000001bfa6cb _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x19fa6cb)
                #18 0x0000000001bfca11 _ZN7seastar8internal8repeaterIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_E15run_and_disposeEv (scylla + 0x19fca11)
                #19 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #20 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #22 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #23 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #24 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26427:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26430:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26428:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26429:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26426:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26421:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26425:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26417:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26416:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

               Stack trace of thread 26419:
                #0  0x0000000001ac982d _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x18c982d)
                #1  0x0000000001b35b2b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b2b)
                #2  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #3  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #4  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #5  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #6  0x0000000001ac89b1 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x18c89b1)
                #7  0x000000000199597f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x179597f)
                #8  0x00000000019bd96c _ZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x17bd96c)
                #9  0x000000000170f019 _ZNK7replica5table14make_reader_v2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x150f019)
                #10 0x00000000017e2d93 _ZNSt17_Function_handlerIF23flat_mutation_reader_v2N7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEEEZNK7replica5table18as_mutation_sourceEvE4$_36E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSS_ (scylla + 0x15e2d93)
                #11 0x000000000176227f _ZN5query7querierC2ERK15mutation_sourceN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERKNS4_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x156227f)
                #12 0x000000000175de47 _ZN7replica5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS7_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISG_EEN7tracing15trace_state_ptrERNS7_21result_memory_limiterENSt6chrono10time_pointINS1_12lowres_clockENSP_8durationIlSt5ratioILl1ELl1000000000EEEEEEPSt8optionalINS7_7querierEE (scylla + 0x155de47)
                #13 0x00000000016a85fd _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN7replica8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSD_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISM_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENST_8durationIlSt5ratioILl1ELl1000000000EEEEEESt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS14_19account_and_enforceEEEE4$_20E4callEPKS5_S3_ (scylla + 0x14a85fd)
                #14 0x0000000003a6b871 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x386b871)
                #15 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #16 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #17 0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #18 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #19 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #20 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26418:
                #0  0x0000000001ac954e _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x18c954e)
                #1  0x0000000001b35b2b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b2b)
                #2  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #3  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #4  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #5  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #6  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #7  0x0000000001ac89b1 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x18c89b1)
                #8  0x000000000199597f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x179597f)
                #9  0x00000000018c7d5b _ZZN7replica12flush_reader18get_next_partitionEvENKUlvE_clEv (scylla + 0x16c7d5b)
                #10 0x00000000018c5bf5 _ZN7replica12flush_reader18get_next_partitionEv (scylla + 0x16c5bf5)
                #11 0x00000000018c4714 _ZN7replica12flush_reader11fill_bufferEv (scylla + 0x16c4714)
                #12 0x0000000003ac1b79 _ZN15mutation_writer11feed_writerINS_41timestamp_based_splitting_mutation_writerEEEN7seastar6futureIvEEO23flat_mutation_reader_v2T_ (scylla + 0x38c1b79)
                #13 0x0000000003ac1440 _ZN15mutation_writer22segregate_by_timestampE23flat_mutation_reader_v2N7seastar20noncopyable_functionIFllEEENS2_IFNS1_6futureIvEES0_EEE (scylla + 0x38c1440)
                #14 0x00000000020db8fc _ZN7seastar20noncopyable_functionIFNS_6futureIvEE23flat_mutation_reader_v2EE19indirect_vtable_forIZN8sstables31time_window_compaction_strategy24make_interposer_consumerERK24mutation_source_metadataS5_E3$_1E4callEPKS5_S3_ (scylla + 0x1edb8fc)
                #15 0x000000000178185b _ZZN7replica5table29try_flush_memtable_to_sstableEN7seastar13lw_shared_ptrINS_8memtableEEEO20sstable_write_permitEN3$_9clEv (scylla + 0x158185b)
                #16 0x0000000001787b68 _ZN7seastar11lambda_taskIZNS_21with_scheduling_groupISt17reference_wrapperIZN7replica5table29try_flush_memtable_to_sstableENS_13lw_shared_ptrINS3_8memtableEEEO20sstable_write_permitE3$_9EJEEEDaNS_16scheduling_groupET_DpOT0_EUlvE_E15run_and_disposeEv (scylla + 0x1587b68)
                #17 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #18 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #19 0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #20 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #21 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #22 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26414:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004cab9cd _ZN7seastar7reactor3runEv (scylla + 0x4aab9cd)
                #6  0x0000000004c51f69 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x4a51f69)
                #7  0x0000000004c51442 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x4a51442)
                #8  0x000000000108120e _ZL11scylla_mainiPPc (scylla + 0xe8120e)
                #9  0x000000000107ed46 main (scylla + 0xe7ed46)
                #10 0x00007eff143ffb75 __libc_start_main (libc.so.6 + 0x27b75)
                #11 0x000000000107ce6e _start (scylla + 0xe7ce6e)

                Stack trace of thread 26415:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz

<<<<<<<

Restore Monitor Stack command: $ hydra investigate show-monitor 215295f3-6048-40c2-818a-47e82e11df1f
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 215295f3-6048-40c2-818a-47e82e11df1f

Logs:

db-cluster-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/db-cluster-215295f3.tar.gz
monitor-set-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/monitor-set-215295f3.tar.gz
loader-set-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/loader-set-215295f3.tar.gz
sct-runner-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/sct-runner-215295f3.tar.gz

Jenkins job URL

avikivity · 2022-08-30T14:00:35Z

Reproduced it without the chunk_length_kb attribute being changed:

Then it's something else (#2577). This issue is about chunk_length_kb.

Installation details

Kernel Version: 5.15.0-1017-aws Scylla version (or git commit hash): 5.2.0~dev-20220824.6ce5e9079c1e with build-id d7559bfdcb8ca68f5a8b7568ee81e1cc505e8498 Cluster size: 4 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

longevity-twcs-48h-master-db-node-215295f3-9 (13.49.77.251 | 10.0.0.243) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-8 (13.53.125.25 | 10.0.2.152) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-7 (13.51.235.9 | 10.0.3.165) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-6 (16.170.230.115 | 10.0.0.208) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-5 (16.170.214.16 | 10.0.2.221) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-4 (13.51.64.169 | 10.0.2.203) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-3 (13.53.124.5 | 10.0.1.134) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-2 (13.51.106.129 | 10.0.3.58) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-13 (16.16.58.122 | 10.0.1.192) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-12 (13.49.224.103 | 10.0.1.49) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-11 (16.170.146.146 | 10.0.0.171) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-10 (16.170.98.172 | 10.0.2.131) (shards: 7)
longevity-twcs-48h-master-db-node-215295f3-1 (13.51.156.30 | 10.0.1.177) (shards: 7)

OS / Image: ami-03b21ba37b2c1398a (aws: eu-north-1)

Test: longevity-twcs-48h-test Test id: 215295f3-6048-40c2-818a-47e82e11df1f Test name: scylla-master/longevity/longevity-twcs-48h-test Test config file(s):

longevity-twcs-48h.yaml

Issue description

>>>>>>> During ModifyTable nemesis:

We alter a table:

< t:2022-08-26 14:26:22,364 f:nemesis.py      l:1746 c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: _modify_table_property: ALTER TABLE scylla_bench.test WITH comment = 'VXG9LBNHQI28XYBZ3Q6GC2OB';

Which leads the changes being processed and Reactor stalls being logged:

2022-08-26T14:26:22+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:  [shard 0] schema_tables - Altering scylla_bench.test id=d5691210-24ef-11ed-9ed9-0355a72b9b92 version=2d887f33-a131-3202-8db2-96cbcd6bda4a
2022-08-26T14:26:22+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:  [shard 0] query_processor - Column definitions for scylla_bench.test changed, invalidating related prepared statements
2022-08-26T14:26:27+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Reactor stalled for 265 ms on shard 4. Backtrace: 0x4c9bba2 0x4c9a800 0x4c9bab0 0x7eff14fbda1f 0x1894292 0x1adde0b 0x1bebbaf 0x1ad6498 0x19959ac 0x18824be 0x1881de8 0x1887d54 0x175bcb3 0x15f551f 0x169302f 0x16b7437 0x16782b6 0x4c5d415 0x4cab394 0x4cac777 0x4ccb7c5 0x4c7f13a 0x92a4 0x100322
2022-08-26T14:26:27+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Reactor stalled for 497 ms on shard 4. Backtrace: 0x4c9bba2 0x4c9a800 0x4c9bab0 0x7eff14fbda1f 0x18943b0 0x1adde0b 0x1bebbaf 0x1ad6498 0x19959ac 0x18824be 0x1881de8 0x1887d54 0x175bcb3 0x15f551f 0x169302f 0x16b7437 0x16782b6 0x4c5d415 0x4cab394 0x4cac777 0x4ccb7c5 0x4c7f13a 0x92a4 0x100322

Then to a abort with coredump on 2 db nodes (attached below logs from db-node-1):

2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Aborting on shard 6.
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Backtrace:
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c9bc08
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4ccb312
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x7eff14fbda1f
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x3d2a1
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x268a3
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c6e635
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c71195
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x853dd
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x83cf0
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x6d922
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0xfa5d2
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0xfaa69
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x503f8d7
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a52192
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a59e2b
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1a62c48
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x19a573a
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x19a4810
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1beda18
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1beea2e
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1bfa6ca
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x1bfca10
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4cab394
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4cac777
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4ccb7c5
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   0x4c7f13a
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libpthread.so.0+0x92a4
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]:   /opt/scylladb/libreloc/libc.so.6+0x100322
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | systemd[1]: Created slice system-systemd\x2dcoredump.slice.
2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | systemd[1]: Started Process Core Dump (PID 30241/UID 0).

2022-08-26T14:26:31+00:00 longevity-twcs-48h-master-db-node-215295f3-1     !INFO | scylla[26414]: Aborting on shard 6.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:790
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3672
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3648
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3644
?? ??:0
?? ??:0
?? ??:0
seastar::memory::on_allocation_failure(unsigned long) at ./build/release/seastar/./seastar/src/core/memory.cc:1801
 (inlined by) seastar::memory::allocate(unsigned long) at ./build/release/seastar/./seastar/src/core/memory.cc:1402
malloc at ./build/release/seastar/./seastar/src/core/memory.cc:1838
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) at ./build/release/seastar/./seastar/src/util/log.cc:319
void seastar::logger::log<char const* const&, long, float, char const*, unsigned long const&, unsigned long const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(seastar::log_level, seastar::logger::format_info, char const* const&, long&&, float&&, char const*&&, unsigned long const&, unsigned long const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ././seastar/include/seastar/util/log.hh:227
 (inlined by) logalloc::reclaim_timer::report() const at ./utils/logalloc.cc:1303
~reclaim_timer at ./utils/logalloc.cc:1286
 (inlined by) logalloc::tracker::impl::compact_and_evict_locked(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2501
logalloc::tracker::impl::compact_and_evict(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2416
 (inlined by) logalloc::segment_pool::allocate_segment(unsigned long) at ./utils/logalloc.cc:1117
 (inlined by) logalloc::segment_pool::refill_emergency_reserve() at ./utils/logalloc.cc:1130
 (inlined by) logalloc::allocating_section::reserve(logalloc::tracker::impl&) at ./utils/logalloc.cc:2669
_ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN5cache11lsa_manager19run_in_read_sectionIZZNS3_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_EUlvE_EEDcRNS_6regionEOS9_EUlvE_EEDcSE_SF_ at ././utils/logalloc.hh:469
 (inlined by) _ZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZZNS2_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_EUlvE_EEDcRNS_6regionEOS8_ at ././utils/logalloc.hh:520
 (inlined by) _ZN5cache11lsa_manager19run_in_read_sectionIZZNS_26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEvEUlvE_EEDcRKT_ at ././row_cache.hh:480
 (inlined by) operator() at ././cache_flat_mutation_reader.hh:332
cache::cache_flat_mutation_reader::fill_buffer() at ././cache_flat_mutation_reader.hh:340
flat_mutation_reader_v2::impl::operator()() at ././readers/flat_mutation_reader_v2.hh:195
 (inlined by) flat_mutation_reader_v2::operator()() at ././readers/flat_mutation_reader_v2.hh:415
 (inlined by) mutation_reader_merger::prepare_one(mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at ./readers/combined.cc:457
 (inlined by) operator() at ./readers/combined.cc:448
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&>(mutation_reader_merger::prepare_next()::$_2&&, mutation_reader_merger::reader_and_last_fragment_kind&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&>(mutation_reader_merger::prepare_next()::$_2&&, mutation_reader_merger::reader_and_last_fragment_kind&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next()::$_2>(mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:566
 (inlined by) seastar::future<void> seastar::internal::parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:616
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2>(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&) at ././seastar/include/seastar/core/loop.hh:631
 (inlined by) mutation_reader_merger::prepare_next() at ./readers/combined.cc:447
mutation_reader_merger::operator()() at ./readers/combined.cc:578
operator() at ./readers/combined.cc:171
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&>(mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::repeat<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}>(mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&&) at ././seastar/include/seastar/core/loop.hh:120
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()() at ./readers/combined.cc:170
 (inlined by) operator() at ./readers/combined.cc:692
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&>(merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&>(merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&) at ././seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::internal::repeater<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}>::run_and_dispose() at ././seastar/include/seastar/core/loop.hh:73
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2345
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2752
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2921
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4145
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_90>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Coredump:

2022-08-26 14:43:26.912 <2022-08-26 14:26:31.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=4700a498-1c46-4066-a970-efe77c42a3e6 node=Node longevity-twcs-48h-master-db-node-215295f3-1 [13.51.156.30 | 10.0.1.177] (seed: True)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz
backtrace=           PID: 26414 (scylla)
           UID: 113 (scylla)
           GID: 119 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Fri 2022-08-26 14:26:31 UTC (3min 43s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: f62071f23c364e6bbc68a0642ff2577b
    Machine ID: 4c635b1baff44156b320ad8682a8f549
      Hostname: longevity-twcs-48h-master-db-node-215295f3-1
       Storage: /var/lib/systemd/coredump/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000
       Message: Process 26414 (scylla) of user 113 dumped core.

                Stack trace of thread 26420:
                #0  0x00007eff144152a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007eff143fe8a4 abort (libc.so.6 + 0x268a4)
                #2  0x0000000004c6e636 _ZN7seastar6memory8allocateEm (scylla + 0x4a6e636)
                #3  0x0000000004c71196 malloc (scylla + 0x4a71196)
                #4  0x00007eff1445d3de _IO_str_overflow (libc.so.6 + 0x853de)
                #5  0x00007eff1445bcf1 _IO_default_xsputn (libc.so.6 + 0x83cf1)
                #6  0x00007eff14445923 __vfprintf_internal (libc.so.6 + 0x6d923)
                #7  0x00007eff144d25d3 __vsyslog_internal (libc.so.6 + 0xfa5d3)
                #8  0x00007eff144d2a6a syslog (libc.so.6 + 0xfaa6a)
                #9  0x000000000503f8d8 _ZN7seastar6logger6do_logENS_9log_levelERNS0_10log_writerE (scylla + 0x4e3f8d8)
                #10 0x0000000001a52193 _ZNK8logalloc13reclaim_timer6reportEv (scylla + 0x1852193)
                #11 0x0000000001a59e2c _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x1859e2c)
                #12 0x0000000001a62c49 _ZN8logalloc18allocating_section7reserveERNS_7tracker4implE (scylla + 0x1862c49)
                #13 0x00000000019a573b _ZZN5cache26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEv (scylla + 0x17a573b)
                #14 0x00000000019a4811 _ZN5cache26cache_flat_mutation_reader11fill_bufferEv (scylla + 0x17a4811)
                #15 0x0000000001beda19 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x19eda19)
                #16 0x0000000001beea2f _ZN22mutation_reader_mergerclEv (scylla + 0x19eea2f)
                #17 0x0000000001bfa6cb _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x19fa6cb)
                #18 0x0000000001bfca11 _ZN7seastar8internal8repeaterIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_E15run_and_disposeEv (scylla + 0x19fca11)
                #19 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #20 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #22 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #23 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #24 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26427:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26430:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26428:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26429:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26426:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26421:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26425:
                #0  0x00007eff14fbc94c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000004cedc75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x4aedc75)
                #2  0x0000000004cedfd0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x4aedfd0)
                #3  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #4  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #5  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26417:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26416:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

               Stack trace of thread 26419:
                #0  0x0000000001ac982d _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x18c982d)
                #1  0x0000000001b35b2b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b2b)
                #2  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #3  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #4  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #5  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #6  0x0000000001ac89b1 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x18c89b1)
                #7  0x000000000199597f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x179597f)
                #8  0x00000000019bd96c _ZN9row_cache15make_reader_optEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS0_17io_priority_classEN7tracing15trace_state_ptrENS0_10bool_classIN17streamed_mutation14forwarding_tagEEENSL_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x17bd96c)
                #9  0x000000000170f019 _ZNK7replica5table14make_reader_v2EN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEE (scylla + 0x150f019)
                #10 0x00000000017e2d93 _ZNSt17_Function_handlerIF23flat_mutation_reader_v2N7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK20nonwrapping_intervalIN3dht13ring_positionEERKN5query15partition_sliceERKNS1_17io_priority_classEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSM_IN15mutation_reader30partition_range_forwarding_tagEEEEZNK7replica5table18as_mutation_sourceEvE4$_36E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_SJ_OSL_OSP_OSS_ (scylla + 0x15e2d93)
                #11 0x000000000176227f _ZN5query7querierC2ERK15mutation_sourceN7seastar13lw_shared_ptrIK6schemaEE13reader_permit20nonwrapping_intervalIN3dht13ring_positionEENS_15partition_sliceERKNS4_17io_priority_classEN7tracing15trace_state_ptrE (scylla + 0x156227f)
                #12 0x000000000175de47 _ZN7replica5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS7_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISG_EEN7tracing15trace_state_ptrERNS7_21result_memory_limiterENSt6chrono10time_pointINS1_12lowres_clockENSP_8durationIlSt5ratioILl1ELl1000000000EEEEEEPSt8optionalINS7_7querierEE (scylla + 0x155de47)
                #13 0x00000000016a85fd _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN7replica8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSD_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISM_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENST_8durationIlSt5ratioILl1ELl1000000000EEEEEESt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS14_19account_and_enforceEEEE4$_20E4callEPKS5_S3_ (scylla + 0x14a85fd)
                #14 0x0000000003a6b871 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x386b871)
                #15 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #16 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #17 0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #18 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #19 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #20 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26418:
                #0  0x0000000001ac954e _ZN19allocation_strategy9constructI10rows_entryJRK6schemaRKS1_EEEPT_DpOT0_ (scylla + 0x18c954e)
                #1  0x0000000001b35b2b _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b2b)
                #2  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #3  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #4  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #5  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #6  0x0000000001b35b7a _ZNK11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE5cloneIRZN18mutation_partitionC1ERK6schemaRKS7_E3$_0RZ15current_deleterIS1_EDavEUlPS1_E_EEPS5_RSJ_SK_OT_OT0_ (scylla + 0x1935b7a)
                #7  0x0000000001ac89b1 _ZN18mutation_partitionC1ERK6schemaRKS_ (scylla + 0x18c89b1)
                #8  0x000000000199597f _ZN15partition_entry7upgradeEN7seastar13lw_shared_ptrIK6schemaEES4_R16mutation_cleanerP13cache_tracker (scylla + 0x179597f)
                #9  0x00000000018c7d5b _ZZN7replica12flush_reader18get_next_partitionEvENKUlvE_clEv (scylla + 0x16c7d5b)
                #10 0x00000000018c5bf5 _ZN7replica12flush_reader18get_next_partitionEv (scylla + 0x16c5bf5)
                #11 0x00000000018c4714 _ZN7replica12flush_reader11fill_bufferEv (scylla + 0x16c4714)
                #12 0x0000000003ac1b79 _ZN15mutation_writer11feed_writerINS_41timestamp_based_splitting_mutation_writerEEEN7seastar6futureIvEEO23flat_mutation_reader_v2T_ (scylla + 0x38c1b79)
                #13 0x0000000003ac1440 _ZN15mutation_writer22segregate_by_timestampE23flat_mutation_reader_v2N7seastar20noncopyable_functionIFllEEENS2_IFNS1_6futureIvEES0_EEE (scylla + 0x38c1440)
                #14 0x00000000020db8fc _ZN7seastar20noncopyable_functionIFNS_6futureIvEE23flat_mutation_reader_v2EE19indirect_vtable_forIZN8sstables31time_window_compaction_strategy24make_interposer_consumerERK24mutation_source_metadataS5_E3$_1E4callEPKS5_S3_ (scylla + 0x1edb8fc)
                #15 0x000000000178185b _ZZN7replica5table29try_flush_memtable_to_sstableEN7seastar13lw_shared_ptrINS_8memtableEEEO20sstable_write_permitEN3$_9clEv (scylla + 0x158185b)
                #16 0x0000000001787b68 _ZN7seastar11lambda_taskIZNS_21with_scheduling_groupISt17reference_wrapperIZN7replica5table29try_flush_memtable_to_sstableENS_13lw_shared_ptrINS3_8memtableEEEO20sstable_write_permitE3$_9EJEEEDaNS_16scheduling_groupET_DpOT0_EUlvE_E15run_and_disposeEv (scylla + 0x1587b68)
                #17 0x0000000004cab395 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x4aab395)
                #18 0x0000000004cac778 _ZN7seastar7reactor6do_runEv (scylla + 0x4aac778)
                #19 0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #20 0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #21 0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #22 0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

                Stack trace of thread 26414:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004cab9cd _ZN7seastar7reactor3runEv (scylla + 0x4aab9cd)
                #6  0x0000000004c51f69 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x4a51f69)
                #7  0x0000000004c51442 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x4a51442)
                #8  0x000000000108120e _ZL11scylla_mainiPPc (scylla + 0xe8120e)
                #9  0x000000000107ed46 main (scylla + 0xe7ed46)
                #10 0x00007eff143ffb75 __libc_start_main (libc.so.6 + 0x27b75)
                #11 0x000000000107ce6e _start (scylla + 0xe7ce6e)

                Stack trace of thread 26415:
                #0  0x00007eff144d2ddd syscall (libc.so.6 + 0xfaddd)
                #1  0x0000000004cf3bd1 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x4af3bd1)
                #2  0x0000000004cef755 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x4aef755)
                #3  0x0000000004cefec4 _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x4aefec4)
                #4  0x0000000004cacafd _ZN7seastar7reactor6do_runEv (scylla + 0x4aacafd)
                #5  0x0000000004ccb7c6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_90E9_M_invokeERKSt9_Any_data (scylla + 0x4acb7c6)
                #6  0x0000000004c7f13b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x4a7f13b)
                #7  0x00007eff14fb32a5 start_thread (libpthread.so.0 + 0x92a5)
                #8  0x00007eff144d8323 __clone (libc.so.6 + 0x100323)

download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.f62071f23c364e6bbc68a0642ff2577b.26414.1661523991000000000000.gz

<<<<<<<

Restore Monitor Stack command: $ hydra investigate show-monitor 215295f3-6048-40c2-818a-47e82e11df1f
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 215295f3-6048-40c2-818a-47e82e11df1f

Logs:

db-cluster-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/db-cluster-215295f3.tar.gz
monitor-set-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/monitor-set-215295f3.tar.gz
loader-set-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/loader-set-215295f3.tar.gz
sct-runner-215295f3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/215295f3-6048-40c2-818a-47e82e11df1f/20220828_035131/sct-runner-215295f3.tar.gz

Jenkins job URL

This patch adds some minimal tests for the "with compression = {..}" table configuration. These tests reproduce three known bugs: Refs scylladb#6442: Always print all schema parameters (including default values) Scylla doesn't return the default chunk_length_in_kb, but Cassandra does. Refs scylladb#8948: Cassandra 3.11.10 uses "class" instead of "sstable_compression" for compression settings by default Cassandra switched, long ago, the "sstable_compression" attribute's name to "class". This can break Cassandra applications that create tables (where we won't understand the "class" parameter) and applications that inquire about the configuration of existing tables. This patch adds tests for both problems. Refs scylladb#9933: ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes Our test for this issue hangs Scylla (or crashes, depending on the test environment configuration), when a huge allocation is attempted during memtable flush. So this test is marked "skip" instead of xfail. The tests included here also uncovered a new minor/insignificant bug, where Scylla allows floating point numbers as chunk_length_in_kb - this number is truncated to an integer, and allowed, unlike Cassandra or common sense. Signed-off-by: Nadav Har'El <nyh@scylladb.com>

nyh · 2023-06-15T12:45:47Z

Added a cql-pytest to reproduce this bug, test_sstable_compression.py::test_huge_chunk_length. This test sets chunk_length_in_kb to 1048576 (i.e., 1GB) and writes some data. The result (without configuring bad_alloc to crash) is an allocation failure during memtable flush, and after that - Scylla hangs:

WARN  2023-06-15 15:40:39,195 [shard 0] seastar_memory - oversized allocation: 1073741824 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues.
ERROR 2023-06-15 15:40:39,195 [shard 0] table - failed to write sstable /tmp/scylla-test-3126104/data/cql_test_1686832839176/cql_test_1686832839185-d54178100b7911eea84d26bdda49ef99/me-2-big-Data.db: std::bad_alloc (std::bad_alloc)
WARN  2023-06-15 15:40:39,195 [shard 0] table - Memtable flush failed due to: std::bad_alloc (std::bad_alloc). Will retry in 100ms

and this repeats indefinitely, with growing (up to 10s) retry delay.

So this issue allows a DoS attack for authenticated users.

Interestingly, this test shows that Cassandra also doesn't protect against huge chunk_length_in_kb. It doesn't seem to crash with a 1GB chunk size, but it's super-slow.

avikivity · 2023-06-15T12:47:18Z

Let's reject anything above 128k.

The chunk size used in sstable compression can be set when creating a table, using the "chunk_length_in_kb" parameter. It can be any power-of-two multiple of 1KB. Very large compression chunks are not useful - they offer diminishing returns on compression ratio, and require very large memory buffers and reading a very large amount of disk data just to read a small row. In fact, small chunks are recommended - Scylla defaults to 4 KB chunks, and Cassandra lowered their default from 64 KB (in Cassandra 3) to 16 KB (in Cassandra 4). Therefore, allowing arbitrarily large chunk sizes is just asking for trouble. Today, a user can ask for a 1 GB chunk size, and crash or hang Scylla when it runs out of memory. So in this patch we add a hard limit of 128 KB for the chunk size - anything larger is refused. Fixes scylladb#9933 Signed-off-by: Nadav Har'El <nyh@scylladb.com>

This patch adds some minimal tests for the "with compression = {..}" table configuration. These tests reproduce three known bugs: Refs scylladb#6442: Always print all schema parameters (including default values) Scylla doesn't return the default chunk_length_in_kb, but Cassandra does. Refs scylladb#8948: Cassandra 3.11.10 uses "class" instead of "sstable_compression" for compression settings by default Cassandra switched, long ago, the "sstable_compression" attribute's name to "class". This can break Cassandra applications that create tables (where we won't understand the "class" parameter) and applications that inquire about the configuration of existing tables. This patch adds tests for both problems. Refs scylladb#9933: ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes Our test for this issue hangs Scylla (or crashes, depending on the test environment configuration), when a huge allocation is attempted during memtable flush. So this test is marked "skip" instead of xfail. The tests included here also uncovered a new minor/insignificant bug, where Scylla allows floating point numbers as chunk_length_in_kb - this number is truncated to an integer, and allowed, unlike Cassandra or common sense. Signed-off-by: Nadav Har'El <nyh@scylladb.com>

This patch adds some minimal tests for the "with compression = {..}" table configuration. These tests reproduce three known bugs: Refs #6442: Always print all schema parameters (including default values) Scylla doesn't return the default chunk_length_in_kb, but Cassandra does. Refs #8948: Cassandra 3.11.10 uses "class" instead of "sstable_compression" for compression settings by default Cassandra switched, long ago, the "sstable_compression" attribute's name to "class". This can break Cassandra applications that create tables (where we won't understand the "class" parameter) and applications that inquire about the configuration of existing tables. This patch adds tests for both problems. Refs #9933: ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes Our test for this issue hangs Scylla (or crashes, depending on the test environment configuration), when a huge allocation is attempted during memtable flush. So this test is marked "skip" instead of xfail. The tests included here also uncovered a new minor/insignificant bug, where Scylla allows floating point numbers as chunk_length_in_kb - this number is truncated to an integer, and allowed, unlike Cassandra or common sense. Signed-off-by: Nadav Har'El <nyh@scylladb.com> Closes #14261

The chunk size used in sstable compression can be set when creating a table, using the "chunk_length_in_kb" parameter. It can be any power-of-two multiple of 1KB. Very large compression chunks are not useful - they offer diminishing returns on compression ratio, and require very large memory buffers and reading a very large amount of disk data just to read a small row. In fact, small chunks are recommended - Scylla defaults to 4 KB chunks, and Cassandra lowered their default from 64 KB (in Cassandra 3) to 16 KB (in Cassandra 4). Therefore, allowing arbitrarily large chunk sizes is just asking for trouble. Today, a user can ask for a 1 GB chunk size, and crash or hang Scylla when it runs out of memory. So in this patch we add a hard limit of 128 KB for the chunk size - anything larger is refused. Fixes scylladb#9933 Signed-off-by: Nadav Har'El <nyh@scylladb.com>

denesb · 2023-12-14T12:49:17Z

Not a regression or bugfix, not backporting.

roydahan added the triage/oss label Jan 17, 2022

roydahan added this to the 4.6 milestone Jan 17, 2022

slivne assigned yarongilor Jan 17, 2022

slivne added area/test Issues related to the testing system code and environment and removed triage/oss labels Jan 17, 2022

yarongilor mentioned this issue Feb 3, 2022

Segmentation fault and core dump on alter table cqlstress_lwt_example.blogposts command #10026

Closed

slivne modified the milestones: 4.6, 5.0 Feb 8, 2022

slivne mentioned this issue Feb 9, 2022

Coredump after snapshot operations #10056

Closed

roydahan changed the title ~~3 cluster nodes got core dump after an alter table command~~ ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes Feb 21, 2022

slivne modified the milestones: 5.0, 5.2 Apr 13, 2022

DoronArazii modified the milestones: 5.2, 5.x Nov 22, 2022

nyh mentioned this issue Jun 15, 2023

test/cql-pytest: add tests reproducing bugs in compression configuration #14261

Closed

mykaul modified the milestones: 5.x, 5.4 Jun 15, 2023

mykaul added P2 High Priority and removed area/test Issues related to the testing system code and environment labels Jun 15, 2023

mykaul assigned eliransin and unassigned yarongilor Jun 15, 2023

nyh self-assigned this Jun 15, 2023

nyh mentioned this issue Jun 15, 2023

sstable: limit compression chunk size to 128 KB #14267

Closed

scylladb-promoter closed this as completed in 8a9de08 Jun 21, 2023

scylladb-promoter added the Backport candidate label Jun 21, 2023

denesb removed the Backport candidate label Dec 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes #9933

ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes #9933

yarongilor commented Jan 17, 2022

slivne commented Jan 17, 2022

slivne commented Jan 17, 2022

slivne commented Jan 17, 2022

yarongilor commented Jan 18, 2022

slivne commented Feb 8, 2022

roydahan commented Feb 9, 2022

bhalevy commented Feb 13, 2022

avikivity commented Feb 15, 2022

slivne commented Apr 13, 2022

ShlomiBalalis commented Aug 16, 2022

KnifeyMoloko commented Aug 30, 2022

avikivity commented Aug 30, 2022

Installation details

Issue description

Logs:

nyh commented Jun 15, 2023

avikivity commented Jun 15, 2023

denesb commented Dec 14, 2023

ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes #9933

ALTER TABLE with "chunk_length_kb" (compression) of 1MB caused a core dump on all nodes #9933

Comments

yarongilor commented Jan 17, 2022

slivne commented Jan 17, 2022

slivne commented Jan 17, 2022

slivne commented Jan 17, 2022

yarongilor commented Jan 18, 2022

slivne commented Feb 8, 2022

roydahan commented Feb 9, 2022

bhalevy commented Feb 13, 2022

avikivity commented Feb 15, 2022

slivne commented Apr 13, 2022

ShlomiBalalis commented Aug 16, 2022

Installation details

Issue description

Logs:

KnifeyMoloko commented Aug 30, 2022

Installation details

Issue description

Logs:

avikivity commented Aug 30, 2022

Installation details

Issue description

Logs:

nyh commented Jun 15, 2023

avikivity commented Jun 15, 2023

denesb commented Dec 14, 2023