Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault and core dump during restart and resharding #9609

Closed
yarongilor opened this issue Nov 9, 2021 · 4 comments
Closed

Segmentation fault and core dump during restart and resharding #9609

yarongilor opened this issue Nov 9, 2021 · 4 comments
Assignees
@xemul
Labels
bug
Milestone
5.0

Comments

@yarongilor
Copy link

Installation details
Kernel version: 5.4.0-1035-aws
Scylla version (or git commit hash): 4.6.dev-0.20211101.9e8fc6358 with build-id 62d8c292f9ddc15e61ebd6cd609ac91b80e1532a
Cluster size: 6 nodes (i3.4xlarge)
Scylla running with shards number (live nodes):
longevity-tls-50gb-3d-master-db-node-c5e416b4-1 (13.51.174.230 | 10.0.3.7): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-3 (13.53.106.2 | 10.0.3.237): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-5 (13.51.64.197 | 10.0.2.160): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-7 (13.49.75.142 | 10.0.2.234): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-11 (13.51.48.138 | 10.0.0.54): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-12 (13.48.70.142 | 10.0.3.91): 14 shards
Scylla running with shards number (terminated nodes):
longevity-tls-50gb-3d-master-db-node-c5e416b4-6 (16.170.157.20 | 10.0.3.19): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-2 (13.49.72.45 | 10.0.3.58): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-4 (13.49.76.98 | 10.0.3.158): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-8 (13.53.36.23 | 10.0.3.76): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-10 (13.51.157.60 | 10.0.0.137): 14 shards
longevity-tls-50gb-3d-master-db-node-c5e416b4-9 (13.51.86.152 | 10.0.3.213): 14 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-01ab7ab1a92d26fed (aws: eu-north-1)

Test: longevity-50gb-3days
Test name: longevity_test.LongevityTest.test_custom_time
Test config file(s):

Issue description

====================================

scenario:
run restart scylla with resharding on node 2.
node 2 got segmentation fault and core dump.
error details:

2021-11-05 08:06:35.315 <2021-11-05 08:06:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=43d48c06-9680-4cc6-9f28-6ebf40a58cf1: type=SEGMENTATION regex=segmentation line_number=150254 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
2021-11-05T08:06:33+00:00  longevity-tls-50gb-3d-master-db-node-c5e416b4-2 !INFO    | scylla[12341]: Segmentation fault on shard 0.
2021-11-05 08:06:35.315 <2021-11-05 08:06:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=43d48c06-9680-4cc6-9f28-6ebf40a58cf1: type=SEGMENTATION regex=segmentation line_number=150255 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
2021-11-05T08:06:33+00:00  longevity-tls-50gb-3d-master-db-node-c5e416b4-2 !INFO    | scylla[12341]: Segmentation fault on shard 0.
2021-11-05 08:06:35.339 <2021-11-05 08:06:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=43d48c06-9680-4cc6-9f28-6ebf40a58cf1: type=SEGMENTATION regex=segmentation line_number=150315 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
2021-11-05T08:06:33+00:00  longevity-tls-50gb-3d-master-db-node-c5e416b4-2 !INFO    | scylla[12341]: Segmentation fault on shard 0.
2021-11-05 08:06:35.341 <2021-11-05 08:06:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=43d48c06-9680-4cc6-9f28-6ebf40a58cf1: type=SEGMENTATION regex=segmentation line_number=150317 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
2021-11-05T08:06:33+00:00  longevity-tls-50gb-3d-master-db-node-c5e416b4-2 !INFO    | scylla[12341]: Segmentation fault on shard 0.
2021-11-05 08:09:47.657: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=f85a19ae-d68a-4463-bef0-3e78b161b5b2 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
2021-11-05 08:40:32.875: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=43a89267-f337-477e-b59b-27513e38a6c9 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-2 [13.49.72.45 | 10.0.3.58] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000.gz
backtrace=           PID: 12341 (scylla)
UID: 113 (scylla)
GID: 119 (scylla)
Signal: 11 (SEGV)
Timestamp: Fri 2021-11-05 08:06:33 UTC (3min 44s ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: 4136e1ecff8847b3970f04a50bd543ec
Machine ID: a9fd037eee714d138a6bdcb65a6ec106
Hostname: longevity-tls-50gb-3d-master-db-node-c5e416b4-2
Storage: /var/lib/systemd/coredump/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000
Message: Process 12341 (scylla) of user 113 dumped core.
Stack trace of thread 12341:
#0  0x00000000013cca79 _ZN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE16handle_exceptionIZZZZN14generic_server6server10do_acceptsEibNS_14socket_addressEENK3$_5clEvENUlNS4_INS_13accept_resultEEEE_clESC_ENKUlS5_E_clES5_EUlNSt15__exception_ptr13exception_ptrEE_EES5_OT_EUlSJ_E_ZNS5_17then_wrapped_nrvoIS5_SK_EENS_8futurizeISI_E4typeEOT0_EUlOS3_RSK_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x11cca79)
#1  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#2  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#3  0x0000000003db79dd _ZN7seastar7reactor3runEv (scylla + 0x3bb79dd)
#4  0x0000000003d6125f _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3b6125f)
#5  0x0000000003d605d7 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3b605d7)
#6  0x0000000000f414f6 main (scylla + 0xd414f6)
#7  0x00007f32115efb75 __libc_start_main (libc.so.6 + 0x27b75)
#8  0x0000000000f3e6ae _start (scylla + 0xd3e6ae)
Stack trace of thread 12367:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12358:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12363:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12364:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12357:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12345:
#0  0x00007f3211882ded _ZNSt6chrono3_V212steady_clock3nowEv (libstdc++.so.6 + 0xd0ded)
#1  0x0000000003db87f1 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb87f1)
#2  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12365:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12347:
#0  0x0000000003dc55b8 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3bc55b8)
#1  0x0000000003df18bb _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3bf18bb)
#2  0x0000000003dd4ea9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3bd4ea9)
#3  0x0000000003db87c6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb87c6)
#4  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#5  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#6  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12377:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12370:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12359:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12344:
#0  0x00007f32116c2e0d syscall (libc.so.6 + 0xfae0d)
#1  0x0000000003df98c0 _ZN7seastar19aio_storage_context11submit_workEv (scylla + 0x3bf98c0)
#2  0x0000000003dfad86 _ZN7seastar19reactor_backend_aio18kernel_submit_workEv (scylla + 0x3bfad86)
#3  0x0000000003dd4ea9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3bd4ea9)
#4  0x0000000003db87c6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb87c6)
#5  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#6  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#7  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#8  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12361:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12362:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12379:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12351:
#0  0x0000000003d7e963 _ZN7seastar6memory8allocateEm (scylla + 0x3b7e963)
#1  0x0000000003d81a82 _Znwm (scylla + 0x3b81a82)
#2  0x0000000003d79fa5 _ZNSt6vectorIN7seastar6futureIvEESaIS2_EE7reserveEm (scylla + 0x3b79fa5)
#3  0x0000000003d797c1 _ZN7seastar23parallel_for_each_stateC1Em (scylla + 0x3b797c1)
#4  0x000000000154922f _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x134922f)
#5  0x0000000001549cff _ZN22mutation_reader_mergerclEv (scylla + 0x1349cff)
#6  0x00000000015815f1 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x13815f1)
#7  0x000000000158076b _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x138076b)
#8  0x00000000011b6670 _ZN20flat_mutation_reader4peekEv (scylla + 0xfb6670)
#9  0x000000000117f9d8 _ZN5query7querierIL19emit_only_live_rows1EE12consume_pageI20query_result_builderEEDaOT_mjNSt6chrono10time_pointI8gc_clockNS7_8durationIlSt5ratioILl1ELl1EEEEEEN7tracing15trace_state_ptrE (scylla + 0xf7f9d8)
#10 0x000000000117a5a4 _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf7a5a4)
#11 0x00000000010f97ec _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef97ec)
#12 0x00000000033db014 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31db014)
#13 0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#14 0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#15 0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#16 0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#17 0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#18 0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12354:
#0  0x0000000003dc55b8 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3bc55b8)
#1  0x0000000003df18bb _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3bf18bb)
#2  0x0000000003dd4ea9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3bd4ea9)
#3  0x0000000003db87c6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb87c6)
#4  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#5  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#6  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12378:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12352:
#0  0x00007f32116c2e0d syscall (libc.so.6 + 0xfae0d)
#1  0x0000000003df98c0 _ZN7seastar19aio_storage_context11submit_workEv (scylla + 0x3bf98c0)
#2  0x0000000003dfad86 _ZN7seastar19reactor_backend_aio18kernel_submit_workEv (scylla + 0x3bfad86)
#3  0x0000000003dd4ea9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3bd4ea9)
#4  0x0000000003db87c6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb87c6)
#5  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#6  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#7  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#8  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12360:
#0  0x00007f32121bf94c read (libpthread.so.0 + 0x1294c)
#1  0x0000000003df8f25 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bf8f25)
#2  0x0000000003df9290 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bf9290)
#3  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#4  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12350:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12342:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12349:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12353:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12348:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12343:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 12346:
#0  0x00007f32121bf9db __close (libpthread.so.0 + 0x129db)
#1  0x0000000003dfd5f3 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3bfd5f3)
#2  0x000000000410ca60 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0ca60)
#3  0x000000000410cc5d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0cc5d)
#4  0x0000000003db73a5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bb73a5)
#5  0x0000000003db8798 _ZN7seastar7reactor6do_runEv (scylla + 0x3bb8798)
#6  0x0000000003dd5dc6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd5dc6)
#7  0x0000000003d8f9db _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b8f9db)
#8  0x00007f32121b6299 start_thread (libpthread.so.0 + 0x9299)
#9  0x00007f32116c8353 __clone (libc.so.6 + 0x100353)
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000.gz

Download core dump by:
gsutil cp gs://upload.scylladb.com/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.4136e1ecff8847b3970f04a50bd543ec.12341.1636099593000000000000.gz

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor c5e416b4-63c6-458e-b8d2-87bef019c6fc
Show all stored logs command: $ hydra investigate show-logs c5e416b4-63c6-458e-b8d2-87bef019c6fc

Test id: c5e416b4-63c6-458e-b8d2-87bef019c6fc

Logs:
critical - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/critical.log.tar.gz
db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/db-cluster-c5e416b4.tar.gz
debug - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/debug.log.tar.gz
email_data - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/email_data.json.tar.gz
error - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/error.log.tar.gz
event - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/events.log.tar.gz
left_processes - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/left_processes.log.tar.gz
loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/loader-set-c5e416b4.tar.gz
monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/monitor-set-c5e416b4.tar.gz
normal - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/normal.log.tar.gz
output - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/output.log.tar.gz
event - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/raw_events.log.tar.gz
summary - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/summary.log.tar.gz
warning - https://cloudius-jenkins-test.s3.amazonaws.com/c5e416b4-63c6-458e-b8d2-87bef019c6fc/20211107_062411/warning.log.tar.gz

Jenkins job URL
@yarongilor
Copy link
Author

Another segmentation fault and core dump happened later on during enospc nemesis on node 10:

(DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=43d48c06-9680-4cc6-9f28-6ebf40a58cf1: type=SEGMENTATION regex=segmentation line_number=984662 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-10 [13.51.157.60 | 10.0.0.137] (seed: False)
2021-11-06T06:07:20+00:00 longevity-tls-50gb-3d-master-db-node-c5e416b4-10 !INFO | scylla[30536]: Segmentation fault on shard 11.

(CoreDumpEvent Severity.ERROR) period_type=one-time event_id=f6d2e5ac-e3d7-4724-bc54-0fdbd652c312 node=Node longevity-tls-50gb-3d-master-db-node-c5e416b4-10 [13.51.157.60 | 10.0.0.137] (seed: False)

@roydahan roydahan added the triage/master Looking for assignee label Nov 10, 2021
@slivne slivne added master/high and removed triage/master Looking for assignee labels Nov 10, 2021
@slivne slivne added this to the 4.6 milestone Nov 10, 2021
@xemul
Copy link
Contributor

xemul commented Nov 10, 2021

@slivne , dup of #9316

@yarongilor
Copy link
Author

Reproduced in next run of 50gb longevity as well:



RestartWithResharding | 2 | nodestartenddurationNode longevity-tls-50gb-3d-master-db-node-1e8c5caf-2 [16.170.173.145 \| 10.0.3.156] (seed: False)2021-11-13 02:43:322021-11-13 03:04:161244Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-4 [13.53.170.171 \| 10.0.0.245] (seed: False)2021-11-14 12:20:172021-11-14 12:41:411283 | node | start | end | duration | Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-2 [16.170.173.145 \| 10.0.3.156] (seed: False) | 2021-11-13 02:43:32 | 2021-11-13 03:04:16 | 1244 | Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-4 [13.53.170.171 \| 10.0.0.245] (seed: False) | 2021-11-14 12:20:17 | 2021-11-14 12:41:41 | 1283
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --

Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-2 [16.170.173.145 \| 10.0.3.156] (seed: False) | 2021-11-13 02:43:32 | 2021-11-13 03:04:16 | 1244
Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-4 [13.53.170.171 \| 10.0.0.245] (seed: False) | 2021-11-14 12:20:17 | 2021-11-14 12:41:41 | 1283

2021-11-14 13:08:37.761 <2021-11-14 13:07:09.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=23badec6-5f41-4e21-bc76-e6cfe6ff55c7: type=SEGMENTATION regex=segmentation line_number=670467 node=Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-7 [13.53.182.234 | 10.0.2.237] (seed: False)
2021-11-14T13:07:09+00:00 longevity-tls-50gb-3d-master-db-node-1e8c5caf-7 !INFO | scylla[211569]: Segmentation fault on shard 0.
2021-11-14 13:10:33.743: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=fbbc3ad8-f1b2-4de0-bd13-0b3edf3ba66c node=Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-7 [13.53.182.234 | 10.0.2.237] (seed: False)
2021-11-14 14:07:59.275: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=90661b9a-76b0-455b-b0fe-3d2661d65618 node=Node longevity-tls-50gb-3d-master-db-node-1e8c5caf-7 [13.53.182.234 | 10.0.2.237] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000.gz
backtrace= PID: 211569 (scylla)
UID: 113 (scylla)
GID: 119 (scylla)
Signal: 11 (SEGV)
Timestamp: Sun 2021-11-14 13:07:09 UTC (3min 55s ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: 96bab7a3d9154821a3740099b0102cba
Machine ID: 242f96a4f688488fa27259e26090b8a5
Hostname: longevity-tls-50gb-3d-master-db-node-1e8c5caf-7
Storage: /var/lib/systemd/coredump/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000
Message: Process 211569 (scylla) of user 113 dumped core.
Stack trace of thread 211569:
#0 0x00000000013cd3f9 _ZN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE16handle_exceptionIZZZZN14generic_server6server10do_acceptsEibNS_14socket_addressEENK3$_5clEvENUlNS4_INS_13accept_resultEEEE_clESC_ENKUlS5_E_clES5_EUlNSt15__exception_ptr13exception_ptrEE_EES5_OT_EUlSJ_E_ZNS5_17then_wrapped_nrvoIS5_SK_EENS_8futurizeISI_E4typeEOT0_EUlOS3_RSK_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x11cd3f9)
#1 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#2 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#3 0x0000000003dbaafd _ZN7seastar7reactor3runEv (scylla + 0x3bbaafd)
#4 0x0000000003d6438f _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3b6438f)
#5 0x0000000003d63707 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3b63707)
#6 0x0000000000f41976 main (scylla + 0xd41976)
#7 0x00007f183406ab75 __libc_start_main (libc.so.6 + 0x27b75)
#8 0x0000000000f3eb2e _start (scylla + 0xd3eb2e)
Stack trace of thread 211587:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211592:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211598:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211585:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211593:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211591:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211594:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211596:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211597:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211589:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211595:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211588:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211580:
#0 0x0000000003dbe077 _ZN7seastar17smp_message_queue19process_completionsEj (scylla + 0x3bbe077)
#1 0x0000000003dc863d _ZN7seastar3smp11poll_queuesEv (scylla + 0x3bc863d)
#2 0x0000000003df49db _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3bf49db)
#3 0x0000000003dd7fc9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3bd7fc9)
#4 0x0000000003dbb8e6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8e6)
#5 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#6 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#7 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#8 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211572:
#0 0x0000000003d84798 free (scylla + 0x3b84798)
#1 0x00007f18352c14ce _mbuffer_head_remove_bytes (libgnutls.so.30 + 0x424ce)
#2 0x00007f18352b9a8d _gnutls_recv_in_buffers (libgnutls.so.30 + 0x3aa8d)
#3 0x00007f18352bb762 _gnutls_recv_int (libgnutls.so.30 + 0x3c762)
#4 0x00000000040f96e3 _ZN7seastar3tls7session6do_getEv (scylla + 0x3ef96e3)
#5 0x00000000040fa676 ZZN7seastar6futureIvE14then_impl_nrvoIZNS_3tls7session6do_getEvEUlvE_NS0_INS_16temporary_bufferIcEEEEEET0_OT_ENKUlONS_8internal22promise_base_with_typeIS7_EERS5_ONS_12future_stateINSC_9monostateEEEE_clESF_SG_SK (scylla + 0x3efa676)
#6 0x00000000040fa5cf _ZN7seastar12continuationINS_8internal22promise_base_with_typeINS_16temporary_bufferIcEEEEZNS_3tls7session6do_getEvEUlvE_ZNS_6futureIvE14then_impl_nrvoIS8_NS9_IS4_EEEET0_OT_EUlOS5_RS8_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x3efa5cf)
#7 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#8 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#9 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#10 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#11 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#12 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211590:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211586:
#0 0x00007f1834c3a94c read (libpthread.so.0 + 0x1294c)
#1 0x0000000003dfc045 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3bfc045)
#2 0x0000000003dfc3b0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3bfc3b0)
#3 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#4 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#5 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211582:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211570:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211575:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211574:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211576:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211581:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211577:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211578:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211571:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211573:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 211579:
#0 0x00007f1834c3a9db __close (libpthread.so.0 + 0x129db)
#1 0x0000000003e00713 _ZN7seastar21aio_pollable_fd_stateD0Ev (scylla + 0x3c00713)
#2 0x000000000410fc50 _ZZN7seastar10fsnotifier4impl4waitEvENK3$_0clEm (scylla + 0x3f0fc50)
#3 0x000000000410fe4d _ZN7seastar12continuationINS_8internal22promise_base_with_typeISt6vectorINS_10fsnotifier5eventESaIS5_EEEEZNS4_4impl4waitEvE3$_0ZNS_6futureImE14then_impl_nrvoISA_NSB_IS7_EEEET0_OT_EUlOS8_RSA_ONS_12future_stateImEEE_mE15run_and_disposeEv (scylla + 0x3f0fe4d)
#4 0x0000000003dba4c5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bba4c5)
#5 0x0000000003dbb8b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bbb8b8)
#6 0x0000000003dd8ee6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3bd8ee6)
#7 0x0000000003d92afb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b92afb)
#8 0x00007f1834c31299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f1834143353 __clone (libc.so.6 + 0x100353)
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.96bab7a3d9154821a3740099b0102cba.211569.1636895229000000000000.gz

@bhalevy
Copy link
Member

bhalevy commented Nov 18, 2021 •
edited

Can be closed as duplicate of #9316 once 3388718 is promoted to branch-4.6

@slivne slivne closed this as completed Nov 22, 2021
@DoronArazii DoronArazii modified the milestones: 4.6, 5.0 Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xemul xemul

Labels
bug
Projects
None yet
Milestone
5.0
Development

No branches or pull requests
6 participants
@slivne @xemul @yarongilor @bhalevy @roydahan @DoronArazii