Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression on I3en with Linux 5.8 #7036

Closed
aleksbykov opened this issue Aug 13, 2020 · 46 comments
Closed

Performance regression on I3en with Linux 5.8 #7036

aleksbykov opened this issue Aug 13, 2020 · 46 comments
Assignees
@penberg
Labels
cloud/aws AWS related issues
Milestone
4.3

Comments

@aleksbykov
Copy link
Contributor

Installation details
Scylla version (or git commit hash): version 4.2.rc2-0.20200811.b052f3f5cea
Cluster size: 3
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-04983b862895e0534

Job longevity-8h-large-num-columns was failed.

gemini command

gemini -d --duration 7h --warmup 1h -c 20 -m mixed -f --non-interactive --cql-features normal --async-objects-stabilization-backoff 500ms --replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '3'}" --max-mutation-retries 5 --max-mutation-retries-backoff 500ms --max-partition-keys 12 --min-partition-keys 8 --max-clustering-keys 20 --min-clustering-keys 12 --max-columns 100 --min-columns 80 --verbose --test-cluster=10.0.3.138 --outfile /home/centos/gemini_result_49145060-43a3-4607-965d-72438f6dc51f.log --seed 9

the command generated table with 9 partition keys and 16 clustering keys and 80 columns.

on node3 scylla was stopped, several tables were removed:

!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/ks1/table1-* -maxdepth 1 -type f
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-CompressionInfo.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Data.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Digest.crc32 /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Filter.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Index.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Scylla.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Statistics.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-Summary.db /var/lib/scylla/data/ks1/table1-97397d31dc9011ea843b000000000004/mc-17-big-TOC.txt
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/ks1/table1_col10_idx_index-* -maxdepth 1 -type f
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-CompressionInfo.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Data.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Digest.crc32 /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Filter.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Index.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Scylla.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Statistics.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-Summary.db /var/lib/scylla/data/ks1/table1_col10_idx_index-9b3080f0dc9011eab362000000000000/mc-19-big-TOC.txt
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/ks1/table1_col30_idx_index-* -maxdepth 1 -type f
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-CompressionInfo.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Data.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Digest.crc32 /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Filter.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Index.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Scylla.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Statistics.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-Summary.db /var/lib/scylla/data/ks1/table1_col30_idx_index-9d121232dc9011eab626000000000007/mc-12-big-TOC.txt
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/ks1/table1_col5_idx_index-* -maxdepth 1 -type f
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-CompressionInfo.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Data.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Digest.crc32 /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Filter.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Index.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Scylla.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Statistics.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-Summary.db /var/lib/scylla/data/ks1/table1_col5_idx_index-99851ae2dc9011ea9ccf000000000006/mc-16-big-TOC.txt
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/ks1/table1_col7_idx_index-* -maxdepth 1 -type f
!NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-CompressionInfo.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Data.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Digest.crc32 /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Filter.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Index.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Scylla.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Statistics.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-Summary.db /var/lib/scylla/data/ks1/table1_col7_idx_index-9a0c4c40dc9011ea89f4000000000006/mc-12-big-TOC.txt

After that scylla was started again. The start took about 20 minutes.

2020-08-12T12:38:01+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Scylla version 4.2.rc2-0.20200811.b052f3f5cea with build-id 7e2fb1a513d6ad0d43ae3361d41c375f045f24d9 starting ...
2020-08-12T12:58:52+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - Scylla version 4.2.rc2-0.20200811.b052f3f5cea initialization completed.

During scylla initialization detected a lot of reactor stalls with delays up to 14400ms and messages:

2020-08-12T12:38:06+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Restarting a node in NORMAL status
2020-08-12T12:38:06+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] database - Schema version changed to 8c2fea4b-7722-3e47-952a-97b91010bd41
2020-08-12T12:38:06+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Starting up server gossip
2020-08-12T12:38:06+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 4] compaction - Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/mc-232-big-Data.db:level=0, /var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/mc-220-big-Data.db:level=0, ]
2020-08-12T12:38:06+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 4] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/mc-244-big-Data.db:level=0, ]. 19kB to 12kB (~63% of original) in 72ms = 171kB/s. ~256 total partitions merged to 1.
2020-08-12T12:38:09+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Node 10.0.3.138 state jump to normal
2020-08-12T12:38:09+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Node 10.0.2.154 state jump to normal
2020-08-12T12:38:09+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - InetAddress 10.0.2.154 is now UP, status = NORMAL
2020-08-12T12:38:10+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - InetAddress 10.0.3.138 is now UP, status = NORMAL
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[18370]: Accepted publickey for centos from 10.0.7.29 port 39298 ssh2: RSA SHA256:g7OmDbWC8qLUZHxjMohQe3LsQHfcOJbG1ZHqodB6smc
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd: Started Scylla Housekeeping restart mode.
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: New session 735 of user centos.
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd: Started Session 735 of user centos.
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[18370]: pam_unix(sshd:session): session opened for user centos by (uid=0)
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/coredumpctl --no-pager --no-legend
2020-08-12T12:38:11+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-08-12T12:38:12+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-08-12T12:38:12+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla-housekeeping: /opt/scylladb/scripts/libexec/scylla-housekeeping:195: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
2020-08-12T12:38:12+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla-housekeeping: config = configparser.SafeConfigParser()
2020-08-12T12:38:13+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[18370]: pam_unix(sshd:session): session closed for user centos
2020-08-12T12:38:13+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: Removed session 735.
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Remove node 10.0.2.154 from pending replacing endpoint
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] storage_service - Remove node 10.0.3.138 from pending replacing endpoint
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 7] compaction - Compacting [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-79-big-Data.db:level=0, /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-67-big-Data.db:level=0, ]
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 2] compaction - Compacting [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-74-big-Data.db:level=0, /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-62-big-Data.db:level=0, ]
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 2] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-86-big-Data.db:level=0, ]. 22kB to 11kB (~50% of original) in 11ms = 1MB/s. ~256 total partitions merged to 1.
2020-08-12T12:38:15+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 7] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-91-big-Data.db:level=0, ]. 22kB to 11kB (~50% of original) in 12ms = 942kB/s. ~256 total partitions merged to 1.
2020-08-12T12:38:16+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 1 polls.
2020-08-12T12:38:18+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 2 polls.
2020-08-12T12:38:19+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 3 polls.
2020-08-12T12:38:20+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 4 polls.
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 6 polls.
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Reactor stalled for 1110 ms on shard 11.
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Backtrace:
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002ec2402
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66940
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66e11
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e67137
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f23674d6a8f
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e18da3
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e225ae
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e22959
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e344dd
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e63ce7
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e6405e
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e9b17d
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002eaab7a
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e2edad
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-08-12T12:38:22+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912

Decoded backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:751
seastar::internal::cpu_stall_detector::generate_trace() at /usr/include/fmt/format.h:2188
seastar::internal::cpu_stall_detector::maybe_report() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1060
 (inlined by) seastar::reactor::block_notifier(int) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1183
?? ??:0
seastar::memory::small_pool::allocate() at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::memory::cpu_pages::allocate_small(unsigned int) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/memory.cc:738
 (inlined by) seastar::memory::allocate(unsigned long) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/memory.cc:1253
 (inlined by) malloc at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/memory.cc:1495
seastar::basic_sstring<char, unsigned int, 15u, true>::basic_sstring(seastar::basic_sstring<char, unsigned int, 15u, true> const&) at /usr/include/c++/10/ext/new_allocator.h:106
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true>::basic_sstring(seastar::basic_sstring<char, unsigned int, 15u, true> const&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/sstring.hh:114
 (inlined by) seastar::metrics::impl::metric_family_info::metric_family_info(seastar::metrics::impl::metric_family_info const&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/metrics_api.hh:154
 (inlined by) seastar::metrics::impl::impl::update_metrics_if_needed() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/metrics.cc:307
seastar::metrics::impl::impl::metadata() at /usr/include/c++/10/ext/new_allocator.h:106
 (inlined by) seastar::metrics::impl::get_values() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/metrics.cc:261
_ZN7seastar17smp_message_queue15async_work_itemIZZNS_10prometheusL13get_map_valueERNS2_26metrics_families_per_shardEENKUlT_E_clIjEEDaS5_EUlvE_E15run_and_disposeEv at prometheus.cc:?
 (inlined by) invoke<seastar::prometheus::get_map_value(seastar::prometheus::metrics_families_per_shard&)::<lambda(auto:72)> [with auto:72 = unsigned int]::<lambda()> > at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1961
 (inlined by) run_and_dispose at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/smp.hh:219
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:2712
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Reactor stalled for 2439 ms on shard 8.
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Backtrace:
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002ec2402
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66940
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66e11
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e67137
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f23674d6a8f
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f23674d59ab
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e7aaf0
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e63ce7
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e6405e
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e9b17d
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002eaab7a
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e2edad
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-08-12T12:38:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912

Decoded backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:751
seastar::internal::cpu_stall_detector::generate_trace() at /usr/include/fmt/format.h:2188
seastar::internal::cpu_stall_detector::maybe_report() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1060
 (inlined by) seastar::reactor::block_notifier(int) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1183
?? ??:0
?? ??:0
seastar::continuation<seastar::internal::promise_base_with_type<seastar::temporary_buffer<char> >, seastar::reactor::do_read_some(seastar::pollable_fd_state&, seastar::internal::buffer_allocator*)::{lambda()#1}, seastar::reactor::do_read_some(seastar::pollable_fd_state&, seastar::internal::buffer_allocator*)::{lambda()#1}<seastar::temporary_buffer<char> > seastar::future<>::then_impl_nrvo<{lambda()#1}, seastar::reactor::do_read_some(seastar::pollable_fd_state&, seastar::internal::buffer_allocator*)::{lambda()#1}<seastar::temporary_buffer<char> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::temporary_buffer<char> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>::run_and_dispose() at /usr/include/fmt/format.h:2188
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:2712
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

2020-08-12T12:42:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 257 polls.
2020-08-12T12:42:45+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 258 polls.
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Reactor stalled for 7980 ms on shard 9.
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Backtrace:
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002ec2402
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66940
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66e11
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e67137
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f23674d6a8f
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f2366bb2512
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000100d1cd
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002103ef0
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000021b6889
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000021b9441
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000222a17b
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000222b80d
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000032831a5
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e63ce7
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e6405e
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e9b17d
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002eaab7a
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e2edad
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912
2020-08-12T12:42:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 259 polls.

Decoded backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:751
seastar::internal::cpu_stall_detector::generate_trace() at /usr/include/fmt/format.h:2188
seastar::internal::cpu_stall_detector::maybe_report() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1060
 (inlined by) seastar::reactor::block_notifier(int) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1183
?? ??:0
?? ??:0
signed char* std::__copy_move<false, true, std::random_access_iterator_tag>::__copy_m<signed char>(signed char const*, signed char const*, signed char*) at /usr/include/boost/intrusive/bstree_algorithms.hpp:1565
 (inlined by) signed char* std::__copy_move_a2<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:472
 (inlined by) signed char* std::__copy_move_a1<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:506
 (inlined by) signed char* std::__copy_move_a<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:513
 (inlined by) signed char* std::copy<signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:569
 (inlined by) signed char* std::__copy_n<signed char*, unsigned int, signed char*>(signed char*, unsigned int, signed char*, std::random_access_iterator_tag) at /usr/include/c++/10/bits/stl_algo.h:750
 (inlined by) signed char* std::copy_n<signed char*, unsigned int, signed char*>(signed char*, unsigned int, signed char*) at /usr/include/c++/10/bits/stl_algo.h:777
 (inlined by) bytes_ostream::linearize() at /jenkins/workspace/scylla-4.2/next/scylla/./bytes_ostream.hh:302
 (inlined by) bytes_ostream::reduce_chunk_count() at /jenkins/workspace/scylla-4.2/next/scylla/./bytes_ostream.hh:408
 (inlined by) frozen_mutation::frozen_mutation(bytes_ostream&&) at /jenkins/workspace/scylla-4.2/next/scylla/frozen_mutation.cc:83
frozen_mutation ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&)::{lambda(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&)#1}::operator()<seastar::fragmented_memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(auto, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) const at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZN7seastar19memory_input_streamIN9__gnu_cxx17__normal_iteratorIPNS_16temporary_bufferIcEESt6vectorIS4_SaIS4_EEEEE11with_streamIZN3ser10serializerI15frozen_mutationE4readISA_EESE_RT_EUlSI_E_EEDcOSH_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/simple-stream.hh:472
 (inlined by) with_serialized_stream<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char> > > >, ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char> > > > >::<lambda(auto:163&)> > at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/simple-stream.hh:626
 (inlined by) frozen_mutation ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/build/release/gen/idl/frozen_mutation.dist.impl.hh:47
auto ser::deserialize<frozen_mutation, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&, boost::type<frozen_mutation>) at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) frozen_mutation netw::read<frozen_mutation, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&, boost::type<frozen_mutation>) at /jenkins/workspace/scylla-4.2/next/scylla/message/messaging_service.cc:108
 (inlined by) seastar::rpc::unmarshal_one<netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >::helper<frozen_mutation>::doit(seastar::rpc::connection&, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:285
 (inlined by) std::tuple<frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > > seastar::rpc::do_unmarshall<netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > >(seastar::rpc::connection&, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:327
 (inlined by) std::tuple<frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > > seastar::rpc::unmarshall<netw::serializer, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > >(seastar::rpc::connection&, seastar::rpc::rcv_buf) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:335
 (inlined by) _ZZZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS0_12no_wait_typeEEEERKNS0_11client_infoENS0_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISF_EESF_jmNS0_8optionalISt8optionalIN7tracing10trace_infoEEEEEES7_JSC_SH_SF_jmSN_ENS0_19do_want_client_infoENS0_18do_want_time_pointEEEDaNS0_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS0_6server10connectionEEESJ_INSt6chrono10time_pointINS_12lowres_clockENS16_8durationIlSt5ratioILl1ELl1000EEEEEEElNS0_7rcv_bufEE_clES15_S1E_lS1F_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES18_EEEEDaS1H_ENUlvE_clEv at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:584
_ZN7seastar8futurizeINS_6futureIJEEEE6invokeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS1_IJNS5_12no_wait_typeEEEERKNS5_11client_infoENS5_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISJ_EESJ_jmNS5_8optionalISt8optionalIN7tracing10trace_infoEEEEEESB_JSG_SL_SJ_jmSR_ENS5_19do_want_client_infoENS5_18do_want_time_pointEEEDaNS5_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS5_6server10connectionEEESN_INSt6chrono10time_pointINS_12lowres_clockENS1A_8durationIlSt5ratioILl1ELl1000EEEEEEElNS5_7rcv_bufEE_clES19_S1I_lS1J_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1C_EEEEDaS1L_EUlvE_JEEES2_OS1L_DpOT0_ at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZN7seastar15futurize_invokeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES19_EEEEDaS1I_EUlvE_JEEEDaOS1I_DpOT0_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) _ZN7seastar8internal21invoke_func_with_gateIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS2_12no_wait_typeEEEERKNS2_11client_infoENS2_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISH_EESH_jmNS2_8optionalISt8optionalIN7tracing10trace_infoEEEEEES9_JSE_SJ_SH_jmSP_ENS2_19do_want_client_infoENS2_18do_want_time_pointEEEDaNS2_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS2_6server10connectionEEESL_INSt6chrono10time_pointINS_12lowres_clockENS18_8durationIlSt5ratioILl1ELl1000EEEEEEElNS2_7rcv_bufEE_clES17_S1G_lS1H_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1A_EEEEDaS1J_EUlvE_EEDaRNS_4gateEOS1J_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/gate.hh:126
 (inlined by) _ZN7seastar13try_with_gateIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES19_EEEEDaS1I_EUlvE_EEDaRNS_4gateEOS1I_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/gate.hh:166
 (inlined by) _ZZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS0_12no_wait_typeEEEERKNS0_11client_infoENS0_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISF_EESF_jmNS0_8optionalISt8optionalIN7tracing10trace_infoEEEEEES7_JSC_SH_SF_jmSN_ENS0_19do_want_client_infoENS0_18do_want_time_pointEEEDaNS0_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS0_6server10connectionEEESJ_INSt6chrono10time_pointINS_12lowres_clockENS16_8durationIlSt5ratioILl1ELl1000EEEEEEElNS0_7rcv_bufEE_clES15_S1E_lS1F_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES18_EEEEDaS1H_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:582
_ZSt13__invoke_implIvZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_JNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEES1I_St14__invoke_otherS10_DpOT1_ at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZSt8__invokeIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_JNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEENSt15__invoke_resultIS1I_JDpT0_EE4typeEOS1I_DpOS1O_ at /usr/include/c++/10/bits/invoke.h:95
 (inlined by) _ZSt12__apply_implIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_St5tupleIJNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEEJLm0EEEDcOS1I_S10_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1723
 (inlined by) _ZSt5applyIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_St5tupleIJNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEEEDcOS1I_S10_ at /usr/include/c++/10/tuple:1734
 (inlined by) _ZN7seastar8futurizeIvE5applyIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS3_12no_wait_typeEEEERKNS3_11client_infoENS3_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISI_EESI_jmNS3_8optionalISt8optionalIN7tracing10trace_infoEEEEEESA_JSF_SK_SI_jmSQ_ENS3_19do_want_client_infoENS3_18do_want_time_pointEEEDaNS3_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS3_6server10connectionEEESM_INSt6chrono10time_pointINS_12lowres_clockENS19_8durationIlSt5ratioILl1ELl1000EEEEEEElNS3_7rcv_bufEE_clES18_S1H_lS1I_EUlT_E1_JNS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1B_EEEEENS8_IJEEEOS1K_OSt5tupleIJDpT0_EE at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1976
 (inlined by) _ZN7seastar6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEE9then_implIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_IJNS7_12no_wait_typeEEEERKNS7_11client_infoENS7_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISL_EESL_jmNS7_8optionalISt8optionalIN7tracing10trace_infoEEEEEESD_JSI_SN_SL_jmST_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESP_INSt6chrono10time_pointIS3_NS1C_8durationIlSt5ratioILl1ELl1000EEEEEEElNS7_7rcv_bufEE_clES1B_S1J_lS1K_EUlT_E1_NS0_IJEEEEES14_OS1M_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1544
 (inlined by) _ZN7seastar8internal14call_then_implINS_6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEEEE3runIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS2_IJNSA_12no_wait_typeEEEERKNSA_11client_infoENSA_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISO_EESO_jmNSA_8optionalISt8optionalIN7tracing10trace_infoEEEEEESG_JSL_SQ_SO_jmSW_ENSA_19do_want_client_infoENSA_18do_want_time_pointEEEDaNSA_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINSA_6server10connectionEEESS_INSt6chrono10time_pointIS5_NS1F_8durationIlSt5ratioILl1ELl1000EEEEEEElNSA_7rcv_bufEE_clES1E_S1M_lS1N_EUlT_E1_EEDaRS7_OS1P_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1198
 (inlined by) _ZN7seastar6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEE4thenIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_IJNS7_12no_wait_typeEEEERKNS7_11client_infoENS7_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISL_EESL_jmNS7_8optionalISt8optionalIN7tracing10trace_infoEEEEEESD_JSI_SN_SL_jmST_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESP_INSt6chrono10time_pointIS3_NS1C_8durationIlSt5ratioILl1ELl1000EEEEEEElNS7_7rcv_bufEE_clES1B_S1J_lS1K_EUlT_E1_NS0_IJEEEEES14_OS1M_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1468
 (inlined by) auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:580
seastar::future<> std::__invoke_impl<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>(std::__invoke_other, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::shared_ptr<seastar::rpc::server::connection>&&, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >&&, long&&, seastar::rpc::rcv_buf&&) at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) std::enable_if<is_invocable_r_v<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>, std::enable_if>::type std::__invoke_r<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>(std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, (seastar::future<>&&)...) at /usr/include/c++/10/bits/invoke.h:115
 (inlined by) std::_Function_handler<seastar::future<> (seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf), auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}>::_M_invoke(std::_Any_data const&, seastar::shared_ptr<seastar::rpc::server::connection>&&, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >&&, long&&, seastar::rpc::rcv_buf&&) at /usr/include/c++/10/bits/std_function.h:292
std::function<seastar::future<> (seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)>::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf) const at /usr/include/c++/10/ext/new_allocator.h:106
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/rpc/rpc.cc:935
 (inlined by) __invoke_impl<seastar::future<>, seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1724
 (inlined by) apply<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1736
 (inlined by) apply<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2009
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future-util.hh:86
 (inlined by) invoke<seastar::with_scheduling_group<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>, {}>::<lambda()>&> at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) run_and_dispose at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/make_task.hh:40
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:2712
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

2020-08-12T12:42:51+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 264 polls.
2020-08-12T12:42:52+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 265 polls.
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Reactor stalled for 14390 ms on shard 9.
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: Backtrace:
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002ec2402
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66940
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e66e11
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e67137
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f23674d6a8f
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00007f2366bb2512
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000100d1cd
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002103ef0
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000021b6889
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000021b9441
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000222a17b
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x000000000222b80d
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x00000000032831a5
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e63ce7
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e6405e
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e9b17d
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002eaab7a
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: 0x0000000002e2edad
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-08-12T12:42:53+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912

Decoded backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:751
seastar::internal::cpu_stall_detector::generate_trace() at /usr/include/fmt/format.h:2188
seastar::internal::cpu_stall_detector::maybe_report() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1060
 (inlined by) seastar::reactor::block_notifier(int) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:1183
?? ??:0
?? ??:0
signed char* std::__copy_move<false, true, std::random_access_iterator_tag>::__copy_m<signed char>(signed char const*, signed char const*, signed char*) at /usr/include/boost/intrusive/bstree_algorithms.hpp:1565
 (inlined by) signed char* std::__copy_move_a2<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:472
 (inlined by) signed char* std::__copy_move_a1<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:506
 (inlined by) signed char* std::__copy_move_a<false, signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:513
 (inlined by) signed char* std::copy<signed char*, signed char*>(signed char*, signed char*, signed char*) at /usr/include/c++/10/bits/stl_algobase.h:569
 (inlined by) signed char* std::__copy_n<signed char*, unsigned int, signed char*>(signed char*, unsigned int, signed char*, std::random_access_iterator_tag) at /usr/include/c++/10/bits/stl_algo.h:750
 (inlined by) signed char* std::copy_n<signed char*, unsigned int, signed char*>(signed char*, unsigned int, signed char*) at /usr/include/c++/10/bits/stl_algo.h:777
 (inlined by) bytes_ostream::linearize() at /jenkins/workspace/scylla-4.2/next/scylla/./bytes_ostream.hh:302
 (inlined by) bytes_ostream::reduce_chunk_count() at /jenkins/workspace/scylla-4.2/next/scylla/./bytes_ostream.hh:408
 (inlined by) frozen_mutation::frozen_mutation(bytes_ostream&&) at /jenkins/workspace/scylla-4.2/next/scylla/frozen_mutation.cc:83
frozen_mutation ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&)::{lambda(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&)#1}::operator()<seastar::fragmented_memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(auto, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) const at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZN7seastar19memory_input_streamIN9__gnu_cxx17__normal_iteratorIPNS_16temporary_bufferIcEESt6vectorIS4_SaIS4_EEEEE11with_streamIZN3ser10serializerI15frozen_mutationE4readISA_EESE_RT_EUlSI_E_EEDcOSH_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/simple-stream.hh:472
 (inlined by) with_serialized_stream<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char> > > >, ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char> > > > >::<lambda(auto:163&)> > at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/simple-stream.hh:626
 (inlined by) frozen_mutation ser::serializer<frozen_mutation>::read<seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/build/release/gen/idl/frozen_mutation.dist.impl.hh:47
auto ser::deserialize<frozen_mutation, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&, boost::type<frozen_mutation>) at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) frozen_mutation netw::read<frozen_mutation, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >(netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&, boost::type<frozen_mutation>) at /jenkins/workspace/scylla-4.2/next/scylla/message/messaging_service.cc:108
 (inlined by) seastar::rpc::unmarshal_one<netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > > >::helper<frozen_mutation>::doit(seastar::rpc::connection&, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:285
 (inlined by) std::tuple<frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > > seastar::rpc::do_unmarshall<netw::serializer, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > >(seastar::rpc::connection&, seastar::memory_input_stream<__gnu_cxx::__normal_iterator<seastar::temporary_buffer<char>*, std::vector<seastar::temporary_buffer<char>, std::allocator<seastar::temporary_buffer<char> > > > >&) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:327
 (inlined by) std::tuple<frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > > seastar::rpc::unmarshall<netw::serializer, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> > >(seastar::rpc::connection&, seastar::rpc::rcv_buf) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:335
 (inlined by) _ZZZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS0_12no_wait_typeEEEERKNS0_11client_infoENS0_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISF_EESF_jmNS0_8optionalISt8optionalIN7tracing10trace_infoEEEEEES7_JSC_SH_SF_jmSN_ENS0_19do_want_client_infoENS0_18do_want_time_pointEEEDaNS0_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS0_6server10connectionEEESJ_INSt6chrono10time_pointINS_12lowres_clockENS16_8durationIlSt5ratioILl1ELl1000EEEEEEElNS0_7rcv_bufEE_clES15_S1E_lS1F_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES18_EEEEDaS1H_ENUlvE_clEv at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:584
_ZN7seastar8futurizeINS_6futureIJEEEE6invokeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS1_IJNS5_12no_wait_typeEEEERKNS5_11client_infoENS5_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISJ_EESJ_jmNS5_8optionalISt8optionalIN7tracing10trace_infoEEEEEESB_JSG_SL_SJ_jmSR_ENS5_19do_want_client_infoENS5_18do_want_time_pointEEEDaNS5_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS5_6server10connectionEEESN_INSt6chrono10time_pointINS_12lowres_clockENS1A_8durationIlSt5ratioILl1ELl1000EEEEEEElNS5_7rcv_bufEE_clES19_S1I_lS1J_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1C_EEEEDaS1L_EUlvE_JEEES2_OS1L_DpOT0_ at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZN7seastar15futurize_invokeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES19_EEEEDaS1I_EUlvE_JEEEDaOS1I_DpOT0_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) _ZN7seastar8internal21invoke_func_with_gateIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS2_12no_wait_typeEEEERKNS2_11client_infoENS2_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISH_EESH_jmNS2_8optionalISt8optionalIN7tracing10trace_infoEEEEEES9_JSE_SJ_SH_jmSP_ENS2_19do_want_client_infoENS2_18do_want_time_pointEEEDaNS2_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS2_6server10connectionEEESL_INSt6chrono10time_pointINS_12lowres_clockENS18_8durationIlSt5ratioILl1ELl1000EEEEEEElNS2_7rcv_bufEE_clES17_S1G_lS1H_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1A_EEEEDaS1J_EUlvE_EEDaRNS_4gateEOS1J_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/gate.hh:126
 (inlined by) _ZN7seastar13try_with_gateIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES19_EEEEDaS1I_EUlvE_EEDaRNS_4gateEOS1I_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/gate.hh:166
 (inlined by) _ZZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS0_12no_wait_typeEEEERKNS0_11client_infoENS0_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISF_EESF_jmNS0_8optionalISt8optionalIN7tracing10trace_infoEEEEEES7_JSC_SH_SF_jmSN_ENS0_19do_want_client_infoENS0_18do_want_time_pointEEEDaNS0_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS0_6server10connectionEEESJ_INSt6chrono10time_pointINS_12lowres_clockENS16_8durationIlSt5ratioILl1ELl1000EEEEEEElNS0_7rcv_bufEE_clES15_S1E_lS1F_ENUlT_E1_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES18_EEEEDaS1H_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:582
_ZSt13__invoke_implIvZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_JNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEES1I_St14__invoke_otherS10_DpOT1_ at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) _ZSt8__invokeIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_JNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEENSt15__invoke_resultIS1I_JDpT0_EE4typeEOS1I_DpOS1O_ at /usr/include/c++/10/bits/invoke.h:95
 (inlined by) _ZSt12__apply_implIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_St5tupleIJNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEEJLm0EEEDcOS1I_S10_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1723
 (inlined by) _ZSt5applyIZZN7seastar3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_6futureIJNS1_12no_wait_typeEEEERKNS1_11client_infoENS1_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISG_EESG_jmNS1_8optionalISt8optionalIN7tracing10trace_infoEEEEEES8_JSD_SI_SG_jmSO_ENS1_19do_want_client_infoENS1_18do_want_time_pointEEEDaNS1_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS0_10shared_ptrINS1_6server10connectionEEESK_INSt6chrono10time_pointINS0_12lowres_clockENS17_8durationIlSt5ratioILl1ELl1000EEEEEEElNS1_7rcv_bufEE_clES16_S1F_lS1G_EUlT_E1_St5tupleIJNS0_15semaphore_unitsINS0_35semaphore_default_exception_factoryES19_EEEEEDcOS1I_S10_ at /usr/include/c++/10/tuple:1734
 (inlined by) _ZN7seastar8futurizeIvE5applyIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS_6futureIJNS3_12no_wait_typeEEEERKNS3_11client_infoENS3_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISI_EESI_jmNS3_8optionalISt8optionalIN7tracing10trace_infoEEEEEESA_JSF_SK_SI_jmSQ_ENS3_19do_want_client_infoENS3_18do_want_time_pointEEEDaNS3_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS3_6server10connectionEEESM_INSt6chrono10time_pointINS_12lowres_clockENS19_8durationIlSt5ratioILl1ELl1000EEEEEEElNS3_7rcv_bufEE_clES18_S1H_lS1I_EUlT_E1_JNS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1B_EEEEENS8_IJEEEOS1K_OSt5tupleIJDpT0_EE at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1976
 (inlined by) _ZN7seastar6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEE9then_implIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_IJNS7_12no_wait_typeEEEERKNS7_11client_infoENS7_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISL_EESL_jmNS7_8optionalISt8optionalIN7tracing10trace_infoEEEEEESD_JSI_SN_SL_jmST_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESP_INSt6chrono10time_pointIS3_NS1C_8durationIlSt5ratioILl1ELl1000EEEEEEElNS7_7rcv_bufEE_clES1B_S1J_lS1K_EUlT_E1_NS0_IJEEEEES14_OS1M_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1544
 (inlined by) _ZN7seastar8internal14call_then_implINS_6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEEEE3runIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS2_IJNSA_12no_wait_typeEEEERKNSA_11client_infoENSA_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISO_EESO_jmNSA_8optionalISt8optionalIN7tracing10trace_infoEEEEEESG_JSL_SQ_SO_jmSW_ENSA_19do_want_client_infoENSA_18do_want_time_pointEEEDaNSA_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINSA_6server10connectionEEESS_INSt6chrono10time_pointIS5_NS1F_8durationIlSt5ratioILl1ELl1000EEEEEEElNSA_7rcv_bufEE_clES1E_S1M_lS1N_EUlT_E1_EEDaRS7_OS1P_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1198
 (inlined by) _ZN7seastar6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENS_12lowres_clockEEEEE4thenIZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS0_IJNS7_12no_wait_typeEEEERKNS7_11client_infoENS7_14opt_time_pointE15frozen_mutationSt6vectorIN3gms12inet_addressESaISL_EESL_jmNS7_8optionalISt8optionalIN7tracing10trace_infoEEEEEESD_JSI_SN_SL_jmST_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESP_INSt6chrono10time_pointIS3_NS1C_8durationIlSt5ratioILl1ELl1000EEEEEEElNS7_7rcv_bufEE_clES1B_S1J_lS1K_EUlT_E1_NS0_IJEEEEES14_OS1M_ at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1468
 (inlined by) auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/rpc/rpc_impl.hh:580
seastar::future<> std::__invoke_impl<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>(std::__invoke_other, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::shared_ptr<seastar::rpc::server::connection>&&, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >&&, long&&, seastar::rpc::rcv_buf&&) at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) std::enable_if<is_invocable_r_v<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>, std::enable_if>::type std::__invoke_r<seastar::future<>, auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&, seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf>(std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, (seastar::future<>&&)...) at /usr/include/c++/10/bits/invoke.h:115
 (inlined by) std::_Function_handler<seastar::future<> (seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf), auto seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}>::_M_invoke(std::_Any_data const&, seastar::shared_ptr<seastar::rpc::server::connection>&&, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >&&, long&&, seastar::rpc::rcv_buf&&) at /usr/include/c++/10/bits/std_function.h:292
std::function<seastar::future<> (seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)>::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf) const at /usr/include/c++/10/ext/new_allocator.h:106
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/rpc/rpc.cc:935
 (inlined by) __invoke_impl<seastar::future<>, seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1724
 (inlined by) apply<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1736
 (inlined by) apply<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>&> at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2009
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future-util.hh:86
 (inlined by) invoke<seastar::with_scheduling_group<seastar::rpc::server::connection::process()::<lambda()> mutable::<lambda()>::<lambda()> mutable::<lambda(seastar::rpc::request_frame::header_and_buffer_type)>::<lambda()>, {}>::<lambda()>&> at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) run_and_dispose at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/make_task.hh:40
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:2712
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Finally scylla was initialized:

2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip settled after 1151 extra polls; proceeding
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature CDC is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature COMPUTED_COLUMNS is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature CORRECT_COUNTER_ORDER is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature CORRECT_NON_COMPOUND_RANGE_TOMBSTONES is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature CORRECT_STATIC_COMPACT_IN_MC is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature COUNTERS is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature DIGEST_INSENSITIVE_TO_EXPIRY is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature DIGEST_MULTIPARTITION_READ is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature HINTED_HANDOFF_SEPARATE_CONNECTION is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature INDEXES is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature LARGE_PARTITIONS is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature LWT is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature MATERIALIZED_VIEWS is enabled
2020-08-12T12:58:30+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] features - Feature MC_SSTABLE_FORMAT is enabled

2020-08-12T12:58:40+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 1 polls.
2020-08-12T12:58:41+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 2 polls.
2020-08-12T12:58:42+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 3 polls.
2020-08-12T12:58:43+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 4 polls.
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32299]: Accepted publickey for centos from 10.0.7.29 port 41968 ssh2: RSA SHA256:g7OmDbWC8qLUZHxjMohQe3LsQHfcOJbG1ZHqodB6smc
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: New session 882 of user centos.
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd: Started Session 882 of user centos.
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32299]: pam_unix(sshd:session): session opened for user centos by (uid=0)
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32299]: pam_unix(sshd:session): session closed for user centos
2020-08-12T12:58:44+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: Removed session 882.
2020-08-12T12:58:45+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 6 polls.
2020-08-12T12:58:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/coredumpctl --no-pager --no-legend
2020-08-12T12:58:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-08-12T12:58:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-08-12T12:58:46+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip not settled after 7 polls.
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] gossip - Gossip settled after 7 extra polls; proceeding
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - allow replaying hints
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - Launching generate_mv_updates for non system tables
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - starting the view builder
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32325]: Accepted publickey for centos from 10.0.7.29 port 41982 ssh2: RSA SHA256:g7OmDbWC8qLUZHxjMohQe3LsQHfcOJbG1ZHqodB6smc
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 1] compaction - Compacting [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-25-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-13-big-Data.db:level=0, ]
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 1] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-37-big-Data.db:level=0, ]. 10kB to 5kB (~53% of original) in 10ms = 555kB/s. ~256 total partitions merged to 1.
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: New session 883 of user centos.
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd: Started Session 883 of user centos.
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32325]: pam_unix(sshd:session): session opened for user centos by (uid=0)
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - starting native transport
2020-08-12T12:58:50+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] cql_server_controller - Starting listening for CQL clients on 10.0.2.158:9042 (unencrypted)
2020-08-12T12:58:51+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | sshd[32325]: pam_unix(sshd:session): session closed for user centos
2020-08-12T12:58:51+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | systemd-logind: Removed session 883.
2020-08-12T12:58:52+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] thrift_controller - Thrift server listening on 10.0.2.158:9160 ...
2020-08-12T12:58:52+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - serving
2020-08-12T12:58:52+00:00  gemini-8h-large-num-columns-master-db-node-6247fd1a-3 !INFO    | scylla: [shard 0] init - Scylla version 4.2.rc2-0.20200811.b052f3f5cea initialization completed.

Gemini generate next schema:

chema: {
    "keyspace": {
        "name": "ks1",
        "replication": {
            "class": "SimpleStrategy",
            "replication_factor": "3"
        },
        "oracle_replication": {
            "class": "SimpleStrategy",
            "replication_factor": 1
        }
    },
    "tables": [
        {
            "name": "table1",
            "partition_keys": [
                {
                    "name": "pk0",
                    "type": "int"
                },
                {
                    "name": "pk1",
                    "type": "int"
                },
                {
                    "name": "pk2",
                    "type": "smallint"
                },
                {
                    "name": "pk3",
                    "type": "smallint"
                },
                {
                    "name": "pk4",
                    "type": "smallint"
                },
                {
                    "name": "pk5",
                    "type": "int"
                },
                {
                    "name": "pk6",
                    "type": "tinyint"
                },
                {
                    "name": "pk7",
                    "type": "tinyint"
                },
                {
                    "name": "pk8",
                    "type": "int"
                }
            ],
            "clustering_keys": [
                {
                    "name": "ck0",
                    "type": "timeuuid"
                },
                {
                    "name": "ck1",
                    "type": "text"
                },
                {
                    "name": "ck2",
                    "type": "float"
                },
                {
                    "name": "ck3",
                    "type": "timeuuid"
                },
                {
                    "name": "ck4",
                    "type": "uuid"
                },
                {
                    "name": "ck5",
                    "type": "int"
                },
                {
                    "name": "ck6",
                    "type": "varint"
                },
                {
                    "name": "ck7",
                    "type": "tinyint"
                },
                {
                    "name": "ck8",
                    "type": "double"
                },
                {
                    "name": "ck9",
                    "type": "blob"
                },
                {
                    "name": "ck10",
                    "type": "bigint"
                },
                {
                    "name": "ck11",
                    "type": "smallint"
                },
                {
                    "name": "ck12",
                    "type": "float"
                },
                {
                    "name": "ck13",
                    "type": "tinyint"
                },
                {
                    "name": "ck14",
                    "type": "bigint"
                },
                {
                    "name": "ck15",
                    "type": "timeuuid"
                }
            ],
            "columns": [
                {
                    "name": "col0",
                    "type": "tinyint"
                },
                {
                    "name": "col1",
                    "type": "int"
                },
                {
                    "name": "col2",
                    "type": "inet"
                },
                {
                    "name": "col3",
                    "type": "tinyint"
                },
                {
                    "name": "col4",
                    "type": "uuid"
                },
                {
                    "name": "col5",
                    "type": "float"
                },
                {
                    "name": "col6",
                    "type": "date"
                },
                {
                    "name": "col7",
                    "type": "smallint"
                },
                {
                    "name": "col8",
                    "type": "float"
                },
                {
                    "name": "col9",
                    "type": "bigint"
                },
                {
                    "name": "col10",
                    "type": "decimal"
                },
                {
                    "name": "col11",
                    "type": "uuid"
                },
                {
                    "name": "col12",
                    "type": "int"
                },
                {
                    "name": "col13",
                    "type": "boolean"
                },
                {
                    "name": "col14",
                    "type": "ascii"
                },
                {
                    "name": "col15",
                    "type": {
                        "kind": "list",
                        "type": "inet",
                        "frozen": true
                    }
                },
                {
                    "name": "col16",
                    "type": "timeuuid"
                },
                {
                    "name": "col17",
                    "type": "ascii"
                },
                {
                    "name": "col18",
                    "type": "inet"
                },
                {
                    "name": "col19",
                    "type": "varchar"
                },
                {
                    "name": "col20",
                    "type": "boolean"
                },
                {
                    "name": "col21",
                    "type": "varchar"
                },
                {
                    "name": "col22",
                    "type": "varint"
                },
                {
                    "name": "col23",
                    "type": {
                        "types": [
                            "bigint",
                            "duration",
                            "bigint",
                            "tinyint",
                            "uuid",
                            "decimal",
                            "smallint",
                            "varchar",
                            "varint",
                            "ascii",
                            "ascii"
                        ],
                        "frozen": false
                    }
                },
                {
                    "name": "col24",
                    "type": "boolean"
                },
                {
                    "name": "col25",
                    "type": "date"
                },
                {
                    "name": "col26",
                    "type": {
                        "types": [
                            "timestamp",
                            "ascii",
                            "varint",
                            "text",
                            "double",
                            "decimal",
                            "timeuuid",
                            "date",
                            "decimal",
                            "ascii",
                            "tinyint",
                            "timeuuid",
                            "int"
                        ],
                        "frozen": true
                    }
                },
                {
                    "name": "col27",
                    "type": "bigint"
                },
                {
                    "name": "col28",
                    "type": "blob"
                },
                {
                    "name": "col29",
                    "type": "boolean"
                },
                {
                    "name": "col30",
                    "type": "tinyint"
                },
                {
                    "name": "col31",
                    "type": {
                        "kind": "list",
                        "type": "double",
                        "frozen": true
                    }
                },
                {
                    "name": "col32",
                    "type": "uuid"
                },
                {
                    "name": "col33",
                    "type": "varchar"
                },
                {
                    "name": "col34",
                    "type": "inet"
                },
                {
                    "name": "col35",
                    "type": "blob"
                },
                {
                    "name": "col36",
                    "type": "text"
                },
                {
                    "name": "col37",
                    "type": "date"
                },
                {
                    "name": "col38",
                    "type": "float"
                },
                {
                    "name": "col39",
                    "type": "timeuuid"
                },
                {
                    "name": "col40",
                    "type": "tinyint"
                },
                {
                    "name": "col41",
                    "type": "duration"
                },
                {
                    "name": "col42",
                    "type": "boolean"
                },
                {
                    "name": "col43",
                    "type": "blob"
                },
                {
                    "name": "col44",
                    "type": "tinyint"
                },
                {
                    "name": "col45",
                    "type": "varchar"
                },
                {
                    "name": "col46",
                    "type": "varchar"
                },
                {
                    "name": "col47",
                    "type": "float"
                },
                {
                    "name": "col48",
                    "type": "date"
                },
                {
                    "name": "col49",
                    "type": "text"
                },
                {
                    "name": "col50",
                    "type": "bigint"
                },
                {
                    "name": "col51",
                    "type": {
                        "types": {
                            "udt_3118612057_0": "smallint",
                            "udt_3118612057_1": "decimal",
                            "udt_3118612057_2": "duration",
                            "udt_3118612057_3": "timeuuid"
                        },
                        "type_name": "udt_3118612057",
                        "frozen": true
                    }
                },
                {
                    "name": "col52",
                    "type": "text"
                },
                {
                    "name": "col53",
                    "type": "float"
                },
                {
                    "name": "col54",
                    "type": {
                        "kind": "list",
                        "type": "inet",
                        "frozen": true
                    }
                },
                {
                    "name": "col55",
                    "type": "inet"
                },
                {
                    "name": "col56",
                    "type": "float"
                },
                {
                    "name": "col57",
                    "type": "text"
                },
                {
                    "name": "col58",
                    "type": "boolean"
                },
                {
                    "name": "col59",
                    "type": "uuid"
                },
                {
                    "name": "col60",
                    "type": "decimal"
                },
                {
                    "name": "col61",
                    "type": "tinyint"
                },
                {
                    "name": "col62",
                    "type": "tinyint"
                },
                {
                    "name": "col63",
                    "type": "varchar"
                },
                {
                    "name": "col64",
                    "type": "int"
                },
                {
                    "name": "col65",
                    "type": "duration"
                },
                {
                    "name": "col66",
                    "type": "uuid"
                },
                {
                    "name": "col67",
                    "type": "boolean"
                },
                {
                    "name": "col68",
                    "type": "boolean"
                },
                {
                    "name": "col69",
                    "type": "timeuuid"
                },
                {
                    "name": "col70",
                    "type": "uuid"
                },
                {
                    "name": "col71",
                    "type": "bigint"
                },
                {
                    "name": "col72",
                    "type": "tinyint"
                },
                {
                    "name": "col73",
                    "type": {
                        "types": [
                            "varint",
                            "inet",
                            "varint",
                            "blob",
                            "double",
                            "text",
                            "duration",
                            "ascii",
                            "double",
                            "varchar",
                            "varchar",
                            "double",
                            "int",
                            "boolean"
                        ],
                        "frozen": false
                    }
                },
                {
                    "name": "col74",
                    "type": "duration"
                },
                {
                    "name": "col75",
                    "type": "float"
                },
                {
                    "name": "col76",
                    "type": "double"
                },
                {
                    "name": "col77",
                    "type": "decimal"
                },
                {
                    "name": "col78",
                    "type": "blob"
                },
                {
                    "name": "col79",
                    "type": "text"
                }
            ],
            "indexes": [
                {
                    "name": "table1_col0_idx",
                    "column": "col0",
                    "column_idx": 0
                },
                {
                    "name": "table1_col1_idx",
                    "column": "col1",
                    "column_idx": 1
                },
                {
                    "name": "table1_col3_idx",
                    "column": "col3",
                    "column_idx": 3
                },
                {
                    "name": "table1_col5_idx",
                    "column": "col5",
                    "column_idx": 5
                },
                {
                    "name": "table1_col7_idx",
                    "column": "col7",
                    "column_idx": 7
                },
                {
                    "name": "table1_col8_idx",
                    "column": "col8",
                    "column_idx": 8
                },
                {
                    "name": "table1_col10_idx",
                    "column": "col10",
                    "column_idx": 10
                },
                {
                    "name": "table1_col12_idx",
                    "column": "col12",
                    "column_idx": 12
                },
                {
                    "name": "table1_col22_idx",
                    "column": "col22",
                    "column_idx": 22
                },
                {
                    "name": "table1_col30_idx",
                    "column": "col30",
                    "column_idx": 30
                }
            ],
            "materialized_views": [
                {
                    "name": "table1_mv_0",
                    "partition_keys": [
                        {
                            "name": "col32",
                            "type": "uuid"
                        },
                        {
                            "name": "pk0",
                            "type": "int"
                        },
                        {
                            "name": "pk1",
                            "type": "int"
                        },
                        {
                            "name": "pk2",
                            "type": "smallint"
                        },
                        {
                            "name": "pk3",
                            "type": "smallint"
                        },
                        {
                            "name": "pk4",
                            "type": "smallint"
                        },
                        {
                            "name": "pk5",
                            "type": "int"
                        },
                        {
                            "name": "pk6",
                            "type": "tinyint"
                        },
                        {
                            "name": "pk7",
                            "type": "tinyint"
                        },
                        {
                            "name": "pk8",
                            "type": "int"
                        }
                    ],
                    "clustering_keys": [
                        {
                            "name": "ck0",
                            "type": "timeuuid"
                        },
                        {
                            "name": "ck1",
                            "type": "text"
                        },
                        {
                            "name": "ck2",
                            "type": "float"
                        },
                        {
                            "name": "ck3",
                            "type": "timeuuid"
                        },
                        {
                            "name": "ck4",
                            "type": "uuid"
                        },
                        {
                            "name": "ck5",
                            "type": "int"
                        },
                        {
                            "name": "ck6",
                            "type": "varint"
                        },
                        {
                            "name": "ck7",
                            "type": "tinyint"
                        },
                        {
                            "name": "ck8",
                            "type": "double"
                        },
                        {
                            "name": "ck9",
                            "type": "blob"
                        },
                        {
                            "name": "ck10",
                            "type": "bigint"
                        },
                        {
                            "name": "ck11",
                            "type": "smallint"
                        },
                        {
                            "name": "ck12",
                            "type": "float"
                        },
                        {
                            "name": "ck13",
                            "type": "tinyint"
                        },
                        {
                            "name": "ck14",
                            "type": "bigint"
                        },
                        {
                            "name": "ck15",
                            "type": "timeuuid"
                        }
                    ]
                }
            ],
            "known_issues": {
                "https://github.com/scylladb/scylla/issues/3708": true
            }
        }
    ]
}
@avikivity
Copy link
Member

I don't see ami-04983b862895e0534 in Ireland. Which region is it in?

@avikivity
Copy link
Member

Also, what instance type was used?

@avikivity
Copy link
Member

Couldn't find an the exact AMI ID, but a similar one:

ami-0bf36261ff9c82df5

has Linux 5.8.1:

Linux ip-172-31-1-234.eu-west-1.compute.internal 5.8.1-1.el7.elrepo.x86_64 #1 SMP Tue Aug 11 12:01:11 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

This is consistent with other reports with problems in 5.8 kernels.

@aleksbykov
Copy link
Contributor Author

Sorry, forgot to mentioned that it on eu-north-1 region.

@aleksbykov
Copy link
Contributor Author

Instance type was : i3en.3xlarge

@roydahan
Copy link

@aleksbykov do you have monitor?

@aleksbykov
Copy link
Contributor Author

@roydahan instances were removed. but i think i can restore it

@avikivity
Copy link
Member

This is consistent with the other reports that mentioned i3en+5.8

@avikivity
Copy link
Member

No need to restore it.

@avikivity avikivity changed the title Huge reactor stalls up to 14400ms and gossip not settled Gossip not settled after 1151 polls Performance regression on I3en with Linux 5.8 Aug 13, 2020
@avikivity
Copy link
Member

Linux kernel version from the reported AMI ID: 5.8.1-1.el7.elrepo.x86_64

@avikivity
Copy link
Member

@roydahan

  • please check on I3en.metal
  • please provide a simple reproducer (launch single-node cluster, launch loader, run command) that is suitable for someone with no Scylla experience to run

slivne added a commit to slivne/scylla-machine-image that referenced this issue Aug 13, 2020
@slivne
Workaround for a performance regression in kernel-ml 5.8.0,5.8.1 in a…
b6c93ab 
…ws i3en

ref scylladb/scylladb#7036

Pin down kernel version used for aws image till the performance regression is fixed

Signed-off-by: Shlomi Livne <shlomi@scylladb.com>
@slivne slivne mentioned this issue Aug 13, 2020
Merged
bentsi pushed a commit to scylladb/scylla-machine-image that referenced this issue Aug 13, 2020
@slivne
Workaround for a performance regression in kernel-ml 5.8.0,5.8.1 in a…
7084433 
…ws i3en

ref scylladb/scylladb#7036

Pin down kernel version used for aws image till the performance regression is fixed

Signed-off-by: Shlomi Livne <shlomi@scylladb.com>
@slivne
Copy link
Contributor

slivne commented Aug 13, 2020 •
edited
Loading

Please note this bug doesn't reproduce every time - for us its 1 out of 4 runs (so you may need to retry provisioning a scylla node till you hit the bad case)

Simple reproducer

  • Run ScyllaDB 4.1.4 AMI on i3en.3xl instance - this is the scylla node
  • Run ScyllaDB 4.1.4 AMI on c4.4xl instance - this is the loader node
    ((in US-EAST 1 - the AMI ID is ami-06f7b1e45dd4b54eb)

On the loader node run the following command

cassandra-stress write no-warmup n=100000000 -node <scylla-node-internal-ip>

If the performance is good you should see

.
.
Running WRITE with 200 threads for 100000000 iteration
Failed to connect over JMX; not collecting these stats
type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
total,         20164,   20164,   20164,   20164,     5.6,     2.2,    23.2,    51.7,    67.3,    72.7,    1.0,  0.00000,      0,      0,       0,       0,       0,       0
total,        131488,  111324,  111324,  111324,     1.8,     1.1,     5.2,    13.8,    26.2,    30.5,    2.0,  0.48192,      0,      0,       0,       0,       0,       0
total,        334791,  203303,  203303,  203303,     1.0,     0.6,     2.8,     5.3,    21.6,    28.0,    3.0,  0.38330,      0,      0,       0,       0,       0,       0
total,        564497,  229706,  229706,  229706,     0.9,     0.5,     2.4,     4.8,    14.7,    18.4,    4.0,  0.28936,      0,      0,       0,       0,       0,       0
total,        800481,  235984,  235984,  235984,     0.8,     0.5,     2.4,     4.7,    13.8,    29.8,    5.0,  0.23014,      0,      0,       0,       0,       0,       0
.
.

The second column is operation per second - 200K is ok (the loader is capped out)

In a bad case you will get something like

.
.
leeping 2s...
Running WRITE with 200 threads for 100000000 iteration
Failed to connect over JMX; not collecting these stats
type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
total,          1663,    1663,    1663,    1663,    60.1,    23.9,   280.0,   545.3,   552.1,   561.5,    1.0,  0.00000,      0,      0,       0,       0,       0,       0
total,          3220,    1557,    1557,    1557,   128.8,    50.3,   541.6,   718.8,   721.9,   721.9,    2.0,  0.13604,      0,      0,       0,       0,       0,       0
total,          4309,    1089,    1089,    1089,   122.7,    50.0,   370.9,   839.9,  1110.4,  1197.5,    3.0,  2.17520,      0,      0,       0,       0,       0,       0
total,          5945,    1636,    1636,    1636,   173.4,    50.2,   747.6,  1130.4,  1220.5,  1220.5,    4.0, -3.19603,      0,      0,       0,       0,       0,       0
total,          6810,     865,     865,     865,   136.5,    89.8,   392.2,   741.3,   840.4,   841.0,    5.0,  6.66430,      0,      0,       0,       0,       0,       0
total,          8271,    1461,    1461,    1461,   157.8,    58.3,   591.4,  1081.1,  1180.7,  1181.7,    6.0, -2.41055,      0,      0,       0,       0,       0,       0
total,          9473,    1202,    1202,    1202,   180.5,    52.1,   720.9,   958.9,  1049.6,  1050.7,    7.0, -0.86077,      0,      0,       0,       0,       0,       0
.
.
.

So in this run the operations per second is less than 2K.

In the bad case doing top on the scylla node shows

B Swap:        0 total,        0 free,        0 used. 94258016 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                   
 1815 scylla    20   0   16.0t   2.2g  59288 R  1108  2.4 262:01.71 scylla                                                                                                                                                                    
   56 root      20   0       0      0      0 R  16.6  0.0   2:58.08 ksoftirqd/9                                                                                                                                                               
   36 root      20   0       0      0      0 S  15.6  0.0   2:02.33 ksoftirqd/5                                                                                                                                                               
   41 root      20   0       0      0      0 S  11.0  0.0   7:04.67 ksoftirqd/6                                                                                                                                                               
   46 root      20   0       0      0      0 S  10.3  0.0   4:07.70 ksoftirqd/7                                                                                                                                                               
   51 root      20   0       0      0      0 S   5.6  0.0   6:48.10 ksoftirqd/8                                                                                                                                                               
 1891 scylla    20   0 2872096  83152  17236 S   3.7  0.1   0:14.84 scylla-jmx                                                                                                                                                                
 1428 root      20   0       0      0      0 I   2.0  0.0   0:04.50 kworker/5:3-mm_                                                                                                                                                           
 2368 root      20   0       0      0      0 I   1.7  0.0   0:06.04 kworker/9:1-dio                                                                                                                                                           
 2370 root      20   0       0      0      0 I   1.7  0.0   0:10.10 kworker/6:2-mm_                                                                                                                                                           
   10 root      20   0       0      0      0 I   0.3  0.0   0:11.17 rcu_sched                                                                                                                                                                 
 2438 centos    20   0  162172   4516   3748 R   0.3  0.0   0:00.15 top

bentsi pushed a commit to scylladb/scylla-machine-image that referenced this issue Aug 13, 2020
@slivne
Workaround for a performance regression in kernel-ml 5.8.0,5.8.1 in a…
ac229fe 
…ws i3en

ref scylladb/scylladb#7036

Pin down kernel version used for aws image till the performance regression is fixed

Signed-off-by: Shlomi Livne <shlomi@scylladb.com>
(cherry picked from commit 7084433)
bentsi pushed a commit to scylladb/scylla-machine-image that referenced this issue Aug 13, 2020
@slivne
Workaround for a performance regression in kernel-ml 5.8.0,5.8.1 in a…
e22898a 
…ws i3en

ref scylladb/scylladb#7036

Pin down kernel version used for aws image till the performance regression is fixed

Signed-off-by: Shlomi Livne <shlomi@scylladb.com>
(cherry picked from commit 7084433)
@roydahan
Copy link

Based on what @amoskong reported on the enterprise issue, in order to reproduce we need a cluster with at least one node that has the “bad” kernel.

5.8.1-1.el7.elrepo.x86_64: Good
5.8.0-1.el7.elrepo.x86_64: Bad
5.7.12-1.el7.elrepo.x86_64: Good
5.7.6-1.el7.elrepo.x86_64: Good

According to @amoskong he couldn’t find which commit in 5.8.1 fixes the issue, but the issue is fixed.

@avikivity
Copy link
Member

There is nothing relevant in 5.8.1, please try again.

@amoskong
Copy link
Contributor

amoskong commented Aug 17, 2020 •
edited
Loading

There is nothing relevant in 5.8.1, please try again.

I do reproduce the issue with 5.8.1. Before retest, I cleaned the data of cluster.

Reproduce scenario:

  • update kernel to 5.8.0 / 5.8.1
  • create a cluster (2 db nodes)
  • clean the cluster data
  • wait two nodes are all UP
  • run c-s workload from db node 1
    • cassandra-stress write no-warmup duration=20s -rate threads=200 -schema 'replication(factor=2)'

@amoskong
Copy link
Contributor

Hi @avikivity @roydahan

I didn't reproduce the problem with kernel 5.7.0 , there are 17595 commits from 5.7 to 5.8
We need build kernel and test 14 times to find the introduce commit. Do we want to find the commit?

[linux]$ git log v5.7..v5.8|grep ^commit|wc -l
17595
>>> 17595 / (2 ** 14)
1

@amoskong
Copy link
Contributor

I only successfully tested 5.8-rc4, the issue can be reproduced.

The built rpm can't be installed to centos7, and no err message from rpm and yum. I have to install the kernel and modules by make install. It's easy to make the instance to be dead after restart.

I can saw an error ofefi/libstub in build 5.8-rc1/5.8-rc2/5.8-rc3, so I disalbed CONFIG_EFI_STUB=n to workaround the build error. But the instance can't boot up after restart.

[centos@ip-10-99-0-233 linux]$ make bzImage -j 1
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  DESCEND  objtool
  CHK     include/generated/compile.h
  CC      drivers/firmware/efi/libstub/efi-stub-helper.o
In file included from drivers/firmware/efi/libstub/efi-stub-helper.c:16:0:
drivers/firmware/efi/libstub/efi-stub-helper.c: In function ‘efi_char16_puts’:
./arch/x86/include/asm/efi.h:355:3: sorry, unimplemented: ms_abi attribute requires -maccumulate-outgoing-args or subtarget optimization implying it
   : __efi64_thunk_map(inst, func, inst, ##__VA_ARGS__))
   ^
drivers/firmware/efi/libstub/efi-stub-helper.c:37:2: note: in expansion of macro  efi_call_proto’
  efi_call_proto(efi_table_attr(efi_system_table, con_out),
  ^
drivers/firmware/efi/libstub/efi-stub-helper.c:37: confused by earlier errors, bailing out

@avikivity
Copy link
Member

This commit should fix it: 5435f73d5c4a1b7504356876e69ba52de83f4975

@avikivity
Copy link
Member

Most likely, the problem was introduced in -rc1 (where most of the patches are).

@avikivity
Copy link
Member

I think we should bisect it, it's the only sure way of arriving at the root cause.

@RobertKettler
Copy link

I have reproduced the issue on 5.8.0 and 5.8.1 . Sometimes it requires several (5-8) restarts of the database in order to reproduce the issue. Usually the DB works fine after a machine reboot.

My bisection is pointing in the direction of the commit: 633260fa143bbed05e65dc557a492667dfdc45bb
But I need to verify this before taking any further action.
Just wanted to let you know so you don't spent to much time repeating my steps.
Any help confirming this is much appreciated ;)

@avikivity
Copy link
Member

Wow, I would have assumed that patch would either work perfectly or crash very early after boot, since it's touching such a senstive area. Any idea how it's broken?

Thanks for the update.

@amoskong
Copy link
Contributor

I have reproduced the issue on 5.8.0 and 5.8.1 . Sometimes it requires several (5-8) restarts of the database in order to reproduce the issue. Usually the DB works fine after a machine reboot.

My bisection is pointing in the direction of the commit: 633260fa143bbed05e65dc557a492667dfdc45bb

I didn't reproduce the problem with above commit.

633260fa143bbed05e65dc557a492667dfdc45b~1 + [PATCH] efi/x86: Fix build with gcc 4

  • Good

633260fa143bbed05e65dc557a492667dfdc45b + [PATCH] efi/x86: Fix build with gcc 4

  • Good

But I need to verify this before taking any further action.
Just wanted to let you know so you don't spent to much time repeating my steps.
Any help confirming this is much appreciated ;)

What's your test scnario?

@RobertKettler
Copy link

I didn't reproduce the problem with above commit.
633260fa143bbed05e65dc557a492667dfdc45b~1 + [PATCH] efi/x86: Fix build with gcc 4

Have you restarted the db several times?

What's your test scnario?

I'm using the ami provided running and on an second node:
cassandra-stress write no-warmup n=100000000 -node

Let me get the build info for you.

@RobertKettler
Copy link

Wow, I would have assumed that patch would either work perfectly or crash very early after boot, since it's touching such a senstive area. Any idea how it's broken?

Thanks for the update.

The bisection there is a bit messy since multiple commits got introduced with a branch merge and several ended up in kernel panics.

Currently I'm trying to run 633260fa143bbed05e65dc557a492667dfdc45b~1 and see if that is good

@amoskong
Copy link
Contributor

amoskong commented Aug 18, 2020 •
edited
Loading

Hi Robert @avikivity ,

In my latest test, I do verified that 633260fa143bbed05e65dc557a492667dfdc45bb introduced the problem.

  • [PATCH] x86/irq: Convey vector as argument and not in ptregs

I didn't reproduce the problem with above commit.
633260fa143bbed05e65dc557a492667dfdc45b~1 + [PATCH] efi/x86: Fix build with gcc 4

Have you restarted the db several times?

Yes. But there are other factors affects the result.

I just found if cassandra-stress workload is executed too early before the second node is UP, the issue can't be reproduced.
So need to make sure two nodes are all UP, then execute c-s workload.

The upstream tree I used to bisect: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Latest result:

633260fa143bbed05e65dc557a492667dfdc45b~1 + [PATCH] efi/x86: Fix build with gcc 4

  • 79b9c183021ef3f5ca2d5168cd3fd442580eca09 + efi fix
  • Good (I restarted the scylla-server of two nodes 10 times)

633260fa143bbed05e65dc557a492667dfdc45b + [PATCH] efi/x86: Fix build with gcc 4

  • Bad

What's your test scnario?

I'm using the ami provided running and on an second node:
cassandra-stress write no-warmup n=100000000 -node

Let me get the build info for you.

@RobertKettler
Copy link

In my latest test, I do verified that 633260fa143bbed05e65dc557a492667dfdc45bb introduced the problem.
* [PATCH] x86/irq: Convey vector as argument and not in ptregs

Thanks for verifying @amoskong ! I'll look into that commit and see what going on.

@avikivity
Copy link
Member

@RobertKettler it's worthwhile to report to lkml, please copy @amoskong and me if you do.

@avikivity
Copy link
Member

Pretty amazing failure.

@amoskong
Copy link
Contributor

Alexander Graf mentioned this issue in LKML thread

@avikivity
Copy link
Member

Patch: https://lore.kernel.org/lkml/20200826115357.3049-1-graf@amazon.com/

@avikivity
Copy link
Member

Patch queued: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=e027fffff799cdd70400c5485b1a54f482255985

@avikivity
Copy link
Member

Queued for 5.9.0, 5.8.6

@avikivity
Copy link
Member

5.8.6 was released. @slivne please undo the kernel pinning.

@slivne
Copy link
Contributor

slivne commented Sep 13, 2020

@penberg - reassigning this to you guys - to remove the changes.

@slivne slivne added this to the 4.2 milestone Sep 13, 2020
@slivne slivne added the cloud/aws AWS related issues label Sep 13, 2020
@avikivity
Copy link
Member

@penberg ping

1 similar comment
@avikivity
Copy link
Member

@penberg ping

@penberg
Copy link
Contributor

penberg commented Oct 27, 2020

@avikivity We never pinned the kernel for open source releases. I queued reverts for enterprise, though.

@avikivity
Copy link
Member

Yes we did, example: scylladb/scylla-machine-image@e22898a

@penberg
Copy link
Contributor

penberg commented Oct 28, 2020

@avikivity Thanks, not sure why I missed them. Reverted the pinning from 4.0, 4.1, and 4.2 machine-image.

@avikivity
Copy link
Member

@penberg what about 4.3? and master?

@penberg
Copy link
Contributor

penberg commented Oct 28, 2020

@avikivity I can't find that pinning in scylla-machine-image.git's master branch...

@penberg
Copy link
Contributor

penberg commented Oct 28, 2020

(and therefore, obviously not in branch-4.3 either)

@dyasny
Copy link
Contributor

dyasny commented Nov 23, 2020

Is this issue resolved? Is it safe to use ElRepo's ML 5.9.10 kernels?
Today's latest up there is kernel-ml-5.9.10-1.el8.elrepo.x86_64.rpm

@dyasny
Copy link
Contributor

dyasny commented Nov 23, 2020

There's also a centos-plus kernel that is 4.18 but seems to have far more backports than the stock kernel:
https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus#CentOSPlus_kernel_differences

@slivne slivne modified the milestones: 4.2, 4.3 Nov 26, 2020
@slivne
Copy link
Contributor

slivne commented Dec 21, 2020

This issue is closed -

for reference the issue was 5.8.X and fixed in 5.8.6

the patch was https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=e027fffff799cdd70400c5485b1a54f482255985

We reverted the pinning of kernel (we did not add a restriction for not using the bad kernels)

@slivne slivne closed this as completed Dec 21, 2020
@syuu1228 syuu1228 mentioned this issue Aug 28, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@penberg penberg

Labels
cloud/aws AWS related issues
Projects
None yet
Milestone
4.3
Development

No branches or pull requests
8 participants
@penberg @amoskong @dyasny @avikivity @slivne @RobertKettler @roydahan @aleksbykov