Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coredump after enospc error #4877

Closed
ShlomiBalalis opened this issue Aug 21, 2019 · 23 comments
Closed

Coredump after enospc error #4877

ShlomiBalalis opened this issue Aug 21, 2019 · 23 comments
Assignees
Labels
Milestone

Comments

@ShlomiBalalis
Copy link

@ShlomiBalalis ShlomiBalalis commented Aug 21, 2019

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

  • I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash):3.1.0.rc3-0.20190816.d06bcef3b
Cluster size:4
OS (RHEL/CentOS/Ubuntu/AWS AMI):ami-07f2007dc543eced5

During the nemesis nodetool_enospc, several backtraces occured, which ended with a coredump:


2019-08-19T20:30:11+00:00  ip-172-30-0-194 !WARNING \| scylla.bin: [shard 6] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace:   0x451d4ab#012  0x406c354#012  0x2307d14#012  0x23088c6#012  0x406aea1#012  0x406b09e#012  0x414d0ed#012  0x417772b#012  0x403915d#012  /opt/scylladb/bin/../libreloc/libpthread.so.0+0x858d#012  /opt/scylladb/bin/../libreloc/libc.so.6+0xfd6a2
--

backtrace<seastar::current_backtrace()::<lambda(seastar::frame)> > at /usr/include/fmt/format.h:396
 (inlined by) seastar::current_backtrace() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/util/backtrace.cc:84
seastar::report_failed_future(std::__exception_ptr::exception_ptr) at /usr/include/boost/program_options/variables_map.hpp:146
seastar::promise<>::~promise() at /usr/include/c++/8/ext/new_allocator.h:102
 (inlined by) std::_Optional_payload<seastar::promise<>, false, false, false>::~_Optional_payload() at /usr/include/c++/8/optional:200
 (inlined by) std::_Optional_base<seastar::promise<>, false, false>::~_Optional_base() at /usr/include/c++/8/optional:661
 (inlined by) std::optional<seastar::promise<> >::~optional() at /usr/include/c++/8/optional:943
 (inlined by) seastar::expiring_fifo<seastar::promise<>, seastar::promise_expiry<>, seastar::lowres_clock>::entry::~entry() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/expiring_fifo.hh:60
 (inlined by) seastar::chunked_fifo<seastar::expiring_fifo<seastar::promise<>, seastar::promise_expiry<>, seastar::lowres_clock>::entry, 128ul>::pop_front() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/chunked_fifo.hh:435
 (inlined by) seastar::expiring_fifo<seastar::promise<>, seastar::promise_expiry<>, seastar::lowres_clock>::pop_front() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/expiring_fifo.hh:165
 (inlined by) seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_state::resolve(seastar::future<>&&) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/shared_future.hh:120
 (inlined by) seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_state::resolve(seastar::future<>&&) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/shared_future.hh:115
seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_future(seastar::future<>&&)::{lambda(seastar::future<>&&)#1}::operator()(seastar::future<>&&) at /usr/include/c++/8/ext/new_allocator.h:102
 (inlined by) seastar::future<> seastar::do_void_futurize_helper<void>::apply<seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_future(seastar::future<>&&)::{lambda(seastar::future<>&&)#1}, seastar::future<> >(seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_future(seastar::future<>&&)::{lambda(seastar::future<>&&)#1}&&, seastar::future<>&&) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:1336
 (inlined by) seastar::future<> seastar::futurize<void>::apply<seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_future(seastar::future<>&&)::{lambda(seastar::future<>&&)#1}, seastar::future<> >(seastar::shared_future<seastar::with_clock<seastar::lowres_clock> >::shared_future(seastar::future<>&&)::{lambda(seastar::future<>&&)#1}&&, seastar::future<>&&) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:1385
 (inlined by) _ZZZN7seastar6futureIJEE17then_wrapped_implIZNS_13shared_futureIJNS_10with_clockINS_12lowres_clockEEEEEC4EOS1_EUlS8_E_S1_EET0_OT_ENKUlvE_clEvENUlSC_E_clINS_12future_stateIJEEEEEDaSC_ at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:1004
 (inlined by) _ZN7seastar12continuationIZZNS_6futureIJEE17then_wrapped_implIZNS_13shared_futureIJNS_10with_clockINS_12lowres_clockEEEEEC4EOS2_EUlS9_E_S2_EET0_OT_ENKUlvE_clEvEUlSD_E_JEE15run_and_disposeEv at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:377
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/boost/program_options/variables_map.hpp:146
seastar::reactor::run_some_tasks() at /usr/include/boost/program_options/variables_map.hpp:146
seastar::reactor::run_some_tasks() at /usr/include/boost/program_options/variables_map.hpp:146
 (inlined by) seastar::reactor::run() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/reactor.cc:4243
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/boost/program_options/variables_map.hpp:146
std::function<void ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::posix_thread::start_routine(void*) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/posix.cc:52
2019-08-19T20:30:11+00:00  ip-172-30-0-194 !WARNING | scylla.bin: [shard 11] large_data - Failed to add a record to system.large_partitions: ks = scylla_bench, table = test, sst = /var/lib/scylla/data/scylla_bench/test-0d861cd0c28611e9992f000000000008/mc-2375-big-Data.db exception = exceptions::mutation_write_failure_exception (Operation failed for system.large_partitions - received 0 responses and 1 failures from 1 CL=ONE.)

/libreloc/libpthread.so.0+0x858d#012  /opt/scylladb/bin/../libreloc/libc.so.6+0xfd6a2
2019-08-19T20:30:12+00:00  ip-172-30-0-194 !ERR     \| scylla.bin: [shard 11] system_keyspace - update compaction history failed: exceptions::mutation_write_failure_exception (Operation failed for system.compaction_history - received 0 responses and 1 failures from 1 CL=ONE.): ignored
2019-08-19T20:30:36+00:00  ip-172-30-0-194 !ERR     | scylla.bin: [shard 11] storage_proxy - exception during mutation write to 172.30.0.194: seastar::broken_promise (broken promise)

/libreloc/libpthread.so.0+0x858d#012  /opt/scylladb/bin/../libreloc/libc.so.6+0xfd6a2
2019-08-19T20:30:36+00:00  ip-172-30-0-194 !ERR     \| scylla.bin: [shard 11] system_keyspace - update compaction history failed: exceptions::mutation_write_failure_exception (Operation failed for system.compaction_history - received 0 responses and 1 failures from 1 CL=ONE.): ignored
(CoreDumpEvent Severity.CRITICAL): node=Node longevity-large-partitions-4d-disab-db-node-1633d10e-3 [52.87.214.209 | 172.30.0.194] (seed: False)
corefile_urls=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.12833d05b94c430a8762bc8e4df93173.9552.1566246112000000.gz/core.scylla.996.12833d05b94c430a8762bc8e4df93173.9552.1566246112000000.gz.aa
backtrace=           PID: 9552 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Mon 2019-08-19 20:21:52 UTC (1min 50s ago)
  Command Line: /usr/bin/scylla /opt/scylladb/bin/../libexec/scylla.bin --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-11
    Executable: /opt/scylladb/libreloc/ld.so
 Control Group: /
       Boot ID: 12833d05b94c430a8762bc8e4df93173
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: ip-172-30-0-194.ec2.internal
      Coredump: /var/lib/systemd/coredump/core.scylla.996.12833d05b94c430a8762bc8e4df93173.9552.1566246112000000
       Message: Process 9552 (scylla) of user 996 dumped core.
                
                Stack trace of thread 9566:
                #0  0x00007fb1933e653f raise (/opt/scylladb/libreloc/libc.so.6)
                #1  0x00007fb1933d095e abort (/opt/scylladb/libreloc/libc.so.6)
                #2  0x0000000004021a2b on_allocation_failure (/opt/scylladb/libexec/scylla.bin)

--
seastar::memory::on_allocation_failure(unsigned long) at memory.cc:?
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 21, 2019

We cannot decode this core due to #4673 , please reproduce with a newer version.

Also please don't split core dumps, it's annoying.

@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 21, 2019

Closing since the core is not debuggable, but please run the test again with a new version.

@avikivity avikivity closed this Aug 21, 2019
@ShlomiBalalis

This comment has been minimized.

Copy link
Author

@ShlomiBalalis ShlomiBalalis commented Aug 26, 2019

Using version 3.1.0.rc4-0.20190824.e4a39ed31, I recreated the issue, and the followed occurred:
During Enospc, a coredump occurred on another node (that did not experience Enospc) -

(CoreDumpEvent Severity.CRITICAL): node=Node longevity-large-partitions-Enospc-l-db-node-17ada1ae-1 [3.80.134.165 | 172.30.0.187] (seed: True)
corefile_urls=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.313079e31cad4f9f8c54d0255d201355.5087.1566767996000000.gz/core.scylla.996.313079e31cad4f9f8c54d0255d201355.5087.1566767996000000.gz.aa
backtrace=           PID: 5087 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Sun 2019-08-25 21:19:56 UTC (1min 56s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-11
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: 313079e31cad4f9f8c54d0255d201355
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: ip-172-30-0-187.ec2.internal
      Coredump: /var/lib/systemd/coredump/core.scylla.996.313079e31cad4f9f8c54d0255d201355.5087.1566767996000000
       Message: Process 5087 (scylla) of user 996 dumped core.
                
                Stack trace of thread 5097:
                #0  0x00007f9ec61a353f raise (libc.so.6)
                #1  0x00007f9ec618d95e abort (libc.so.6)
                #2  0x000000000402190b on_allocation_failure (scylla)

The coredump occurred several times after that for the next hour, and didn't occur again afterwards

@ShlomiBalalis ShlomiBalalis reopened this Aug 26, 2019
@tzach tzach added this to the 3.2 milestone Aug 26, 2019
@tzach

This comment has been minimized.

Copy link
Contributor

@tzach tzach commented Aug 26, 2019

Not a regression, moving to 3.2

@juliayakovlev

This comment has been minimized.

Copy link

@juliayakovlev juliayakovlev commented Aug 27, 2019

I got coredump also during Enospc in longevity-mv-si.
Scylla version: 3.1.0.rc4-0.20190824.e4a39ed31
There is no backtrace and no info for coredump :

           PID: 2448 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 8 (FPE)
     Timestamp: Mon 2019-08-26 10:08:55 UTC (22h ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: 7438aaaef202430ebfe7efe2760b750c
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: ip-10-0-240-3.eu-west-1.compute.internal
       Message: Process 2448 (scylla) of user 996 dumped core.

From journalctl in the time of coredump:

Aug 26 10:09:00 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 0] database - Keyspace system_traces: Reading CF node_slow_log id=bfcc4e62-5b63-3aa1-a1c3-6f5e47f3325c version=3e616c29-16d1-3939-92ea-8ca1d859561c
Aug 26 10:09:01 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 8] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-144115188075966399.log. 1380344 bytes skipped.
Aug 26 10:09:01 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 6] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-108086391057002433.log. 864248 bytes skipped.
Aug 26 10:09:01 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 13] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-234187180623376641.log. 31371256 bytes skipped.
Aug 26 10:09:01 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 9] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-162129586585448396.log. 32911352 bytes skipped.
Aug 26 10:09:01 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 1] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-18014398509592511.log. 5435384 bytes skipped.
Aug 26 10:09:02 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 12] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-216172782113894334.log. 13410296 bytes skipped.
Aug 26 10:09:10 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 0] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-110527.log. 16814072 bytes skipped.
Aug 26 10:09:15 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 7] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-126100789566484415.log. 31240184 bytes skipped.
Aug 26 10:09:17 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 3] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-54043195528556477.log. 29446136 bytes skipped.
Aug 26 10:09:21 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 11] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-198158383604412350.log. 19808248 bytes skipped.
Aug 26 10:09:22 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 4] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-72057594038038480.log. 19828728 bytes skipped.
Aug 26 10:09:26 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 5] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-90071992547520445.log. 14335992 bytes skipped.
Aug 26 10:09:26 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 10] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-180143985094930365.log. 24948728 bytes skipped.
Aug 26 10:09:35 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 2] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-36028797019074497.log. 25460728 bytes skipped.
Aug 26 10:10:06 ip-10-0-240-3.eu-west-1.compute.internal scylla[17172]:  [shard 0] commitlog_replayer - Log replay complete, 38397170 replayed mutations (0 invalid, 0 skipped)
ShlomiBalalis added a commit to ShlomiBalalis/scylla-cluster-tests that referenced this issue Aug 27, 2019
bentsi added a commit to scylladb/scylla-cluster-tests that referenced this issue Aug 27, 2019
@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Aug 27, 2019

The commitlog issue looks like #4700

@roydahan roydahan changed the title Coredump in large_partition longevity during nodetool_enospc Coredump during enospc error Oct 22, 2019
@roydahan roydahan changed the title Coredump during enospc error Coredump after enospc error Oct 22, 2019
@fgelcer

This comment has been minimized.

Copy link

@fgelcer fgelcer commented Oct 23, 2019

i have this issue reproduced on:

scylla version - 666.development-0.20191018.d7c3e48e8c4
longevity job - longevity-200gb-48h

           PID: 103741 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Tue 2019-10-22 03:12:06 UTC (1 day 6h ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: d11d1dea8e1c4ade8918f25dc1a87686
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: ip-10-0-43-96.eu-west-1.compute.internal
      Coredump: /var/lib/systemd/coredump/core.scylla.996.d11d1dea8e1c4ade8918f25dc1a87686.103741.1571713926000000
       Message: Process 103741 (scylla) of user 996 dumped core.
                
                Stack trace of thread 103745:
                #0  0x00007fd6be991e75 raise (libc.so.6)
                #1  0x00007fd6be97c95e abort (libc.so.6)
                #2  0x00007fd6be97c769 __assert_fail_base.cold (libc.so.6)
                #3  0x00007fd6be98a566 __assert_fail (libc.so.6)
                #4  0x00000000028ed230 _ZThn48_N7seastar33append_challenged_posix_file_implD0Ev (scylla)
                #5  0x0000000000a25a59 _ZN7seastar20shared_ptr_count_forI17checked_file_implED0Ev (scylla)
                #6  0x00000000016ff65e _ZN7seastar12continuationIZZNS_6futureIJEE9then_implIZZN2db9commitlog15segment_manager19allocate_segment_exERKNS5_10descriptorENS_13basic_sstringIcjLj15ELb1EEENS_10open_flagsEbENKUlNS_4fileEE0_clESD_EUlvE0_NS1_IJNS_10shared_ptrINS5_7seg
                #7  0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #8  0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #9  0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #10 0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #11 0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #12 0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #13 0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103767:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103764:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103756:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103757:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103769:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103758:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103744:
                #0  0x000000000295e3ca _ZN7seastar17smp_message_queue20flush_response_batchEv (scylla)
                #1  0x00000000029994f6 _ZN7seastar3smp11poll_queuesEv (scylla)
                #2  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #3  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #4  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #5  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #6  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scy
coredumpctl dump 103741
           PID: 103741 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Tue 2019-10-22 03:12:06 UTC (1 day 6h ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: d11d1dea8e1c4ade8918f25dc1a87686
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: ip-10-0-43-96.eu-west-1.compute.internal
      Coredump: /var/lib/systemd/coredump/core.scylla.996.d11d1dea8e1c4ade8918f25dc1a87686.103741.1571713926000000
       Message: Process 103741 (scylla) of user 996 dumped core.
                
                Stack trace of thread 103745:
                #0  0x00007fd6be991e75 raise (libc.so.6)
                #1  0x00007fd6be97c95e abort (libc.so.6)
                #2  0x00007fd6be97c769 __assert_fail_base.cold (libc.so.6)
                #3  0x00007fd6be98a566 __assert_fail (libc.so.6)
                #4  0x00000000028ed230 _ZThn48_N7seastar33append_challenged_posix_file_implD0Ev (scylla)
                #5  0x0000000000a25a59 _ZN7seastar20shared_ptr_count_forI17checked_file_implED0Ev (scylla)
                #6  0x00000000016ff65e _ZN7seastar12continuationIZZNS_6futureIJEE9then_implIZZN2db9commitlog15segment_manager19allocate_segment_exERKNS5_10descriptorENS_13basic_sstringIcjLj15ELb1EEENS_10open_flagsEbENKUlNS_4fileEE0_clESD_EUlvE0_NS1_IJNS_10shared_ptrINS5_7segmentEEEEEEEET0_OT_ENKUlvE_clEvEUlONS_12future_stateIJEEEE_JEE15run_and_disposeEv (scylla)
                #7  0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #8  0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #9  0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #10 0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #11 0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #12 0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #13 0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103767:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103764:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103756:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103757:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103769:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103758:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103744:
                #0  0x000000000295e3ca _ZN7seastar17smp_message_queue20flush_response_batchEv (scylla)
                #1  0x00000000029994f6 _ZN7seastar3smp11poll_queuesEv (scylla)
                #2  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #3  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #4  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #5  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #6  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #7  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #8  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103766:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103760:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103751:
                #0  0x0000000000daf6dc _ZN8sstables27index_consume_entry_contextINS_14index_consumerEE13process_stateERN7seastar16temporary_bufferIcEE (scylla)
                #1  0x0000000000dd3694 _ZN7seastar8internal8repeaterIZNS_12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS6_14index_consumerEEEEEENS_6futureIJEEEOT_EUlvE_E15run_and_disposeEv (scylla)
                #2  0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #3  0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #4  0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #5  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #6  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #7  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #8  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103762:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103759:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103750:
                #0  0x0000000002959e33 _ZN7seastar7reactor9poll_onceEv (scylla)
                #1  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #2  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103761:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103753:
                #0  0x00007ffc0fffe6c4 n/a (linux-vdso.so.1)
                #1  0x00007ffc0fffe992 __vdso_clock_gettime (linux-vdso.so.1)
                #2  0x00007fd6bea632c3 __clock_gettime (libc.so.6)
                #3  0x00007fd6bed4eb85 _ZNSt6chrono3_V212steady_clock3nowEv (libstdc++.so.6)
                #4  0x00000000029da795 _ZN7seastar7reactor3runEv (scylla)
                #5  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #6  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #7  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #8  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103749:
                #0  0x000000000299954f _ZN7seastar17smp_message_queue19flush_request_batchEv (scylla)
                #1  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #2  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #3  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #4  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #5  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #6  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #7  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103746:
                #0  0x0000000002b21661 _ZN7seastar3tls7session4pullEPvm (scylla)
                #1  0x00007fd6bfb447df n/a (libgnutls.so.30)
                #2  0x00007fd6bfb3b230 n/a (libgnutls.so.30)
                #3  0x00007fd6bfb3d1ee n/a (libgnutls.so.30)
                #4  0x0000000002b47c35 _ZN7seastar3tls7session6do_getEv (scylla)
                #5  0x0000000002b41ae5 _ZSt13__invoke_implIN7seastar6futureIJNS0_16temporary_bufferIcEEEEERMNS0_3tls7sessionEFS4_vERPS6_JEET_St21__invoke_memfun_derefOT0_OT1_DpOT2_ (scylla)
                #6  0x0000000002b4689e _ZN7seastar6futureIJNS_15semaphore_unitsINS_35semaphore_default_exception_factoryENSt6chrono3_V212steady_clockEEEEE4thenIZNS_14with_semaphoreIS2_St5_BindIFMNS_3tls7sessionEFNS0_IJNS_16temporary_bufferIcEEEEEvEPSC_EES5_EENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreIT_T1_EEmOSN_EUlSU_E_SF_EESN_OSU_ (scylla)
                #7  0x0000000000dbac29 _ZN7seastar12input_streamIcE12read_exactlyEm (scylla)
                #8  0x0000000002b99008 _ZN7seastar3rpc10connection10read_frameINS0_26request_frame_with_timeoutEEENT_11return_typeENS_14socket_addressERNS_12input_streamIcEE (scylla)
                #9  0x0000000002ba043f operator() (scylla)
                #10 0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #11 0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #12 0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #13 0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #14 0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #15 0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #16 0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103747:
                #0  0x0000000002998d24 _ZNKSt13__atomic_baseImE4loadESt12memory_order (scylla)
                #1  0x000000000299956e _ZN7seastar17smp_message_queue19process_completionsEj (scylla)
                #2  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #3  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #4  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #5  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #6  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #7  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #8  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103748:
                #0  0x0000000000c9d3c6 _ZN27restricting_mutation_reader15fast_forward_toE14position_rangeNSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #1  0x0000000000b29502 _ZN5cache26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #2  0x0000000000b2a665 _ZN7seastar6futureIJEE9then_implIZN5cache26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS5_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_S1_EET0_OT_ (scylla)
                #3  0x0000000000b29a95 _ZN5cache26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #4  0x0000000000b2ad52 _ZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #5  0x0000000000b2b604 _ZZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE_clEv (scylla)
                #6  0x0000000000b2ae7d _ZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #7  0x0000000000cc5593 _ZN20flat_mutation_reader4impl16fill_buffer_fromIS_EEN7seastar6futureIJbEEERT_NSt6chrono10time_pointINS2_12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #8  0x0000000000b10100 _ZN34single_partition_populating_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #9  0x0000000000b11971 _ZN7seastar12continuationIZZNS_6futureIJEE9then_implIZN34single_partition_populating_reader11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS5_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_S2_EET0_OT_ENKUlvE_clEvEUlONS_12future_stateIJEEEE_JEE15run_and_disposeEv (scylla)
                #10 0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #11 0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #12 0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #13 0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #14 0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #15 0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #16 0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103743:
                #0  0x000000000299954f _ZN7seastar17smp_message_queue19flush_request_batchEv (scylla)
                #1  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #2  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #3  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #4  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #5  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #6  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #7  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103741:
                #0  0x00000000029da754 _ZNKSt8functionIFbvEEclEv (scylla)
                #1  0x000000000290a402 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla)
                #2  0x000000000290b01f _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIJiEEEvEE (scylla)
                #3  0x000000000077e722 main (scylla)
                #4  0x00007fd6be97df33 __libc_start_main (libc.so.6)
                #5  0x00000000007ca9be _start (scylla)
                
                Stack trace of thread 103765:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103768:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103763:
                #0  0x00007fd6bf374bec read (libpthread.so.0)
                #1  0x0000000002dc4d76 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
                #2  0x0000000002dc5062 operator() (scylla)
                #3  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #4  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #5  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103754:
                #0  0x000000000299958c _ZN7seastar3smp11poll_queuesEv (scylla)
                #1  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #2  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #3  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #4  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #5  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #6  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #7  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103755:
                #0  0x000000000299954f _ZN7seastar17smp_message_queue19flush_request_batchEv (scylla)
                #1  0x0000000002a108c6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
                #2  0x0000000002959e3d _ZN7seastar7reactor9poll_onceEv (scylla)
                #3  0x00000000029da769 _ZNKSt8functionIFbvEEclEv (scylla)
                #4  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #5  0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #6  0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #7  0x00007fd6bea55303 __clone (libc.so.6)
                
                Stack trace of thread 103752:
                #0  0x000000000100ae70 _ZNK5query15result_row_view8iteratorEv (scylla)
                #1  0x000000000100d05c _ZN4cql316result_generator20query_result_visitorIRZN13cql_transport10cql_server11fmt_visitor5visitERKNS2_8messages14result_message4rowsEE7visitorE14accept_new_rowEN5query15result_row_viewESE_ (scylla)
                #2  0x0000000001015594 _ZNK5query11result_view7consumeIRN4cql316result_generator20query_result_visitorIRZN13cql_transport10cql_server11fmt_visitor5visitERKNS5_8messages14result_message4rowsEE7visitorEEEEvRKNS_15partition_sliceEOT_ (scylla)
                #3  0x000000000101d709 _ZN13cql_transport10cql_server11fmt_visitor5visitERKNS_8messages14result_message4rowsE (scylla)
                #4  0x0000000000fcf95d _ZN13cql_transport10cql_server10connection11make_resultEsN7seastar10shared_ptrINS_8messages14result_messageEEERKN7tracing15trace_state_ptrEb (scylla)
                #5  0x0000000000fd2484 _ZN7seastar12continuationIZZNS_6futureIJNS_10shared_ptrIN13cql_transport8messages14result_messageEEEEE9then_implIZNS3_10cql_server10connection15process_executeEtNS3_14request_readerERN7service12client_stateE14service_permitEUlT_E0_NS1_IJSt10unique_ptrINS3_8responseESt14default_deleteISJ_EEEEEEET0_OSG_ENKUlvE_clEvEUlONS_12future_stateIJS6_EEEE_JS6_EE15run_and_disposeEv (scylla)
                #6  0x000000000295d492 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
                #7  0x000000000295d6a0 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #8  0x00000000029da736 _ZN7seastar7reactor14run_some_tasksEv (scylla)
                #9  0x00000000029f14dd _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
                #10 0x000000000293fcee _ZNKSt8functionIFvvEEclEv (scylla)
                #11 0x00007fd6bf36b5a2 start_thread (libpthread.so.0)
                #12 0x00007fd6bea55303 __clone (libc.so.6)

backtrace:

0x00000000029fe332
0x0000000002960d5b
0x0000000002961055
0x00000000029610f0
0x00007fd6bf375e7f
/opt/scylladb/libreloc/libc.so.6+0x0000000000037e74
/opt/scylladb/libreloc/libc.so.6+0x0000000000022894
/opt/scylladb/libreloc/libc.so.6+0x0000000000022768
/opt/scylladb/libreloc/libc.so.6+0x0000000000030565
0x00000000028ed22f
0x0000000000a25a58
0x00000000016ff65d
0x000000000295d491
0x000000000295d69f
0x00000000029da735
0x00000000029f14dc
0x000000000293fced
/opt/scylladb/libreloc/libpthread.so.0+0x00000000000085a1
/opt/scylladb/libreloc/libc.so.6+0x00000000000fb302

decoded:

0x00000000029fe332
0x0000000002960d5b
0x0000000002961055
0x00000000029610f0
0x00007fd6bf375e7f
/opt/scylladb/libreloc/libc.so.6+0x0000000000037e74
/opt/scylladb/libreloc/libc.so.6+0x0000000000022894
/opt/scylladb/libreloc/libc.so.6+0x0000000000022768
/opt/scylladb/libreloc/libc.so.6+0x0000000000030565
0x00000000028ed22f
0x0000000000a25a58
0x00000000016ff65d
0x000000000295d491
0x000000000295d69f
0x00000000029da735
0x00000000029f14dc
0x000000000293fced
/opt/scylladb/libreloc/libpthread.so.0+0x00000000000085a1
/opt/scylladb/libreloc/libc.so.6+0x00000000000fb302
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(char const*) at /usr/include/c++/9/ext/new_allocator.h:105
void seastar::install_oneshot_signal_handler<6, &seastar::sigabrt_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /usr/include/c++/9/ext/new_allocator.h:105
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
non-virtual thunk to seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl() at /jenkins/workspace/scylla-master/build/scylla/seastar/include/seastar/core/future.hh:942
addr2line: Dwarf Error: Offset (84016184) greater than or equal to .debug_info size (5976661).
addr2line: Dwarf Error: Unable to read alt ref 84016184.
seastar::shared_ptr_count_for<checked_file_impl>::~shared_ptr_count_for() at /usr/include/c++/9/ext/new_allocator.h:105
seastar::continuation<seastar::future<seastar::shared_ptr<db::commitlog::segment> > seastar::future<>::then_impl<db::commitlog::segment_manager::allocate_segment_ex(db::commitlog::descriptor const&, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::open_flags, bool)::{lambda(seastar::file)#2}::operator()(seastar::file) const::{lambda()#2}, seastar::future<seastar::shared_ptr<db::commitlog::segment> > >(db::commitlog::segment_manager::allocate_segment_ex(db::commitlog::descriptor const&, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::open_flags, bool)::{lambda(seastar::file)#2}::operator()(seastar::file) const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}>::run_and_dispose() at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run_some_tasks() [clone .part.0] at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run() at /usr/include/c++/9/ext/new_allocator.h:105
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/c++/9/ext/new_allocator.h:105
seastar::posix_thread::start_routine(void*) at /usr/include/c++/9/bits/std_function.h:690
@fgelcer

This comment has been minimized.

Copy link

@fgelcer fgelcer commented Oct 23, 2019

this error was also seen during one of the enospc:

2019-10-20 17:09:54.000: (DatabaseLogEvent Severity.CRITICAL): type=RUNTIME_ERROR regex=std::runtime_error line_number=63929 node=Node longevity-200gb-48h-verify-limited--db-node-0ceff346-3 [52.208.191.110 | 10.0.36.189] (seed: False)
2019-10-20T17:09:54+00:00  ip-10-0-36-189 !INFO    | scylla: terminate called after throwing an instance of 'seastar::internal::backtraced<std::runtime_error>'
0x00000000029fe332
0x0000000002960d5b
0x0000000002961055
0x00000000029610f0
0x00007fbc344eae7f
/libreloc/libc.so.6+0x0000000000037e74
/libreloc/libc.so.6+0x0000000000022894
/libreloc/libstdc++.so.6+0x000000000009e6b5
/libreloc/libstdc++.so.6+0x00000000000aa86b
/libreloc/libstdc++.so.6+0x00000000000a9878
/libreloc/libstdc++.so.6+0x00000000000aa223
0x00007fbc33ca5c62
0x00007fbc33ca61b0
/libreloc/libstdc++.so.6+0x00000000000aab1b
0x000000000055285f
0x0000000000cb791b
0x0000000000485e65
0x0000000000d58b68
0x00000000007c85f3

decoded:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(char const*) at /usr/include/c++/9/ext/new_allocator.h:105
void seastar::install_oneshot_signal_handler<6, &seastar::sigabrt_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /usr/include/c++/9/ext/new_allocator.h:105
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
on_internal_error(seastar::logger&, seastar::basic_sstring<char, unsigned int, 15u, true> const&) [clone .cold] at /usr/include/c++/9/ext/new_allocator.h:105
mutation_fragment_stream_validator::~mutation_fragment_stream_validator() at /usr/include/c++/9/ext/new_allocator.h:105
sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}::operator()() [clone .cold] at /usr/include/c++/9/ext/new_allocator.h:105
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::result_of<std::decay<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /usr/include/c++/9/ext/new_allocator.h:105
@fgelcer

This comment has been minimized.

Copy link

@fgelcer fgelcer commented Oct 23, 2019

and this error as well:

2019-10-20 17:46:39.000: (DatabaseLogEvent Severity.ERROR): type=BACKTRACE regex=backtrace line_number=68596 node=Node longevity-200gb-48h-verify-limited--db-node-0ceff346-2 [34.240.254.255 | 10.0.43.96] (seed: False)
2019-10-20T17:46:39+00:00  ip-10-0-43-96 !INFO    | scylla: Backtrace:
0x00000000029fe332
0x0000000002960d5b
0x0000000002961055
0x00000000029610f0
0x00007f837b52be7f
/libreloc/libc.so.6+0x0000000000037e74
/libreloc/libc.so.6+0x0000000000022894
/libreloc/libc.so.6+0x0000000000022768
/libreloc/libc.so.6+0x0000000000030565
0x000000000295c8ce
0x00000000029cd0fe
0x00000000028e9de2
0x00000000029a26c4
0x00000000029a2e25
0x000000000295d491
0x000000000295d69f
0x00000000029da735
0x000000000290a401
0x000000000290b01e
0x000000000077e721
/libreloc/libc.so.6+0x0000000000023f32
0x00000000007ca9bd

decoded:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::print_with_backtrace(char const*) at /usr/include/c++/9/ext/new_allocator.h:105
void seastar::install_oneshot_signal_handler<6, &seastar::sigabrt_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /usr/include/c++/9/ext/new_allocator.h:105
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::reactor::submit_io(seastar::io_desc*, seastar::noncopyable_function<void (seastar::internal::linux_abi::iocb&)>) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::fdatasync(int) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::posix_file_impl::flush() at /jenkins/workspace/scylla-master/build/scylla/seastar/include/seastar/core/future.hh:942
seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::sync_directory(seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda(seastar::file)#1}, seastar::file>(seastar::sync_directory(seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda(seastar::file)#1}&&, std::tuple<seastar::file>&&) [clone .isra.0] at /usr/include/c++/9/ext/new_allocator.h:105
seastar::continuation<seastar::future<> seastar::future<seastar::file>::then_impl<seastar::sync_directory(seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda(seastar::file)#1}, seastar::future<> >(seastar::sync_directory(seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda(seastar::file)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::file>&&)#1}, seastar::file>::run_and_dispose() at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run_some_tasks() [clone .part.0] at /usr/include/c++/9/ext/new_allocator.h:105
seastar::reactor::run() at /usr/include/c++/9/ext/new_allocator.h:105
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /jenkins/workspace/scylla-master/build/scylla/seastar/include/seastar/core/future.hh:942
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /jenkins/workspace/scylla-master/build/scylla/seastar/include/seastar/core/future.hh:942
main at /usr/include/c++/9/ext/new_allocator.h:105
@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

@espindola please look into this issue.
There are quite a few symptoms reported here. I'm not sure they are all related to each other, or different symptoms of hitting OOS in different, unrelated parts of the code.

@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

@fgelcer some of the file names in the decoded backtraces look bogus (like new_allocator.h).
How did you generate the decoded backtraces?

@fgelcer

This comment has been minimized.

Copy link

@fgelcer fgelcer commented Oct 23, 2019

forgot to attach the link to download the coredump

@fgelcer

This comment has been minimized.

Copy link

@fgelcer fgelcer commented Oct 23, 2019

@fgelcer some of the file names in the decoded backtraces look bogus (like new_allocator.h).
How did you generate the decoded backtraces?

those were backtraces found in the logs, and i decoded them by using:

addr2line -Cpfie /usr/lib/debug/opt/scylladb/libexec/scylla-666.development-0.20191018.d7c3e48e8c4.el7.x86_64.debug
@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

@fgelcer, thanks. Also, full logs would be appreciated

@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

Looks like allocate_segment_ex doesn't close the file it opens in the lambda function on exception.

future<db::commitlog::segment_manager::sseg_ptr> db::commitlog::segment_manager::allocate_segment_ex(const descriptor& d, sstring filename, open_flags flags, bool active) {
file_open_options opt;
opt.extent_allocation_size_hint = max_size;
auto fut = do_io_check(commit_error_handler, [=] {
auto fut = open_file_dma(filename, flags, opt);
if (cfg.extensions && !cfg.extensions->commitlog_file_extensions().empty()) {
fut = fut.then([this, filename, flags](file f) {
return do_with(std::move(f), [this, filename, flags](file& f) {
auto ext_range = cfg.extensions->commitlog_file_extensions();
return do_for_each(ext_range.begin(), ext_range.end(), [&f, filename, flags](auto& ext) {
// note: we're potentially wrapping more than once. extension mechanism
// is responsible for order being sane.
return ext->wrap_file(filename, f, flags).then([&f](file of) {
if (of) {
f = std::move(of);
}
});
}).then([&f] {
return f;
});
});
});
}
return fut;
});

@elcallio

This comment has been minimized.

Copy link
Contributor

@elcallio elcallio commented Oct 23, 2019

file destructor closes fd. Since no objects should escape alive on exception here, it should be ok. Sidenote, any c++ object not adhering to raii should be chastised and flogged. And author to.
Also, exception where?

@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

file destructor closes fd. Since no objects should escape alive on exception here, it should be ok. Sidenote, any c++ object not adhering to raii should be chastised and flogged. And author to.
Also, exception where?

looks like on open (due to out of space).
We don't close files upon destruction on purpose so to not hit an exception there - hence the assert in seastar wining about destructing a file while open.
So the expectation is that whoever opened the file is responsible for closing it before it gets destructed .

@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

The relevant backtrace is this one:

                #3  0x00007fd6be98a566 __assert_fail (libc.so.6)
                #4  0x00000000028ed230 _ZThn48_N7seastar33append_challenged_posix_file_implD0Ev (scylla)
                #5  0x0000000000a25a59 _ZN7seastar20shared_ptr_count_forI17checked_file_implED0Ev (scylla)
                #6  0x00000000016ff65e _ZN7seastar12continuationIZZNS_6futureIJEE9then_implIZZN2db9commitlog15segment_manager19allocate_segment_exERKNS5_10descriptorENS_13basic_sstringIcjLj15ELb1EEENS_10open_flagsEbENKUlNS_4fileEE0_clESD_EUlvE0_NS1_IJNS_10shared_ptrINS5_7segmentEEEEEEEET0_OT_ENKUlvE_clEvEUlONS_12future_stateIJEEEE_JEE15run_and_disposeEv (scylla)
@bhalevy

This comment has been minimized.

Copy link
Contributor

@bhalevy bhalevy commented Oct 23, 2019

Hmm, I'm guessing that maybe truncate hit the exception (due to the involvement of checked_file_impl)

@elcallio

This comment has been minimized.

Copy link
Contributor

@elcallio elcallio commented Oct 23, 2019

We absolutely close fd:s on destruction. Otherwise we would have file leaks an masse, because, no matter how stringent, we'd miss places where continuations would otherwise make us loose a file object.
As far as for the assert in append_challenged_file_impl destructor, the logic is somewhat debatable. Having called close is no more a guarantee that noone is trying to access the object/fd than holding the object alive. It just lessens the racing a bit by maybe removing the factor of pending read/writes. But if so, it should accept direct destruction of any object that has not yet initiated any op that enters the completion queue.
Also, this restriction is again only on append_challenged...
But more to the point, this restriction break RAII. My suggestion would be to augment the file object (not impl) destructor to check if impl is safely destructable, and if not, place it on a dangling task that forces pending ops out before destruction, and thus making object ownership the only factor counted.

As for the case above, sure, you should maybe hang on "on_exception" handler to both blocks in the function, and it will deal with the truncation exception. But again, I am somewhat against because this is a very C-like restriction that has no place in a nice c++ universe.

@espindola

This comment has been minimized.

Copy link
Contributor

@espindola espindola commented Oct 23, 2019

We don't close on destructor as we have to wait and there is no guarantee that we are on a thread.

I am looking at the code Benny pointed out it #4877 (comment)

@elcallio

This comment has been minimized.

Copy link
Contributor

@elcallio elcallio commented Oct 23, 2019

Again, if we have potential wait condition (beyond ::close - not file::close()), it should be the responsibility of file to queue this up somewhere in background (see tls socket close for example).

avikivity added a commit that referenced this issue Nov 21, 2019
If allocate or truncate throws, we have to close the file.

Fixes #4877

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20191114174810.49004-1-espindola@scylladb.com>
@roydahan roydahan added the longevity label Nov 24, 2019
avikivity added a commit that referenced this issue Nov 24, 2019
If allocate or truncate throws, we have to close the file.

Fixes #4877

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20191114174810.49004-1-espindola@scylladb.com>
(cherry picked from commit 6160b90)
avikivity added a commit that referenced this issue Nov 24, 2019
If allocate or truncate throws, we have to close the file.

Fixes #4877

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20191114174810.49004-1-espindola@scylladb.com>
(cherry picked from commit 6160b90)
avikivity added a commit that referenced this issue Nov 24, 2019
If allocate or truncate throws, we have to close the file.

Fixes #4877

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20191114174810.49004-1-espindola@scylladb.com>
(cherry picked from commit 6160b90)
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Dec 23, 2019

Already backported to 3.0+.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.