Assertion `_state != state::future' failed during read workload

[pdziepak]

Installation details
Scylla version (or git commit hash): 2.0-rc1
Cluster of 3, RF=3

Read workload with medium-size partitions:

scylla-bench -duration 15m -partition-count 1000000 -clustering-row-count 1000 -clustering-row-size 128 -connection-count 32 -measure-latency=false -replication-factor 3 -concurrency=256 -workload=uniform -mode=read

During a 1 hour run this assertion failure has happened twice:

Assertion `_state != state::future' failed.
Aborting on shard 11.
Backtrace:
0x00000000004eef08
0x00000000004ef025
0x00000000004ef0d3
0x00007f826c2630ff
0x00007f826bec75f6
0x00007f826bec8ce7
0x00007f826bec0565
0x00007f826bec0611
0x000000000058a437
0x0000000000599adb
0x00000000004eccbf
0x00000000005407aa
0x0000000000542050
0x000000000059a36d
0x00007f826c25bdc4
0x00007f826bf88cec
seastar::backtrace_buffer::append_backtrace() at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:277
 (inlined by) print_with_backtrace at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:288
print_with_backtrace at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:295
sigabrt_action at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:3512
 (inlined by) operator() at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:3494
 (inlined by) _FUN at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:3490
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::future<seastar::temporary_buffer<char> >::ignore_ready_future() at /usr/src/debug/scylla-2.0.rc1/seastar/core/future.hh:258
 (inlined by) seastar::file_data_source_impl::close()::{lambda()#1}::operator()() const at /usr/src/debug/scylla-2.0.rc1/seastar/core/fstream.cc:218
seastar::apply_helper<seastar::file_data_source_impl::close()::{lambda()#1}, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply({lambda()#1}&&, std::tuple<>) at /usr/src/debug/scylla-2.0.rc1/seastar/core/apply.hh:36
 (inlined by) auto seastar::apply<seastar::file_data_source_impl::close()::{lambda()#1}>(seastar::file_data_source_impl::close()::{lambda()#1}&&, std::tuple<>&&) at /usr/src/debug/scylla-2.0.rc1/seastar/core/apply.hh:44
 (inlined by) seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::file_data_source_impl::close()::{lambda()#1}>(seastar::file_data_source_impl::close()::{lambda()#1}&&, std::tuple<>&&) at /usr/src/debug/scylla-2.0.rc1/seastar/core/future.hh:1302
 (inlined by) _ZZN7seastar6futureIJEE4thenIZNS_21file_data_source_impl5closeEvEUlvE_S1_EET0_OT_ENUlS7_E_clINS_12future_stateIJEEEEEDaS7_ at /usr/src/debug/scylla-2.0.rc1/seastar/core/future.hh:900
 (inlined by) _ZN7seastar12continuationIZNS_6futureIJEE4thenIZNS_21file_data_source_impl5closeEvEUlvE_S2_EET0_OT_EUlS8_E_JEE3runEv at /usr/src/debug/scylla-2.0.rc1/seastar/core/future.hh:395
seastar::reactor::run_tasks(seastar::circular_buffer<std::unique_ptr<seastar::task, std::default_delete<seastar::task> >, std::allocator<std::unique_ptr<seastar::task, std::default_delete<seastar::task> > > >&) at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:2316
seastar::reactor::run() at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:2774
operator() at /usr/src/debug/scylla-2.0.rc1/seastar/core/reactor.cc:3707
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/5.3.1/functional:2271
 (inlined by) seastar::posix_thread::start_routine(void*) at /usr/src/debug/scylla-2.0.rc1/seastar/core/posix.cc:52
?? ??:0
?? ??:0

Could be related to #2696.

[pdziepak]

It is very likely that this is caused by scylladb/seastar#322 (and probably not related to #2696). I'm going to try to verify that.