SIGFAULT while running select * in a 3 node cluster on AWS

[slivne]

Head at:

commit 5f88707
Merge: 5d68f33 83a64d7
Author: Avi Kivity avi@cloudius-systems.com
Date: Wed Jul 15 14:52:15 2015 +0300

Merge "API: Replacing empty string to void" from Amnon

All nodes are running with --smp 1

Running cassandra-stress of insert works

cassandra/tools/bin/cassandra-stress user n=100 no-warmup profile=./simple_test_no_compression.yaml ops(insert=1) -mode cql3 native -rate threads=10 -node 172.31.19.92

Connecting with CQLSH and executing select * from keyspace2.standrad1 fails on the node I connect to

Program received signal SIGSEGV, Segmentation fault.
__destroy<partition*> (__last=0x0, __first=0x1) at /usr/include/c++/4.9/bits/stl_construct.h:103
103 std::_Destroy(std::__addressof(__first));
(gdb) where
#0 __destroy<partition> (__last=0x0, __first=0x1) at /usr/include/c++/4.9/bits/stl_construct.h:103
#1 _Destroy<partition*> (__last=0x0, __first=) at /usr/include/c++/4.9/bits/stl_construct.h:126
#2 _Destroy<partition*, partition> (__last=0x0, __first=) at /usr/include/c++/4.9/bits/stl_construct.h:151
#3 std::vector<partition, std::allocator >::~vector (this=0x7fffffffda20, __in_chrg=) at /usr/include/c++/4.9/bits/stl_vector.h:424
#4 0x000000000058a1e8 in _M_move_assign (__x=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x2021b53, DIE 0x20cef30>, this=0x6000006d02c8) at /usr/include/c++/4.9/bits/stl_vector.h:1459
#5 operator= (__x=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x2021b53, DIE 0x20cef6c>, this=0x6000006d02c8) at /usr/include/c++/4.9/bits/stl_vector.h:453
#6 operator= (this=0x6000006d02c0) at mutation_query.hh:47
#7 db::serializer<reconcilable_result>::read (v=..., in=...) at mutation_query.cc:90
#8 0x00000000007a4ecf in operator() (__closure=, __closure=, buf=)

at ./message/messaging_service.hh:144

#9 apply (args=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4f35684>, func=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4f35672>)

at ./core/apply.hh:34

#10 applynet::serializer::read_serializable(input_stream<char&, Serializable&)::<lambda(auto:27)> mutable [with auto:27 = unsigned int; Serializable = reconcilable_result]::<lambda(temporary_buffer)>, temporary_buffer > (args=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4ddddfe>,

func=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4ddddea>) at ./core/apply.hh:42

#11 applynet::serializer::read_serializable(input_stream<char&, Serializable&)::<lambda(auto:27)> mutable [with auto:27 = unsigned int; Serializable = reconcilable_result]::<lambda(temporary_buffer)>, temporary_buffer > (args=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4e341fa>,

func=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x4a8be79, DIE 0x4e341e7>) at ./core/future.hh:948

#12 ZZN6futureII16temporary_bufferIcEEE4thenIvZZN3net10serializer17read_serializableI19reconcilable_resultEES_IIEER12input_streamIcERT_ENUlSC_E_clIjEEDaT_EUlS1_E_ZNS2_4thenISH_S8_EET0_OSG_EUlO12future_stateIIS1_EEE_EEN8futurizeISG_E4typeEOSJ_OT1_ENUlOSC_E_clISM_EEDaSK (__closure=0x600000215cd8, state=) at ./core/future.hh:626
#13 0x00000000009a70f2 in reactor::run_tasks (this=0x7fffffffda20, this@entry=0x60000020b000, tasks=..., quota=105553117362128) at core/reactor.cc:1011
#14 0x00000000009c894b in reactor::run (this=0x60000020b000) at core/reactor.cc:1108
#15 0x00000000009ecc76 in app_template::run(int, char**, std::function<void ()>&&) (this=this@entry=0x7fffffffe3a0, ac=ac@entry=9, av=av@entry=0x7fffffffe5f8,

func=func@entry=<unknown type in /home/ubuntu/urchin/build/release/seastar, CU 0x9815342, DIE 0x98ab69a>) at core/app-template.cc:104

#16 0x000000000041622e in main (ac=9, av=0x7fffffffe5f8) at main.cc:157

Verified that if I do this toward a single node running on the same HW - everything is working ok

[slivne]

We do not support it at this point - doing range_queries in read in cluster - closing

[gleb-cloudius]

On Wed, Jul 15, 2015 at 09:31:05AM -0700, slivne wrote:

We do not support it at this point - doing range_queries in read in cluster - closing

The crash has nothing to do with range_queries read in cluster though.
It crashes in deserializer because it uses assign operator instead of
placement new. This patch should fix the crash:

diff --git a/mutation_query.cc b/mutation_query.cc
index 50b6f27..539dd1e 100644
--- a/mutation_query.cc
+++ b/mutation_query.cc
@@ -87,7 +87,7 @@ void db::serializer<reconcilable_result>::read(reconcilable_result& v, input& in
auto fm = db::frozen_mutation_serializer::read(in);
partitions.emplace_back(partition(p_row_count, std::move(fm)));
}

• v = reconcilable_result(row_count, std::move(partitions));
• new (&v) reconcilable_result(row_count, std::move(partitions));
  }

future<reconcilable_result>

    Gleb.

[tgrabiec]

2015-07-15 18:49 GMT+02:00 Gleb Natapov notifications@github.com:

On Wed, Jul 15, 2015 at 09:31:05AM -0700, slivne wrote:

We do not support it at this point - doing range_queries in read in
cluster - closing

The crash has nothing to do with range_queries read in cluster though.
It crashes in deserializer because it uses assign operator instead of
placement new. This patch should fix the crash:

diff --git a/mutation_query.cc b/mutation_query.cc
index 50b6f27..539dd1e 100644
--- a/mutation_query.cc
+++ b/mutation_query.cc
@@ -87,7 +87,7 @@ void
db::serializer<reconcilable_result>::read(reconcilable_result& v, input& in
auto fm = db::frozen_mutation_serializer::read(in);
partitions.emplace_back(partition(p_row_count, std::move(fm)));
}

• v = reconcilable_result(row_count, std::move(partitions));
• new (&v) reconcilable_result(row_count, std::move(partitions));

The output of db::serializer::read() is not said to be uninitialized,
there's this for example:

template<>
void db::serializer::read(sstring& s, input& in) {
s = in.read();
}

The fix could be in net::serializer::read_serializable(), but it assumes
that values are movable:

diff --git a/message/messaging_service.hh b/message/messaging_service.hh
index 2a3033d..897969b 100644
--- a/message/messaging_service.hh
+++ b/message/messaging_service.hh
@@ -136,7 +136,7 @@ struct serializer {
}
bytes_view bv(reinterpret_cast<const int8_t*>(buf.get()),
sz);
data_input in(bv);

•            db::serializer<Serializable>::read(v, in);
  

•            new (&v) (db::serializer<Serializable>::read(in));
       });
   });
  

  }

Or we could align db::serializer<>::read() so that they're like rpc
deserializers wrt to output parameter.

[gleb-cloudius]

On Wed, Jul 15, 2015 at 10:09:07AM -0700, Tomasz Grabiec wrote:

2015-07-15 18:49 GMT+02:00 Gleb Natapov notifications@github.com:

On Wed, Jul 15, 2015 at 09:31:05AM -0700, slivne wrote:

We do not support it at this point - doing range_queries in read in
cluster - closing

The crash has nothing to do with range_queries read in cluster though.
It crashes in deserializer because it uses assign operator instead of
placement new. This patch should fix the crash:

diff --git a/mutation_query.cc b/mutation_query.cc
index 50b6f27..539dd1e 100644
--- a/mutation_query.cc
+++ b/mutation_query.cc
@@ -87,7 +87,7 @@ void
db::serializer<reconcilable_result>::read(reconcilable_result& v, input& in
auto fm = db::frozen_mutation_serializer::read(in);
partitions.emplace_back(partition(p_row_count, std::move(fm)));
}

• v = reconcilable_result(row_count, std::move(partitions));
• new (&v) reconcilable_result(row_count, std::move(partitions));

The output of db::serializer::read() is not said to be uninitialized,
there's this for example:

template<>
void db::serializer::read(sstring& s, input& in) {
s = in.read();
}

The fix could be in net::serializer::read_serializable(), but it assumes
that values are movable:

diff --git a/message/messaging_service.hh b/message/messaging_service.hh
index 2a3033d..897969b 100644
--- a/message/messaging_service.hh
+++ b/message/messaging_service.hh
@@ -136,7 +136,7 @@ struct serializer {
}
bytes_view bv(reinterpret_cast<const int8_t*>(buf.get()),
sz);
data_input in(bv);

•            db::serializer<Serializable>::read(v, in);
  

•            new (&v) (db::serializer<Serializable>::read(in));
       });
   });
  

  }

Or we could align db::serializer<>::read() so that they're like rpc
deserializers wrt to output parameter.

I think your fix is good. Serializable class should be easily movable.

        Gleb.