New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault on INSERT #3688

Closed
withings-sas opened this Issue Aug 13, 2018 · 29 comments

Comments

Projects
None yet
4 participants
@withings-sas

withings-sas commented Aug 13, 2018

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

  • I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 2.2.0
Cluster size: 1 node (testing)
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu

While trying to replace cassandra with scylla in our test environment, we are experiencing a segmentation fault with some INSERT queries.

The query:

INSERT INTO table (uid, id, brand, id2, dtype, model, attrib, pp, startdate, exp, array1, array2) VALUES (:uid, :id, :brand, :id2, :dtype, :model, :attrib, :pp, :startdate, :exp, :array1, :array2)

The table schema:

CREATE TABLE table (
    uid bigint,
    dtype int,
    brand int,
    startdate int,
    id int,
    id2 int,
    attrib int,
    array1 list<int>,
    array2 list<int>,
    enddate int,
    exp int,
    model int,
    pp int,
    PRIMARY KEY ((uid, dtype, brand), startdate, id, id2, attrib)
) WITH CLUSTERING ORDER BY (startdate ASC, id ASC, id2 ASC, attrib ASC)

As you can see uid is bigint and array1 and array2 are list, other than that nothing peculiar.

The backtrace:

root@vm:~# addr2line -Cfpi -e /usr/lib/debug/.build-id/3d/1b89d220a3a938bd1dc33a8c0f1094d13fe1be.debug 0x00000000021edb94, 0x00000000005b34ec, 0x00000000005b3795, 0x00000000005b37e3, 0x00000000021edb93, 0x0000000002195ba2, 0x00000000019a44f6, 0x00000000019ade13, 0x00000000019af02b, 0x00000000019d5639, 0x00000000019da78d, 0x000000000067c3ec, 0x00000000005962d6
cql3::raw_value& std::vector<cql3::raw_value, std::allocator<cql3::raw_value> >::emplace_back<cql3::raw_value&>(cql3::raw_value&) at /opt/scylladb/include/c++/7/bits/vector.tcc:98
 (inlined by) cql3::query_options::prepare(std::vector<seastar::shared_ptr<cql3::column_specification>, std::allocator<seastar::shared_ptr<cql3::column_specification> > > const&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/cql3/query_options.cc:218
seastar::backtrace_buffer::append_backtrace() at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:303
 (inlined by) print_with_backtrace at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:324
seastar::print_with_backtrace(char const*) at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:331
sigsegv_action at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:3797
 (inlined by) operator() at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:3783
 (inlined by) _FUN at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:3779
cql3::raw_value& std::vector<cql3::raw_value, std::allocator<cql3::raw_value> >::emplace_back<cql3::raw_value&>(cql3::raw_value&) at /opt/scylladb/include/c++/7/bits/vector.tcc:98
 (inlined by) cql3::query_options::prepare(std::vector<seastar::shared_ptr<cql3::column_specification>, std::allocator<seastar::shared_ptr<cql3::column_specification> > > const&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/cql3/query_options.cc:218
cql3::query_processor::process(std::experimental::fundamentals_v1::basic_string_view<char, std::char_traits<char> > const&, service::query_state&, cql3::query_options&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/cql3/query_processor.cc:193
cql_transport::cql_server::connection::process_query(unsigned short, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, service::client_state) at /build/scylla-server-2.2.0-0.20180705.240b9f122/transport/server.cc:864
cql_transport::cql_server::connection::process_request_one(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)::{lambda()#1}::operator()() at /build/scylla-server-2.2.0-0.20180705.240b9f122/transport/server.cc:512
apply at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/apply.hh:36
 (inlined by) apply<cql_transport::cql_server::connection::process_request_one(bytes_view, uint8_t, uint16_t, service::client_state, cql_transport::cql_server::connection::tracing_request_type)::<lambda()> > at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/apply.hh:44
 (inlined by) apply<cql_transport::cql_server::connection::process_request_one(bytes_view, uint8_t, uint16_t, service::client_state, cql_transport::cql_server::connection::tracing_request_type)::<lambda()> > at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/future.hh:1362
 (inlined by) then<cql_transport::cql_server::connection::process_request_one(bytes_view, uint8_t, uint16_t, service::client_state, cql_transport::cql_server::connection::tracing_request_type)::<lambda()> > at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/future.hh:932
 (inlined by) cql_transport::cql_server::connection::process_request_one(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type) at /build/scylla-server-2.2.0-0.20180705.240b9f122/transport/server.cc:475
seastar::future<cql_transport::cql_server::connection::processing_result> std::__invoke_impl<seastar::future<cql_transport::cql_server::connection::processing_result>, seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::* const&)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>(std::__invoke_memfun_deref, seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::* const&)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), cql_transport::cql_server::connection*&&, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >&&, unsigned char&&, unsigned short&&, service::client_state&&, cql_transport::cql_server::connection::tracing_request_type&&) at /opt/scylladb/include/c++/7/bits/invoke.h:73
 (inlined by) std::__invoke_result<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::* const&)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>::type std::__invoke<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::* const&)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>(seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::* const&)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), cql_transport::cql_server::connection*&&, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >&&, unsigned char&&, unsigned short&&, service::client_state&&, cql_transport::cql_server::connection::tracing_request_type&&) at /opt/scylladb/include/c++/7/bits/invoke.h:96
 (inlined by) decltype (__invoke((*this)._M_pmf, (forward<cql_transport::cql_server::connection*>)({parm#1}), (forward<std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> > >)({parm#1}), (forward<unsigned char>)({parm#1}), (forward<unsigned short>)({parm#1}), (forward<service::client_state>)({parm#1}), (forward<cql_transport::cql_server::connection::tracing_request_type>)({parm#1}))) std::_Mem_fn_base<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::*)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type), true>::operator()<cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>(cql_transport::cql_server::connection*&&, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >&&, unsigned char&&, unsigned short&&, service::client_state&&, cql_transport::cql_server::connection::tracing_request_type&&) const at /opt/scylladb/include/c++/7/functional:175
 (inlined by) seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>::direct_vtable_for<std::_Mem_fn<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection::*)(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)> >::call(seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)> const*, cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type) at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/util/noncopyable_function.hh:71
seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>::operator()(cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type) const at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/util/noncopyable_function.hh:145
 (inlined by) seastar::apply_helper<seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, std::tuple<cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>&&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul> >::apply(seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, std::tuple<cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>&&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/apply.hh:36
 (inlined by) auto seastar::apply<seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>(seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, std::tuple<cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>&&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/apply.hh:44
 (inlined by) seastar::future<cql_transport::cql_server::connection::processing_result> seastar::futurize<seastar::future<cql_transport::cql_server::connection::processing_result> >::apply<seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>(seastar::noncopyable_function<seastar::future<cql_transport::cql_server::connection::processing_result> (cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type)>&, std::tuple<cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>&&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/future.hh:1362
 (inlined by) seastar::concrete_execution_stage<seastar::future<cql_transport::cql_server::connection::processing_result>, cql_transport::cql_server::connection*, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, unsigned char, unsigned short, service::client_state, cql_transport::cql_server::connection::tracing_request_type>::do_flush() at /build/scylla-server-2.2.0-0.20180705.240b9f122/./seastar/core/execution_stage.hh:286
seastar::execution_stage::flush()::{lambda()#1}::operator()() const at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/execution_stage.hh:164
 (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::{lambda()#1}>::run_and_dispose() at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/task.hh:48
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:2500
seastar::reactor::run_some_tasks() at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:2910
seastar::reactor::run_some_tasks() at /opt/scylladb/include/c++/7/chrono:377
 (inlined by) seastar::reactor::run() at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/reactor.cc:3057
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /build/scylla-server-2.2.0-0.20180705.240b9f122/seastar/core/app-template.cc:180
main at /build/scylla-server-2.2.0-0.20180705.240b9f122/main.cc:749
_start at ??:?

@withings-sas withings-sas changed the title from Segmentation fault to Segmentation fault on INSERT Aug 13, 2018

@tzach tzach added the bug label Aug 13, 2018

@tzach

This comment has been minimized.

Contributor

tzach commented Aug 13, 2018

@withings-sas thanks for reporting
can you please upload the core dump as explain here
http://docs.scylladb.com/operating-scylla/troubleshooting/report_scylla_problem/#core-dump

@withings-sas

This comment has been minimized.

withings-sas commented Aug 13, 2018

I am not really enclined to upload a coredump on a public tracker, is there another way I can transfer it ?

@tgrabiec

This comment has been minimized.

Contributor

tgrabiec commented Aug 13, 2018

@withings-sas If you upload using method described in the following link, the files will be accessible only to ScyllaDB employees:

http://docs.scylladb.com/operating-scylla/troubleshooting/report_scylla_problem/#send-files-to-scylladb-support

@withings-sas

This comment has been minimized.

withings-sas commented Aug 13, 2018

thank you: bfd89779-dee9-4421-8c59-4f1724b9b6d5

@duarten

This comment has been minimized.

Member

duarten commented Aug 14, 2018

query_options::_values is empty:

(gdb) p options._values._M_impl
$7 = {<std::allocator<cql3::raw_value>> = {<__gnu_cxx::new_allocator<cql3::raw_value>> = {<No data fields>}, <No data fields>}, _M_start = 0x0, _M_finish = 0x0,
  _M_end_of_storage = 0x0}

While query_options::names and prepared_statement::bound_names are not:

(gdb) p (options._names._M_payload._M_impl._M_finish - options._names._M_payload._M_impl._M_start)
$12 = 12

(gdb) p (*((cql3::statements::prepared_statement*)p._M_t._M_t)).bound_names._M_impl._M_finish - (*((cql3::statements::prepared_statement*)p._M_t._M_t)).bound_names._M_impl._M_start
$28 = 12

This matches the amount of fields in the insert statement.

There's definitely a bug on our side here since we validate this mismatch too late:

options.prepare(p->bound_names); // oops
...
if (cql_statement->get_bound_terms() != options.get_values_count()) {
    throw exceptions::invalid_request_exception("Invalid amount of bind variables"); // too late
}

We'll fix this on our side. Another issue is that your driver doesn't seem to be passing the prepared statement values to Scylla. Are you using the Java driver? Have you tried inserting data through cqlsh? @withings-sas

Memory and tasks seem okay:

(gdb) scylla memory
Used memory:     101322752
Free memory:     420868096
Total memory:    522190848

Small pools:
objsz spansz    usedobj       memory  wst%
    1   4096          0            0   0.0
    1   4096          0            0   0.0
    1   4096          0            0   0.0
    1   4096          0            0   0.0
    2   4096          0            0   0.0
    2   4096          0            0   0.0
    3   4096          0            0   0.0
    3   4096          0            0   0.0
    4   4096          0            0   0.0
    5   4096          0            0   0.0
    6   4096          0            0   0.0
    7   4096          0            0   0.0
    8   4096       2113        20480  17.5
   10   4096          3         8192  99.5
   12   4096        187         8192  72.5
   14   4096          1         8192  99.6
   16   4096       3661        65536  10.6
   20   4096       2247        49152   8.2
   24   4096       3417        90112   8.6
   28   4096       1120        32768   4.1
   32   4096       3113       106496   6.5
   40   4096       4538       188416   3.3
   48   4096       1408        73728   7.9
   56   4096       7195       409600   1.4
   64   4096       6174       401408   1.6
   80   4096      11418       921600   0.5
   96   4096       1894       188416   1.9
  112   4096       1328       159744   5.3
  128   4096        392        61440  18.3
  160   4096       1937       327680   3.1
  192   4096       1120       233472   6.3
  224   4096        130        45056  33.8
  256   4096        705       200704  10.1
  320   8192        997       352256   7.1
  384   8192        686       294912   9.1
  448   4096        258       159744  26.1
  512   4096        229       159744  26.6
  640  12288         56        49152  26.0
  768  12288       1202       983040   6.1
  896   8192        244       294912  24.3
 1024   4096         83       176128  51.7
 1280  20480        304       512000  24.0
 1536  12288         78       245760  51.2
 1792  16384          8       147456  88.7
 2048   8192        364       778240   4.2
 2560  20480          4       204800  95.0
 3072  12288        374      1167360   1.6
 3584  28672          0       286720 100.0
 4096  16384        186       933888  18.4
 5120  20480          8       389120  89.5
 6144  24576         48       466944  36.8
 7168  28672         49       544768  35.5
 8192  32768        397      3735552  12.9
10240  40960         49      1351680  62.9
12288  49152          5       933888  93.4
14336  57344          0      1089536 100.0
16384  65536        146      3735552  36.0
Page spans:
index      size [B] free [B]
    0          4096 131072
    1          8192 458752
    2         16384 479232
    3         32768 356352
    4         65536 0
    5        131072 10223616
    6        262144 0
    7        524288 0
    8       1048576 0
    9       2097152 0
   10       4194304 0
   11       8388608 0
   12      16777216 0
   13      33554432 0
   14      67108864 0
   15     134217728 0
   16     268435456 409219072
   17     536870912 0
   18    1073741824 0
   19    2147483648 0
   20    4294967296 0
   21    8589934592 0
   22   17179869184 0
   23   34359738368 0
   24   68719476736 0
   25  137438953472 0
   26  274877906944 0
   27  549755813888 0
   28 1099511627776 0
   29 2199023255552 0
   30 4398046511104 0
   31 8796093022208 0
(gdb) scylla task_histogram
      3713: 0x3f37838 vtable for seastar::shared_ptr_count_for<cql3::column_specification> + 16
      3711: 0x3f37858 vtable for seastar::shared_ptr_count_for<cql3::column_identifier> + 16
       365: 0x3e477f8 vtable for seastar::httpd::function_handler + 16
       359: 0x3f71700 vtable for seastar::shared_ptr_count_for<logalloc::region_impl> + 16
       358: 0x3f34798 _ZTVN7seastar12continuationIZNS_6futureIJEE12then_wrappedIZNS_13shared_futureIJEEC4EOS2_EUlS6_E_S2_EET0_OT_EUlSA_E_JEEE + 16
       301: 0x3de2c90 vtable for seastar::shared_ptr_count_for<seastar::metrics::impl::registered_metric> + 16
       235: 0x3de2bf8 vtable for seastar::metrics::impl::metric_groups_impl + 16
       208: 0x3f37878 vtable for seastar::shared_ptr_count_for<query::partition_slice> + 16
       192: 0x3ffdbf8 vtable for sstables::bag_sstable_set + 16
       178: 0x3f3d098 _ZTVN7seastar12continuationIZNS_6futureIJNS_13lw_shared_ptrIK6schemaEEEE12then_wrappedIZNS_13shared_futureIJS5_EEC4EOS6_EUlSA_E_NS1_IJEEEEET0_OT_EUlSF_E_JS5_EEE + 16
       153: 0x3e6e618 vtable for boost::detail::sp_counted_impl_p<boost::program_options::value_semantic const> + 16
       153: 0x3e6e660 vtable for boost::detail::sp_counted_impl_p<boost::program_options::option_description> + 16
       153: 0x3e6e3d8 vtable for boost::program_options::option_description + 16
       128: 0x3ffdd48 vtable for sstables::unimplemented_backlog_tracker + 16
       126: 0x3ffdef8 vtable for seastar::shared_ptr_count_for<sstables::date_tiered_compaction_strategy> + 16
       100: 0x3e53710 vtable for seastar::httpd::str_matcher + 16
        97: 0x3e536e8 vtable for seastar::httpd::param_matcher + 16
        62: 0x41bac70 vtable for seastar::shared_ptr_count_for<cql3::restrictions::single_column_restrictions> + 16
        51: 0x4028e30 vtable for cql3::constants::marker + 16
        50: 0x3ffdd10 vtable for sstables::size_tiered_backlog_tracker + 16
        48: 0x3ffdeb8 vtable for seastar::shared_ptr_count_for<sstables::size_tiered_compaction_strategy> + 16
        42: 0x3de0358 vtable for boost::any::holder<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > + 16
        40: 0x3f342c8 vtable for seastar::shared_ptr_count_for<checked_file_impl> + 16
        38: 0x402d018 vtable for seastar::shared_ptr_count_for<cql3::cql3_type> + 16
        38: 0x3dd16b0 vtable for seastar::shared_ptr_count_for<seastar::posix_file_impl> + 16
        34: 0x3dd1810 vtable for boost::any::holder<bool> + 16
        34: 0x408b910 vtable for seastar::shared_ptr_count_for<cql3::restrictions::statement_restrictions> + 16
        33: 0x41a1a88 vtable for seastar::shared_ptr_count_for<cql3::restrictions::single_column_restriction::EQ> + 16
        28: 0x3dd17e0 vtable for boost::any::holder<unsigned int> + 16
        27: 0x41ba110 vtable for cql3::restrictions::single_column_primary_key_restrictions<partition_key> + 16

@duarten duarten self-assigned this Aug 14, 2018

@duarten

This comment has been minimized.

Member

duarten commented Aug 14, 2018

Looking more into this, I don't understand how we could have parsed the names, but not the values, given that the protocol specifies each name should precede the corresponding value.

@duarten

This comment has been minimized.

Member

duarten commented Aug 14, 2018

I wrote too soon. I now see what's going. We have two query_option ctors:

query_options::query_options(db::consistency_level consistency,
                             std::experimental::optional<std::vector<sstring_view>> names,
                             std::vector<cql3::raw_value> values,
                             bool skip_metadata,
                             specific_options options,
                             cql_serialization_format sf)
    : _consistency(consistency)
    , _names(std::move(names))
    , _values(std::move(values))
    , _value_views()
    , _skip_metadata(skip_metadata)
    , _options(std::move(options))
    , _cql_serialization_format(sf)
{
    fill_value_views();
}

query_options::query_options(db::consistency_level consistency,
                             std::experimental::optional<std::vector<sstring_view>> names,
                             std::vector<cql3::raw_value_view> value_views,
                             bool skip_metadata,
                             specific_options options,
                             cql_serialization_format sf)
    : _consistency(consistency)
    , _names(std::move(names))
    , _values()
    , _value_views(std::move(value_views))
    , _skip_metadata(skip_metadata)
    , _options(std::move(options))
    , _cql_serialization_format(sf)
{
}

One receives a std::vector<cql3::raw_value_view>, and the other a std::vector<cql3::raw_value>. transport::server::read_options calls the former, for which _values will be empty.

In query_options::prepare we try to access _values, but the authoritative data structure is _value_views.

So all bugs are on our side.

@duarten

This comment has been minimized.

Member

duarten commented Aug 14, 2018

Yet another issue is that in 2.2 we do:

void cql_server::connection::read_name_and_value_list(bytes_view& buf, std::vector<sstring_view>& names, std::vector<cql3::raw_value_view>& values) {
    uint16_t size = read_short(buf);
    names.reserve(size);
    values.reserve(size);
    for (uint16_t i = 0; i < size; i++) {
        names.emplace_back(read_string(buf)); // Creates a sstring_view for a temporary sstring
        values.emplace_back(read_value_view(buf));
    }
}

We're creating sstring_views for temporary sstrings.

This has been fixed in master, but we should fix it in 2.2 too.

@tzach tzach added this to the 2.3 milestone Aug 15, 2018

pdziepak added a commit that referenced this issue Aug 15, 2018

cql3/query_processor: Validate presence of statement values timeously
We need to validate before calling query_options::prepare() whether
the set of prepared statement values sent in the query matches the
amount of names we need to bind, otherwise we risk an out-of-bounds
access if the client also specified names together with the values.

Refs #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814225607.14215-1-duarte@scylladb.com>

avikivity added a commit that referenced this issue Aug 21, 2018

cql3/query_options: Use _value_views in prepare()
_value_views is the authoritative data structure for the
client-specified values. Indeed, the ctor called
transport::request::read_options() leaves _values completely empty.

In query_options::prepare() we were, however, using _values to
associated values to the client-specified column names, and not
_value_views. Fix this by using _value_views instead.

As for the reasons we didn't see this bug earlier, I assume it's
because very few drivers set the 0x04 query options flag, which means
column names are omitted. This is the right thing to do since most
drivers have enough information to correctly position the values.

Fixes #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814234605.14775-1-duarte@scylladb.com>
(cherry picked from commit a4355fe)

avikivity added a commit that referenced this issue Aug 21, 2018

cql3/query_options: Use _value_views in prepare()
_value_views is the authoritative data structure for the
client-specified values. Indeed, the ctor called
transport::request::read_options() leaves _values completely empty.

In query_options::prepare() we were, however, using _values to
associated values to the client-specified column names, and not
_value_views. Fix this by using _value_views instead.

As for the reasons we didn't see this bug earlier, I assume it's
because very few drivers set the 0x04 query options flag, which means
column names are omitted. This is the right thing to do since most
drivers have enough information to correctly position the values.

Fixes #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814234605.14775-1-duarte@scylladb.com>
(cherry picked from commit a4355fe)

avikivity added a commit that referenced this issue Aug 21, 2018

cql3/query_options: Use _value_views in prepare()
_value_views is the authoritative data structure for the
client-specified values. Indeed, the ctor called
transport::request::read_options() leaves _values completely empty.

In query_options::prepare() we were, however, using _values to
associated values to the client-specified column names, and not
_value_views. Fix this by using _value_views instead.

As for the reasons we didn't see this bug earlier, I assume it's
because very few drivers set the 0x04 query options flag, which means
column names are omitted. This is the right thing to do since most
drivers have enough information to correctly position the values.

Fixes #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814234605.14775-1-duarte@scylladb.com>
(cherry picked from commit a4355fe)

duarten added a commit that referenced this issue Aug 21, 2018

cql3/query_options: Use _value_views in prepare()
_value_views is the authoritative data structure for the
client-specified values. Indeed, the ctor called
transport::request::read_options() leaves _values completely empty.

In query_options::prepare() we were, however, using _values to
associated values to the client-specified column names, and not
_value_views. Fix this by using _value_views instead.

As for the reasons we didn't see this bug earlier, I assume it's
because very few drivers set the 0x04 query options flag, which means
column names are omitted. This is the right thing to do since most
drivers have enough information to correctly position the values.

Fixes #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814234605.14775-1-duarte@scylladb.com>
(cherry picked from commit a4355fe)
@withings-sas

This comment has been minimized.

withings-sas commented Sep 20, 2018

Although there is no longer a segfault in 2.3, I now get a "Invalid amount of bind variables" error, even though it is very likely correct since all our queries are working fine with Cassandra.

It seems to me that the commit 805ce6e is introducing the issue.

@duarten

This comment has been minimized.

Member

duarten commented Sep 20, 2018

Do you have a reproducer?

@duarten

This comment has been minimized.

Member

duarten commented Sep 20, 2018

Or is it still the same one?

@withings-sas

This comment has been minimized.

withings-sas commented Sep 20, 2018

I have several case, each one is 100% reproductible, example with this one:

INSERT INTO table_name (did, ts, vs) VALUES (:did, :ts, {1226535917: 10500, 1226535917: 10700, 1226538917: 10500})

We are giving the params by name:

{
  "did": <integer>,
  "ts": <bigint>
}
@withings-sas

This comment has been minimized.

withings-sas commented Sep 20, 2018

Also, it is now failing for queries that were working in 2.2, exact same code/queries

@duarten

This comment has been minimized.

Member

duarten commented Sep 20, 2018

What driver are you using?

@duarten duarten reopened this Sep 20, 2018

@withings-sas

This comment has been minimized.

withings-sas commented Sep 20, 2018

duoshuo/php-cassandra

@duarten

This comment has been minimized.

Member

duarten commented Sep 20, 2018

What commit of php-cassandra? What's the exact schema of table_name? I've failed to reproduce with duoshuo/php-cassandra@323613d and with the following schema: create table cf (p int, c bigint, v frozen<map<int, in>>, primary key (p, c));.

This is the program I've used:

<?php
require 'php-cassandra/php-cassandra.php';

$nodes = [
    '127.0.0.1'
];

$connection = new Cassandra\Connection($nodes, 'ks');

try
{
    $connection->connect();
}
catch (Cassandra\Exception $e)
{
    echo 'Caught exception: ', $e->getMessage(), "\n";
    exit;
}

$preparedData = $connection->prepare('INSERT INTO "cf" (p, c, v) VALUES (:p, :c, {1226535917: 10500, 1226535917: 10700, 1226538917: 10500})');

$strictValues = Cassandra\Request\Request::strictTypeValues(
    [
        'p' => 1,
        'c' => 0,
    ],
    $preparedData['metadata']['columns']
);

$response = $connection->executeSync(
    $preparedData['id'],
    $strictValues,
    Cassandra\Request\Request::CONSISTENCY_QUORUM,
    [
        'page_size' => 100,
        'names_for_values' => true,
        'skip_metadata' => true,
    ]
);
?>
@withings-sas

This comment has been minimized.

withings-sas commented Sep 21, 2018

It seems the issue comes from names_for_values, if I set it to false it's working, I don't know why it works on your example with it at true.

@duarten

This comment has been minimized.

Member

duarten commented Sep 21, 2018

What version/HEAD of php-cassandra are you using?

syuu1228 added a commit to syuu1228/scylla that referenced this issue Sep 22, 2018

cql3/query_processor: Validate presence of statement values timeously
We need to validate before calling query_options::prepare() whether
the set of prepared statement values sent in the query matches the
amount of names we need to bind, otherwise we risk an out-of-bounds
access if the client also specified names together with the values.

Refs scylladb#3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814225607.14215-1-duarte@scylladb.com>

syuu1228 added a commit to syuu1228/scylla that referenced this issue Sep 22, 2018

cql3/query_options: Use _value_views in prepare()
_value_views is the authoritative data structure for the
client-specified values. Indeed, the ctor called
transport::request::read_options() leaves _values completely empty.

In query_options::prepare() we were, however, using _values to
associated values to the client-specified column names, and not
_value_views. Fix this by using _value_views instead.

As for the reasons we didn't see this bug earlier, I assume it's
because very few drivers set the 0x04 query options flag, which means
column names are omitted. This is the right thing to do since most
drivers have enough information to correctly position the values.

Fixes scylladb#3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814234605.14775-1-duarte@scylladb.com>
@withings-sas

This comment has been minimized.

withings-sas commented Sep 26, 2018

we are using the latest commit on master

I guess you can close this ticket

@duarten

This comment has been minimized.

Member

duarten commented Sep 26, 2018

Well, something still seems to be off if it's not working for you. I can't figure what just from reading the code, so I'll try to repro again.

@withings-sas

This comment has been minimized.

withings-sas commented Sep 26, 2018

it seems our issues with names_for_values are mostly with batch queries, maybe you could try batch queries ?

@duarten

This comment has been minimized.

Member

duarten commented Sep 26, 2018

Will do, thanks for the tip

@duarten

This comment has been minimized.

Member

duarten commented Sep 26, 2018

When I try batch statements, I get:

PHP Fatal error:  Method Cassandra\Request\Batch::__toString() must not throw an exception, caught Cassandra\Exception: NAMES_FOR_VALUES in batch request seems never work in Cassandra 2.1.x.  Keep NAMES_FOR_VALUES flag false to avoid this bug. in /home/duarten/scylla-issues/php-cassandra/src/Request/Execute.php on line 63
@duarten

This comment has been minimized.

Member

duarten commented Sep 26, 2018

I modified php-cassandra to not throw that exception, and found an issue in our handling of batch statements (#3785). It doesn't conform to the protocol, but it turns out the protocol is not parseable, so it's understandable that even Cassandra can't support it.

Anyway, we fail before the error you said you're getting ("Invalid amount of bind variables") is reported. So I'm actually lost: how can you make batch statements work with Cassandra?

Without batch statements, prepared statements still seem to work fine with or without names_for_values.

I think the best way to move forward is if you could provide a small reproducer?

@withings-sas

This comment has been minimized.

withings-sas commented Sep 26, 2018

I think we either tweaked the library or used to use a slightly older version of php-cassandra without this exception. Doing that Batch was still working with Cassandra but one explication may be that somehow names_for_values was ignored and params were given by position.

I will try to provide a reproducer, but in the meantime you may want to close the issue since I don't know when I'll be able to do it.

@duarten

This comment has been minimized.

Member

duarten commented Sep 26, 2018

I see. Thanks. Let's leave it open in any case.

@withings-sas

This comment has been minimized.

withings-sas commented Sep 27, 2018

I do reproduce it with php-cassandra head (with just a fix in Batch.php for queryParameters() to match the definition in Request.php:

<?php
require 'lib/php-cassandra/php-cassandra.php';

// CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
// USE ks;
// CREATE TABLE cf (p int, c bigint, v frozen<map<int, int>>, primary key (p, c));

$nodes = [
	'127.0.0.1:9042',
];

$connection = new Cassandra\Connection($nodes, 'ks');

try
{
    $connection->connect();
}
catch (Cassandra\Exception $e)
{
    echo 'Caught exception: ', $e->getMessage(), "\n";
    exit;
}

$preparedData = $connection->prepare('INSERT INTO "cf" (p, c, v) VALUES (:p, :c, {1226535917: 10500, 1226535917: 10700, 1226538917: 10500})');

$strictValues = Cassandra\Request\Request::strictTypeValues(
    [
        'p' => 1,
        'c' => 0,
    ],
    $preparedData['metadata']['columns']
);

$response = $connection->executeSync(
    $preparedData['id'],
    $strictValues,
    Cassandra\Request\Request::CONSISTENCY_QUORUM,
    [
        'page_size' => 100,
        'names_for_values' => true,
        'skip_metadata' => true,
    ]
);

I get this error:
$ php test_scylla.php

PHP Fatal error:  Uncaught Cassandra\Response\Exception: Invalid: Invalid amount of bind variables in /home/scaleweb/websites/apiv2/lib/php-cassandra/src/Response/Error.php:126
Stack trace:
#0 /home/scaleweb/websites/apiv2/lib/php-cassandra/src/Connection.php(244): Cassandra\Response\Error->getException()
#1 /home/scaleweb/websites/apiv2/lib/php-cassandra/src/Connection.php(307): Cassandra\Connection->syncRequest(Object(Cassandra\Request\Execute))
#2 /home/scaleweb/websites/apiv2/test_scylla.php(37): Cassandra\Connection->executeSync('=\n\x85N\xFA\xF5[\x11\xA2\xBD\xBA}\x15\xD8\xC3...', Array, 4, Array)
#3 {main}
  thrown in /home/scaleweb/websites/apiv2/lib/php-cassandra/src/Response/Error.php on line 126

$ dpkg -l scylla*

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                                      Version                                   Architecture                              Description
+++-=========================================================================-=========================================-=========================================-========================================================================================================================================================
ii  scylla-conf                                                               2.3.0-0.20180917.178f870a0-0ubuntu1~xenia amd64                                     Scylla database main configuration file
ii  scylla-env                                                                1.3-ubuntu1~xenialppa1                    all                                       language tool for constructing recognizers, compilers etc
un  scylla-gcc72-gcc-7-base                                                   <none>                                    <none>                                    (no description available)
un  scylla-gcc72-libgcc1                                                      <none>                                    <none>                                    (no description available)
un  scylla-gcc72-libstdc++6                                                   <none>                                    <none>                                    (no description available)
ii  scylla-gcc73-gcc-7-base:amd64                                             7.3.0-3ubuntu2~xenialppa1                 amd64                                     GCC, the GNU Compiler Collection (base package)
ii  scylla-gcc73-libgcc1                                                      1:7.3.0-3ubuntu2~xenialppa1               amd64                                     GCC support library
ii  scylla-gcc73-libstdc++6                                                   7.3.0-3ubuntu2~xenialppa1                 amd64                                     GNU Standard C++ Library v3
ii  scylla-server                                                             2.3.0-0.20180917.178f870a0-0ubuntu1~xenia amd64                                     Scylla database server binaries
ii  scylla-tools                                                              2.3.0-20180917.14d640edcf-0ubuntu1~xenial all                                       Scylla database tools
ii  scylla-tools-core                                                         2.3.0-20180917.14d640edcf-0ubuntu1~xenial all                                       Scylla database tools core files
@duarten

This comment has been minimized.

Member

duarten commented Sep 27, 2018

I see what's going on. 2.3 is missing the following patches:

  1. 805ce6e
  2. 1eeef43

duarten added a commit that referenced this issue Sep 27, 2018

cql3/query_processor: Validate presence of statement values timeously
We need to validate before calling query_options::prepare() whether
the set of prepared statement values sent in the query matches the
amount of names we need to bind, otherwise we risk an out-of-bounds
access if the client also specified names together with the values.

Refs #3688

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814225607.14215-1-duarte@scylladb.com>
(cherry picked from commit 805ce6e)
@duarten

This comment has been minimized.

Member

duarten commented Sep 27, 2018

I thought they were both in 2.3 and I was testing master 🤦‍♂️ I able to repro with 2.3, and I've pushed the missing commits to branch-2.3, which will be available on the next release of 2.3 (and 3.0). Thanks for the help!

@duarten duarten closed this Sep 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment