Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request with Tuple filtering => Client and server exception #3204

Closed
nicolas-dardenne opened this issue Feb 12, 2018 · 8 comments

Comments

@nicolas-dardenne
Copy link

commented Feb 12, 2018

Installation details
Scylla version : 2.0.1-0.20171115.7b19167
Cluster size: 1 node
OS : CentOS 7.4.1708

After the join code was executed we can see the systemd-coredump process running :
GitHubErrorDemo.zip

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
108216 root 20 0 34888 1504 896 R 18,7 0,0 0:04.79 systemd-coredum
9581 scylla 20 0 16,000t 7,741g 24808 D 7,3 49,4 678:15.13 scylla

The ScyllaDB host is slow and the database is unreachable.
We have to stop and start de Syclla service :
sudo service scylla-server stop
sudo service scylla-server start

@duarten

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2018

Can you please upload scylla-server logs and the core dump? See here how to report an issue.

@nicolas-dardenne

This comment has been minimized.

Copy link
Author

commented Feb 12, 2018

I have uploaded Coredump and the file generate by the scylla shell script (node_health_check) with UUID=54f57d14-b831-45e2-ac98-0a37716969e9.
Thanks

@slivne

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2018

the code sample is very clear


// CREATE TABLE test.test_table (
//    partition_key text,
//    x text,
//    y text,
//    status boolean,
//    PRIMARY KEY (partition_key,x,y)
// )
// WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  }
// AND caching = {'keys' : 'ALL' , 'rows_per_partition' : 'NONE' };


        PreparedStatement selectStatement = session.prepare("SELECT partition_key,x,y,status FROM test_table WHERE partition_key = :partitionkey AND (x, y) IN :coords");

I don't think we support all possible queries #1437, #2473 - we have some limitations on IN queries

@nicolas-dardenne having said that - it maybe incorrect syntax of the bound value, can you please share the value of :coords in your prepared statement.

I ran the following in CQL and it did work

 SELECT * FROM test_table WHERE partition_key = 'A'  and (x,y) in (('a','b'));

 partition_key | x | y | status
---------------+---+---+--------
             A | a | b |   True

(1 rows)
@duarten

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2018

Relevant lines from the log file:

févr. 08 08:07:06 DEVDB-01-SCUS scylla[46935]:  [shard 5] seastar - Exceptional future ignored: exceptions::invalid_request_exception (No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename), backtrace: 0x4f0173, 0x4f01d1, 0xd6d291, 0x515f3f, 0x516a67, 0x4e9ee4, 0x540907, 0x542170, 0x59a5dd, 0x7f8a70cb4e24, 0x7f8a709e234c

This seems like you're trying to execute statements without specifying a keyspace.

==========

févr. 08 08:13:15 DEVDB-01-SCUS scylla[46935]:  [shard 3] trace_keyspace_helper - Tracing is enabled but system_traces.events doesn't meet expected schema: Unknown identifier scylla_parent_id

I don't know what this is, but I bet @vladzcloudius does.

==========

févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: Segmentation fault on shard 4.
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: Backtrace:
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00000000004eefe8
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00000000004ef105
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00000000004ef153
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00007f8a70cbc5df
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000001aae3de
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000001aa9799
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000001aca314
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x000000000105b033
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x000000000109789c
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000e6351f
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000e643a3
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000e092bb
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x000000000106484c
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000001064c7a
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000d6d253
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000515f3f
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000516a67
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00000000004e9ee4
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000540907
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x0000000000542170
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x000000000059a5dd
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00007f8a70cb4e24
févr. 08 08:14:38 DEVDB-01-SCUS scylla[46935]: 0x00007f8a709e234c

This decodes to

cql3::tuples::in_marker::in_marker(int, seastar::shared_ptr<cql3::column_specification>) at /usr/src/debug/scylla-2.0.1/cql3/tuples.hh:406
 (inlined by) seastar::shared_ptr<cql3::tuples::in_marker> seastar::shared_ptr_make_helper<cql3::tuples::in_marker, true>::make<int const&, seastar::shared_ptr<cql3::column_specification> >(int const&, seastar::shared_ptr<cql3::column_specification>&&) at /usr/src/debug/scylla-2.0.1/seastar/core/shared_ptr.hh:508
 (inlined by) seastar::shared_ptr<cql3::tuples::in_marker> seastar::make_shared<cql3::tuples::in_marker, int const&, seastar::shared_ptr<cql3::column_specification> >(int const&, seastar::shared_ptr<cql3::column_specification>&&) at /usr/src/debug/scylla-2.0.1/seastar/core/shared_ptr.hh:518
 (inlined by) cql3::tuples::in_raw::prepare(database&, seastar::basic_sstring<char, unsigned int, 15u> const&, std::vector<seastar::shared_ptr<cql3::column_specification>, std::allocator<seastar::shared_ptr<cql3::column_specification> > > const&) at /usr/src/debug/scylla-2.0.1/cql3/tuples.hh:339
cql3::multi_column_relation::to_term(std::vector<seastar::shared_ptr<cql3::column_specification>, std::allocator<seastar::shared_ptr<cql3::column_specification> > > const&, seastar::shared_ptr<cql3::term::raw>, database&, seastar::basic_sstring<char, unsigned int, 15u> const&, seastar::shared_ptr<cql3::variable_specifications>) at /usr/src/debug/scylla-2.0.1/cql3/multi_column_relation.hh:198
cql3::multi_column_relation::new_IN_restriction(database&, seastar::lw_shared_ptr<schema const>, seastar::shared_ptr<cql3::variable_specifications>) at /usr/src/debug/scylla-2.0.1/cql3/multi_column_relation.hh:159
cql3::relation::to_restriction(database&, seastar::lw_shared_ptr<schema const>, seastar::shared_ptr<cql3::variable_specifications>) at /usr/src/debug/scylla-2.0.1/cql3/relation.hh:154
cql3::restrictions::statement_restrictions::statement_restrictions(database&, seastar::lw_shared_ptr<schema const>, cql3::statements::statement_type, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > > const&, seastar::shared_ptr<cql3::variable_specifications>, bool, bool, bool) at /usr/src/debug/scylla-2.0.1/cql3/restrictions/statement_restrictions.cc:197
seastar::shared_ptr_count_for<cql3::restrictions::statement_restrictions>::shared_ptr_count_for<database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >, seastar::shared_ptr<cql3::variable_specifications>&, bool, bool, bool&>(database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >&&, seastar::shared_ptr<cql3::variable_specifications>&, bool&&, bool&&, bool&) at /usr/src/debug/scylla-2.0.1/seastar/core/shared_ptr.hh:337
 (inlined by) seastar::shared_ptr<cql3::restrictions::statement_restrictions> seastar::shared_ptr_make_helper<cql3::restrictions::statement_restrictions, false>::make<database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >, seastar::shared_ptr<cql3::variable_specifications>&, bool, bool, bool&>(database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >&&, seastar::shared_ptr<cql3::variable_specifications>&, bool&&, bool&&, bool&) at /usr/src/debug/scylla-2.0.1/seastar/core/shared_ptr.hh:500
 (inlined by) seastar::shared_ptr<cql3::restrictions::statement_restrictions> seastar::make_shared<cql3::restrictions::statement_restrictions, database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >, seastar::shared_ptr<cql3::variable_specifications>&, bool, bool, bool&>(database&, seastar::lw_shared_ptr<schema const>&, cql3::statements::statement_type const&, std::vector<seastar::shared_ptr<cql3::relation>, std::allocator<seastar::shared_ptr<cql3::relation> > >&&, seastar::shared_ptr<cql3::variable_specifications>&, bool&&, bool&&, bool&) at /usr/src/debug/scylla-2.0.1/seastar/core/shared_ptr.hh:518
 (inlined by) cql3::statements::raw::select_statement::prepare_restrictions(database&, seastar::lw_shared_ptr<schema const>, seastar::shared_ptr<cql3::variable_specifications>, seastar::shared_ptr<cql3::selection::selection>, bool) at /usr/src/debug/scylla-2.0.1/cql3/statements/select_statement.cc:464
cql3::statements::raw::select_statement::prepare(database&, cql3::cql_stats&, bool) at /usr/src/debug/scylla-2.0.1/cql3/statements/select_statement.cc:422
cql3::statements::raw::select_statement::prepare(database&, cql3::cql_stats&) at /usr/src/debug/scylla-2.0.1/cql3/statements/raw/select_statement.hh:105
cql3::query_processor::get_statement(std::experimental::fundamentals_v1::basic_string_view<char, std::char_traits<char> > const&, service::client_state const&) at /usr/src/debug/scylla-2.0.1/cql3/query_processor.cc:324
cql3::query_processor::prepare(std::experimental::fundamentals_v1::basic_string_view<char, std::char_traits<char> > const&, service::client_state const&, bool) at /usr/src/debug/scylla-2.0.1/cql3/query_processor.cc:217
 (inlined by) apply<cql3::query_processor::prepare(const string_view&, const service::client_state&, bool)::<lambda()> > at /usr/src/debug/scylla-2.0.1/seastar/core/future.hh:1242
 (inlined by) cql3::query_processor::prepare(std::experimental::fundamentals_v1::basic_string_view<char, std::char_traits<char> > const&, service::client_state const&, bool) at /usr/src/debug/scylla-2.0.1/cql3/query_processor.cc:224
seastar::smp_message_queue::async_work_item<cql_transport::cql_server::connection::process_prepare(unsigned short, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, service::client_state)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>::process() at /usr/src/debug/scylla-2.0.1/transport/server.cc:829
 (inlined by) do_void_futurize_apply<cql_transport::cql_server::connection::process_prepare(uint16_t, bytes_view, service::client_state)::<lambda(unsigned int)> mutable::<lambda()>&> at /usr/src/debug/scylla-2.0.1/seastar/core/future.hh:1253
 (inlined by) apply<cql_transport::cql_server::connection::process_prepare(uint16_t, bytes_view, service::client_state)::<lambda(unsigned int)> mutable::<lambda()>&> at /usr/src/debug/scylla-2.0.1/seastar/core/future.hh:1301
 (inlined by) process at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.hh:339
unsigned long seastar::smp_message_queue::process_queue<2ul, seastar::smp_message_queue::process_incoming()::{lambda(seastar::smp_message_queue::work_item*)#1}>(seastar::smp_message_queue::lf_queue&, seastar::smp_message_queue::process_incoming()::{lambda(seastar::smp_message_queue::work_item*)#1}) [clone .isra.3544] at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:3160
 (inlined by) process_queue<2ul, seastar::smp_message_queue::process_incoming()::<lambda(seastar::smp_message_queue::work_item*)> > at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:3133
seastar::smp_message_queue::process_incoming() at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:3163
 (inlined by) seastar::smp::poll_queues() at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:3749
seastar::reactor::poll_once() at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:2871
 (inlined by) operator() at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:2768
 (inlined by) _M_invoke at /opt/scylladb/include/c++/5.3.1/functional:1857
std::function<bool ()>::operator()() const at /opt/scylladb/include/c++/5.3.1/functional:2271
 (inlined by) seastar::reactor::run() at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:2793
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const [clone .constprop.4244] at /usr/src/debug/scylla-2.0.1/seastar/core/reactor.cc:3707
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/5.3.1/functional:2271
 (inlined by) seastar::posix_thread::start_routine(void*) at /usr/src/debug/scylla-2.0.1/seastar/core/posix.cc:52

I'm not sure if doing (x, y) to treat them as a tuple is valid CQL, but either way we should handle this and not crash.

@vladzcloudius

This comment has been minimized.

Copy link
Contributor

commented Feb 19, 2018

@nicolas-dardenne The tracing-related error reported in the log above is usually happens when you modify the system_traces tables or if you use the new scylla version without running fix_system_distributed_tables.py script first.

@slivne grepping files in dist it looks like we do not package this script in our RPMs. Is it intentional?

@nicolas-dardenne A simple workaround is to drop the system_traces tables (on one of the nodes) and restart this node:

From cqlsh:
> DROP TABLE system_traces.events;
> DROP TABLE system_traces.sessions;
> DROP TABLE system_traces.node_slow_log;
> exit

Restart the node.

This will re-create the tables above with the proper configuration.
I also recommend to run a repair for system_traces KS after the above procedure before using tracing.

@nicolas-dardenne This will unlikely fix the SIGSEG issue.

@slivne

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2018

@nicolas-dardenne

  • vladzcloudius provided a workaround to the system traces not using the latest schema
  • I tried to reproduce your case and asked you to check the bound values
  • duarten analyzed your coredump - and it may be related to incorrect params

Can you please update us - if this helped you, we will fix the error handling code

@slivne slivne added this to the 2.3 milestone Feb 22, 2018

@slivne

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2018

@vladzcloudius can you also try and tackle the error handling code in CQL

@duarten

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2018

We should also confirm whether that syntax is correct or not ((x,y) in (('a','b'))).

@slivne slivne modified the milestones: 2.3, 2.4 Jul 29, 2018

@slivne slivne assigned eliransin and unassigned vladzcloudius Sep 2, 2018

@slivne slivne modified the milestones: 3.0, 3.1 Sep 16, 2018

pdziepak added a commit that referenced this issue Mar 7, 2019

cql3 : fix a crash upon preparing select with an IN restriction due t…
…o memory violation

When preparing a select query with a multicolumn in restriction, the
node crashed due to using a parameter after using a move on it.

Tests:
1. UnitTests (release)
2. Preparing a select statement that crashed the system before,
and verify it is not crashing.

Fixes #3204
Fixes #3692

Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Message-Id: <7ebd210cd714a460ee5557ac612da970cee03270.1537947897.git.eliransin@scylladb.com>
(cherry picked from commit 22ad543)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.