Incorrcect error code return when consistency_level can not be met #32

slivne · 2015-07-21T07:22:11Z

Urchin returns code=0000 [Server error] Origin returns code=1000 [Unavailable exception]

create a 3 node cluster

run the following via cqlsh

CREATE KEYSPACE keyspace1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} ;

CREATE TABLE keyspace1.standard1 (
key blob PRIMARY KEY,
"C0" blob,
"C1" blob,
"C2" blob,
"C3" blob,
"C4" blob
) WITH compression = {};

stop node 3

connect via cqlsh and run

consistency all
insert into keyspace1.standard1 (key) values (0x01)

On Urchin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key,c0) values (0x01,0x02)
... ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key,C0) values (0x01,0x02) ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
<ErrorMessage code=0000 [Server error] message="Cannot achieve consistency level">
cqlsh>

On origin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
Traceback (most recent call last):
File "/home/shlomi/cassandra/bin/cqlsh", line 980, in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled)
File "/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py", line 1295, in execute
result = future.result(timeout)
File "/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py", line 2799, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={'required_replicas': 3, 'alive_replicas': 2, 'consistency': 'ALL'}

This is failing the DTEST consistency_test.py - test_simple_strategy (modified version avilable on slivne/urchin-dtest branch consistency_test_simple_strategy)
nosetests -v consistency_test.py:TestAvailability.test_simple_strategy

The text was updated successfully, but these errors were encountered:

gleb-cloudius · 2015-07-21T15:28:45Z

Currently the only information that is propagated from storage_proxy to
the code that generates replies to users is error string. We need to
define a way to propagate error code too (common exception base class
with error_code() method.

On Tue, Jul 21, 2015 at 12:22:13AM -0700, slivne wrote:

Urchin returns code=0000 [Server error] Origin returns code=1000 [Unavailable exception]

create a 3 node cluster

run the following via cqlsh

CREATE KEYSPACE keyspace1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} ;

CREATE TABLE keyspace1.standard1 (
key blob PRIMARY KEY,
"C0" blob,
"C1" blob,
"C2" blob,
"C3" blob,
"C4" blob
) WITH compression = {};

stop node 3

connect via cqlsh and run

consistency all
insert into keyspace1.standard1 (key,c0) values (0x01,0x02)

On Urchin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key,c0) values (0x01,0x02)
... ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key,C0) values (0x01,0x02) ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
<ErrorMessage code=0000 [Server error] message="Cannot achieve consistency level">
cqlsh>

On origin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
Traceback (most recent call last):
File "/home/shlomi/cassandra/bin/cqlsh", line 980, in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled)
File "/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py", line 1295, in execute
result = future.result(timeout)
File "/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py", line 2799, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={'required_replicas': 3, 'alive_replicas': 2, 'consistency': 'ALL'}

This is failing the DTEST consistency_test.py - test_simple_strategy (modified version avilable on slivne/urchin-dtest branch consistency_test_simple_strategy)
nosetests -v consistency_test.py:TestAvailability.test_simple_strategy

Reply to this email directly or view it on GitHub:
#32

        Gleb.

tgrabiec · 2015-07-21T16:12:09Z

2015-07-21 17:28 GMT+02:00 Gleb Natapov notifications@github.com:

Currently the only information that is propagated from storage_proxy to
the code that generates replies to users is error string. We need to
define a way to propagate error code too (common exception base class
with error_code() method.

There is already such a base class, see cassandra_exception in
exceptions.hh. We converted it from Origin.

Origin throws UnavailableException which passes error code in its
constructor to base classes, eventually to CassandraException.

On Tue, Jul 21, 2015 at 12:22:13AM -0700, slivne wrote:

Urchin returns code=0000 [Server error] Origin returns code=1000 [Unavailable exception]

create a 3 node cluster

run the following via cqlsh

CREATE KEYSPACE keyspace1 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} ;

CREATE TABLE keyspace1.standard1 (
key blob PRIMARY KEY,
"C0" blob,
"C1" blob,
"C2" blob,
"C3" blob,
"C4" blob
) WITH compression = {};

stop node 3

connect via cqlsh and run

consistency all
insert into keyspace1.standard1 (key,c0) values (0x01,0x02)

On Urchin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key,c0) values (0x01,0x02)
... ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key,C0) values (0x01,0x02) ;
code=2200 [Invalid query] message="Unknown identifier c0"
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
<ErrorMessage code=0000 [Server error] message="Cannot achieve
consistency level">
cqlsh>

On origin you get

cqlsh> consistency all
Consistency level set to ALL.
cqlsh> insert into keyspace1.standard1 (key) values (0x01) ;
Traceback (most recent call last):
File "/home/shlomi/cassandra/bin/cqlsh", line 980, in
perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled)
File
"/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py",
line 1295, in execute
result = future.result(timeout)
File
"/home/shlomi/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.3.zip/cassandra-driver-2.1.3/cassandra/cluster.py",
line 2799, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve
consistency level ALL" info={'required_replicas': 3, 'alive_replicas': 2,
'consistency': 'ALL'}

This is failing the DTEST consistency_test.py - test_simple_strategy
(modified version avilable on slivne/urchin-dtest branch
consistency_test_simple_strategy)
nosetests -v consistency_test.py:TestAvailability.test_simple_strategy

Reply to this email directly or view it on GitHub:
#32

Gleb.

—
Reply to this email directly or view it on GitHub
#32 (comment)
.

slivne · 2015-07-25T20:25:33Z

following commit

[COMMIT urchin master] storage_proxy: return proper error codes to transport layer

the problem is still not resolved the issue may be related to #43

shlomi@localhost~/urchin-dtest (add_range_queries)$  nosetests -v consistency_test.py:TestAvailability.test_simple_strategy
test_simple_strategy (consistency_test.TestAvailability) ... ERROR

======================================================================
ERROR: test_simple_strategy (consistency_test.TestAvailability)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomi/urchin-dtest/dtest.py", line 250, in setUp
    self.cluster = self._get_cluster()
  File "/home/shlomi/urchin-dtest/dtest.py", line 152, in _get_cluster
    cluster = Cluster(self.test_path, name, cassandra_dir=cdir)
  File "/home/shlomi/urchin-ccm/ccmlib/cluster.py", line 52, in __init__
    self.__version = self.__get_version_from_build()
  File "/home/shlomi/urchin-ccm/ccmlib/cluster.py", line 429, in __get_version_from_build
    return common.get_version_from_build(self.get_install_dir())
  File "/home/shlomi/urchin-ccm/ccmlib/cluster.py", line 94, in get_install_dir
    common.validate_install_dir(self.__install_dir)
  File "/home/shlomi/urchin-ccm/ccmlib/common.py", line 344, in validate_install_dir
    elif isDse(install_dir):
  File "/home/shlomi/urchin-ccm/ccmlib/common.py", line 302, in isDse
    raise ArgumentError('Installation directory does not contain a bin directory: %s' % install_dir)
ArgumentError: Installation directory does not contain a bin directory: /home/shlomi/urchin-dtest
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-uL1G6f
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 0.004s

FAILED (errors=1)
shlomi@localhost~/urchin-dtest (add_range_queries)$ export CASSANDRA_DIR=~/urchin
shlomi@localhost~/urchin-dtest (add_range_queries)$  nosetests -v consistency_test.py:TestAvailability.test_simple_strategy
test_simple_strategy (consistency_test.TestAvailability) ... FAIL

======================================================================
FAIL: test_simple_strategy (consistency_test.TestAvailability)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomi/urchin-dtest/consistency_test.py", line 281, in test_simple_strategy
    self._test_simple_strategy(combinations)
  File "/home/shlomi/urchin-dtest/consistency_test.py", line 198, in _test_simple_strategy
    self._test_insert_query_from_node(session, 0, [rf], [num_alive], *combination)
  File "/home/shlomi/urchin-dtest/consistency_test.py", line 244, in _test_insert_query_from_node
    assert_unavailable(self.insert_user, session, end, age, write_cl, serial_cl)
  File "/home/shlomi/urchin-dtest/assertions.py", line 15, in assert_unavailable
    assert False, "Expecting unavailable exception, got: " + str(e)
AssertionError: Expecting unavailable exception, got: unpack requires a string argument of length 2
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-Iq0w4t
cassandra.cluster: WARNING: Excluding host (127.0.0.3) with no tokens in system.peers table of 127.0.0.1.
cassandra.cluster: WARNING: Excluding host (127.0.0.2) with no tokens in system.peers table of 127.0.0.1.
dtest: DEBUG: Testing node 0 in single dc with 3 nodes alive
cassandra.cluster: WARNING: Excluding host (127.0.0.3) with no tokens in system.peers table of 127.0.0.1.
cassandra.cluster: WARNING: Excluding host (127.0.0.2) with no tokens in system.peers table of 127.0.0.1.
dtest: DEBUG: Connected to ['127.0.0.1'] for ALL/ALL/-
dtest: DEBUG: Connected to ['127.0.0.1'] for QUORUM/QUORUM/-
dtest: DEBUG: Connected to ['127.0.0.1'] for LOCAL_QUORUM/LOCAL_QUORUM/-
dtest: DEBUG: Connected to ['127.0.0.1'] for EACH_QUORUM/LOCAL_QUORUM/-
dtest: DEBUG: Connected to ['127.0.0.1'] for ONE/ONE/-
dtest: DEBUG: Connected to ['127.0.0.1'] for ONE/ALL/-
dtest: DEBUG: Connected to ['127.0.0.1'] for ALL/ONE/-
dtest: DEBUG: Connected to ['127.0.0.1'] for QUORUM/TWO/-
dtest: DEBUG: Connected to ['127.0.0.1'] for QUORUM/THREE/-
dtest: DEBUG: Connected to ['127.0.0.1'] for TWO/TWO/-
dtest: DEBUG: Connected to ['127.0.0.1'] for THREE/ONE/-
dtest: DEBUG: Connected to ['127.0.0.1'] for ANY/ONE/-
dtest: DEBUG: Connected to ['127.0.0.1'] for LOCAL_ONE/LOCAL_ONE/-
dtest: DEBUG: Testing node 1 in single dc with 2 nodes alive
cassandra.cluster: WARNING: Excluding host (127.0.0.3) with no tokens in system.peers table of 127.0.0.2.
cassandra.cluster: WARNING: Excluding host (127.0.0.1) with no tokens in system.peers table of 127.0.0.2.
dtest: DEBUG: Connected to ['127.0.0.2'] for ALL/ALL/-
cassandra.connection: ERROR: Error decoding response from Cassandra. opcode: 0000; message contents: '\x83\x00\x00\x03\x00\x00\x00\x005\x00\x00\x11\x00\x00/Mutation write timeout: waited for 3 got 2 acks'
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cassandra/connection.py", line 431, in process_msg
    flags, opcode, body, self.decompressor)
  File "/usr/lib64/python2.7/site-packages/cassandra/protocol.py", line 123, in decode_response
    msg = msg_class.recv_body(body, protocol_version, user_type_map)
  File "/usr/lib64/python2.7/site-packages/cassandra/protocol.py", line 147, in recv_body
    extra_info = subcls.recv_error_info(f)
  File "/usr/lib64/python2.7/site-packages/cassandra/protocol.py", line 239, in recv_error_info
    'consistency': read_consistency_level(f),
  File "/usr/lib64/python2.7/site-packages/cassandra/protocol.py", line 875, in read_consistency_level
    return read_short(f)
  File "/usr/lib64/python2.7/site-packages/cassandra/protocol.py", line 867, in read_short
    return uint16_unpack(f.read(2))
  File "/usr/lib64/python2.7/site-packages/cassandra/marshal.py", line 22, in <lambda>
    unpack = lambda s: packer.unpack(s)[0]
error: unpack requires a string argument of length 2
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 88.188s

FAILED (failures=1)
shlomi@localhost~/urchin-dtest (add_ran

penberg · 2015-07-27T15:16:43Z

@slivne, hopefully fixed by commit 584b70a.

slivne · 2015-07-27T19:35:40Z

no it does not solve the issue

using
commit 584b70a
Merge: 2e745be a4225fa
Author: Avi Kivity avi@cloudius-systems.com
Date: Mon Jul 27 17:59:39 2015 +0300

======================================================================
FAIL: test_simple_strategy (consistency_test_modified.TestAvailability)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 281, in test_simple_strategy
    self._test_simple_strategy(combinations)
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 198, in _test_simple_strategy
    self._test_insert_query_from_node(session, 0, [rf], [num_alive], *combination)
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 244, in _test_insert_query_from_node
    assert_unavailable(self.insert_user, session, end, age, write_cl, serial_cl)
  File "/home/shlomi/urchin-dtest/assertions.py", line 15, in assert_unavailable
    assert False, "Expecting unavailable exception, got: " + str(e)
AssertionError: Expecting unavailable exception, got: unpack requires a string argument of length 2
-------------------- >> begin captured logging << --------------------

It may be related to the encoding error we have #43

penberg · 2015-07-28T05:41:07Z

@slivne, that's a problem with UNAVAILABLE_ERROR CQL transport encoding. I'll send a patch.

penberg · 2015-07-28T06:57:41Z

@slivne, should be fixed by commit 826f216.

slivne · 2015-07-28T07:26:37Z

@penberg

still does not work

head at:

shlomi@localhost~/urchin (master)$ git log
commit 826f216
Author: Pekka Enberg penberg@cloudius-systems.com
Date: Tue Jul 28 09:18:09 2015 +0300

transport/server: Fix UNAVAILABLE error encoding

This fixes UNAVAILABLE error encoding to follow the CQL binary protocol
spec.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

still getting

======================================================================
FAIL: test_simple_strategy (consistency_test_modified.TestAvailability)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 281, in test_simple_strategy
    self._test_simple_strategy(combinations)
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 198, in _test_simple_strategy
    self._test_insert_query_from_node(session, 0, [rf], [num_alive], *combination)
  File "/home/shlomi/urchin-dtest/consistency_test_modified.py", line 244, in _test_insert_query_from_node
    assert_unavailable(self.insert_user, session, end, age, write_cl, serial_cl)
  File "/home/shlomi/urchin-dtest/assertions.py", line 15, in assert_unavailable
    assert False, "Expecting unavailable exception, got: " + str(e)
AssertionError: Expecting unavailable exception, got: unpack requires a string argument of length 2
-------------------- >> begin captured logging << --------------------

I created a very simple (incorrect) dtest that will always fail - that allows you to reproduce the issue on your own:

from dtest import Tester, debug
from tools import since
import subprocess, tempfile, os, shutil
import time
from ccmlib.urchin_cluster import UrchinCluster
from cassandra.query import SimpleStatement
from cassandra import Unavailable,ConsistencyLevel

class TestSimple(Tester):

    __urchin_args__=[]

    def __init__(self, *args, **kwargs):
        Tester.__init__(self, *args, **kwargs)

    def prepare(self):
        """
        Sets up cluster to test against.
        """
        cluster = self.cluster
        return cluster


    def simple_create_insert_select_test(self):
        cluster = self.prepare()
        jvm_args=[]
        if type(cluster) is UrchinCluster:
           jvm_args=self.__urchin_args__
        cluster.populate(1).start(jvm_args=jvm_args)
        node1 = cluster.nodelist()[0]
        session = self.patient_cql_connection(node1)
        self.create_ks(session, 'ks', 3)

        session.execute("""
            CREATE TABLE test1 (
                k int PRIMARY KEY,
                c int
            )
        """)

        session.execute("insert into test1  (k,c) values (1,2);")

        # Select
        query = SimpleStatement("SELECT * FROM ks.test1 WHERE k=1",consistency_level=ConsistencyLevel.QUORUM)
        res = session.execute(query)

        time.sleep(10)

penberg · 2015-07-28T09:38:46Z

Your test case doesn't use the proper assertion:

        query = SimpleStatement("SELECT * FROM ks.test1 WHERE k=1",consistency_level=ConsistencyLevel.QUORUM)
        assert_unavailable(session.execute, query)

Anyway, I can reproduce. Will send a fix.

@fruch

The patch introduces the _partition_is_open flag, inject partition_end only if there was some data in the input reader. A simple unit test has been added for the nonforwardable reader which checks this new behaviour. Merge "Cqlsh serverless v2" from Karol Baryla This PR adds serverless support for CQLSH. It was previosuly merged, but dtests later discovered a bug in python driver. The bug should be fixed now, so let's merge this again. @fruch As you had more luck reproducing the issue with dtests than us, could you try and run them with this PR to make sure it works now? Fixes scylladb#317 (again) Closes scylladb#323 * github.com:scylladb/scylla-tools-java: Add Scylla Cloud serverless support Switch cqlsh to use scylla-driver install.sh: drop locale workaround from python3 thunk Since scylladb#7408 does not occur on current python3 version (3.11.0), let's drop the workarond. Closes scylladb#32 tls: add missing include <map> std::multimap is later used in the file. Closes scylladb#1314

slivne added the type/bug label Jul 28, 2015

slivne assigned penberg Jul 28, 2015

slivne added this to the Beta milestone Jul 28, 2015

avikivity closed this as completed in c30afdc Jul 28, 2015

slivne mentioned this issue Aug 10, 2015

SIGSEGV during commitlog write when low on memory (fragmentation problem) #108

Closed

slivne unassigned penberg Sep 22, 2015

tzach mentioned this issue Oct 7, 2015

nodetool drain #442

Closed

nirmaayan mentioned this issue Aug 1, 2016

Scylla Crash When Running Write Load #1510

Closed

slivne mentioned this issue Aug 8, 2016

SIGSEGV in scollectd ? #1558

Closed

This was referenced Sep 26, 2016

Assertion failure in memory allocator with large partitions while compacting #1589

Closed

Failure during compaction. #1706

Closed

GpStore mentioned this issue Nov 10, 2016

Scylla crashes during query #1831

Closed

slivne mentioned this issue Jul 3, 2017

Row Cache continuity information is not used for queries of partitions that do not exist (bloom filters are still checked) #2544

Closed

frank8989 mentioned this issue Mar 13, 2018

Reactor stalled for 2029 ms on shard for scylla 2.0.0 #3240

Closed

tgrabiec mentioned this issue Sep 24, 2019

Coredumps during restart_then_repair_node nemesis #4918

Closed

1 task

ChrisHampu mentioned this issue Mar 24, 2021

cassandra-stress starts to timeout and fail after ~20min with GCE local disks #7341

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrcect error code return when consistency_level can not be met #32

Incorrcect error code return when consistency_level can not be met #32

slivne commented Jul 21, 2015

gleb-cloudius commented Jul 21, 2015

tgrabiec commented Jul 21, 2015

slivne commented Jul 25, 2015

penberg commented Jul 27, 2015

slivne commented Jul 27, 2015

penberg commented Jul 28, 2015

penberg commented Jul 28, 2015

slivne commented Jul 28, 2015

penberg commented Jul 28, 2015

Incorrcect error code return when consistency_level can not be met #32

Incorrcect error code return when consistency_level can not be met #32

Comments

slivne commented Jul 21, 2015

gleb-cloudius commented Jul 21, 2015

tgrabiec commented Jul 21, 2015

slivne commented Jul 25, 2015

penberg commented Jul 27, 2015

slivne commented Jul 27, 2015

penberg commented Jul 28, 2015

penberg commented Jul 28, 2015

slivne commented Jul 28, 2015

penberg commented Jul 28, 2015