cqlsh times out in test_copy_to_with_child_process_crashing dtest #37

bhalevy · 2023-05-16T06:46:25Z

See for example https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-daily-release/249/testReport/cqlsh_tests.cqlsh_copy_tests/TestCqlshCopy/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split000___test_copy_to_with_child_process_crashing/

self = <cqlsh_tests.cqlsh_copy_tests.TestCqlshCopy object at 0x7fcd74a51490>

    @pytest.mark.single_node
    def test_copy_to_with_child_process_crashing(self):
        """
        Test exporting rows with failure injection by setting the environment variable CQLSH_COPY_TEST_FAILURES,
        which is used by ExportProcess in pylib/copy.py to deviate its behavior from performing normal queries.
        Here we set a token range that will cause a child process processing this range to exit, therefore
        we expect this COPY TO job to fail.
    
        @jira_ticket CASSANDRA-9304
        """
        num_records = 100000
        self.prepare(nodes=1)
    
        logger.debug('Running stress')
        stress_table = 'keyspace1.standard1'
        self.node1.stress(['write', 'n={}'.format(num_records), '-rate', 'threads=50'])
    
        self.tempfile = NamedTemporaryFile(mode='w+', delete=False, encoding='utf-8')
        failures = {'exit_range': {'start': 0, 'end': 5000000000000000000}}
        os.environ['CQLSH_COPY_TEST_FAILURES'] = json.dumps(failures)
    
        logger.debug(f'Exporting to csv file: {self.tempfile.name} with {os.environ["CQLSH_COPY_TEST_FAILURES"]}')
>       out, err = self.node1.run_cqlsh(cmds="COPY {} TO '{}'"
                                        .format(stress_table, self.tempfile.name),
                                        return_output=True)

cqlsh_tests/cqlsh_copy_tests.py:1526: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../scylla/.local/lib/python3.11/site-packages/ccmlib/node.py:892: in run_cqlsh
    output = p.communicate(timeout=timeout)
/usr/lib64/python3.11/subprocess.py:1207: in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib64/python3.11/subprocess.py:2076: in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: None args: ['/jenkins/workspace/scylla-master/dtest-dail...>
endtime = 5180.818233196, orig_timeout = 600
stdout_seq = [b'Using 16 child processes\n', b'\nStarting copy of keyspace1.standard1 with columns [key, C0, C1, C2, C3, C4].\n', b'OSError: handle is closed', b'\n', b'OSError: handle is closed', b'\n', ...]
stderr_seq = [], skip_check_and_raise = False

    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
                       skip_check_and_raise=False):
        """Convenience for checking if a timeout has expired."""
        if endtime is None:
            return
        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,
                    stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['/jenkins/workspace/scylla-master/dtest-daily-release/scylla/.ccm/scylla-repository/b112a3b78a4fa8b92216b3ec9920270100e40a85/share/cassandra/bin/cqlsh', '127.0.55.1', '9042']' timed out after 600 seconds

/usr/lib64/python3.11/subprocess.py:1251: TimeoutExpired

This might be related to the error injection that is triggered in this test.

The text was updated successfully, but these errors were encountered:

fruch · 2023-06-15T11:29:06Z

Seen a couple of times, never could reproduce it locally

bhalevy · 2023-08-17T05:46:33Z

Happened again:
https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-release/334/testReport/cqlsh_tests.cqlsh_copy_tests/TestCqlshCopy/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split000___test_copy_to_with_child_process_crashing/

self = <cqlsh_tests.cqlsh_copy_tests.TestCqlshCopy object at 0x7f905d0d95d0>

    @pytest.mark.single_node
    # Test had history of timing out in debug, see: https://github.com/scylladb/scylla-dtest/issues/3275
    @pytest.mark.scylla_mode('!debug')
    @unmark.next_gating  # unmark cause of: https://github.com/scylladb/scylla-cqlsh/issues/37
    def test_copy_to_with_child_process_crashing(self):
        """
        Test exporting rows with failure injection by setting the environment variable CQLSH_COPY_TEST_FAILURES,
        which is used by ExportProcess in pylib/copy.py to deviate its behavior from performing normal queries.
        Here we set a token range that will cause a child process processing this range to exit, therefore
        we expect this COPY TO job to fail.
    
        @jira_ticket CASSANDRA-9304
        """
        num_records = 100000
        self.prepare(nodes=1)
    
        logger.debug('Running stress')
        stress_table = 'keyspace1.standard1'
        self.node1.stress(['write', 'n={}'.format(num_records), '-rate', 'threads=50'])
    
        self.tempfile = NamedTemporaryFile(mode='w+', delete=False, encoding='utf-8')
        failures = {'exit_range': {'start': 0, 'end': 5000000000000000000}}
        os.environ['CQLSH_COPY_TEST_FAILURES'] = json.dumps(failures)
    
        logger.debug(f'Exporting to csv file: {self.tempfile.name} with {os.environ["CQLSH_COPY_TEST_FAILURES"]}')
>       out, err = self.node1.run_cqlsh(cmds="COPY {} TO '{}'"
                                        .format(stress_table, self.tempfile.name),
                                        return_output=True)

cqlsh_tests/cqlsh_copy_tests.py:1530: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../scylla/.local/lib/python3.11/site-packages/ccmlib/node.py:892: in run_cqlsh
    output = p.communicate(timeout=timeout)
/usr/lib64/python3.11/subprocess.py:1209: in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib64/python3.11/subprocess.py:2109: in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: None args: ['/jenkins/workspace/scylla-master/dtest-rele...>
endtime = 4609.906105098, orig_timeout = 600
stdout_seq = [b'Using 16 child processes\n', b'\nStarting copy of keyspace1.standard1 with columns [key, C0, C1, C2, C3, C4].\n', b...ile descriptor\n', b'OSError: handle is closed\nOSError: handle is closed\n', b'OSError: handle is closed', b'\n', ...]
stderr_seq = [], skip_check_and_raise = False

    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
                       skip_check_and_raise=False):
        """Convenience for checking if a timeout has expired."""
        if endtime is None:
            return
        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,
                    stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/d57a951d484bec775ffe82e82815205f533dadb8/share/cassandra/bin/cqlsh', '127.0.67.1', '9042']' timed out after 600 seconds

/usr/lib64/python3.11/subprocess.py:1253: TimeoutExpired

I wonder if those OSError: handle is closed messages in stdout_seq are normal and benign considering the error injection or are they related to the timeout.

bhalevy · 2023-08-17T06:33:48Z

FWIW, I saw this locally after adding some debug printouts:

06:23:53,032 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> Using 16 child processes
06:23:53,032 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> 
06:23:53,032 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> Starting copy of keyspace1.standard1 with columns [key, C0, C1, C2, C3, C4].
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> OSError: handle is closed
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> OSError: handle is closed
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> Processed: 380 rows; Rate:    5359 rows/s; Avg. rate:    5358 rows/s
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> 380 rows exported to 1 files in 0.086 seconds.
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> 
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> <stdin>:2:Child process 529 died with exit code 1
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> <stdin>:2:Child process 530 died with exit code 1
06:23:53,033 ccm             ERROR cluster.py     :718  | node1: (EE) <stdout> <stdin>:2:Exported 6 ranges out of 257 total ranges, some records might be missing

To help understand spurious timeout error we encounter. Refs scylladb/scylla-cqlsh#37 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

bhalevy · 2023-08-31T05:39:18Z

Hi again, now with printing stdout to the log.

https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-release/346/artifact/logs-full.release.000/dtest-gw1.log

04:54:10,322 784     dtest_setup                    DEBUG    dtest_setup.py      :59   | test_copy_to_with_child_process_crashing: Allocated cluster ID 6: /jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-hkpt7y0u
04:54:10,370 784     ccm                            DEBUG    cluster.py          :709  | test_copy_to_with_child_process_crashing: start_nodes: no_wait=False wait_for_binary_proto=True wait_other_notice=True force_wait_for_cluster_start=True
04:54:10,373 784     ccm                            DEBUG    cluster.py          :709  | test_copy_to_with_child_process_crashing: node1: Starting scylla: args=['/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-hkpt7y0u/test/node1/bin/scylla', '--options-file', '/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-hkpt7y0u/test/node1/conf/scylla.yaml', '--log-to-stdout', '1', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--api-address', '127.0.6.1', '--collectd-hostname', 'ip-10-0-4-201.ec2.internal.node1', '--smp', '2', '--memory', '1024M', '--developer-mode', 'true', '--default-log-level', 'info', '--collectd', '0', '--overprovisioned', '--prometheus-address', '127.0.6.1', '--unsafe-bypass-fsync', '1', '--kernel-page-cache', '1', '--commitlog-use-o-dsync', '0', '--max-networking-io-control-blocks', '1000'] wait_other_notice=True wait_for_binary_proto=True
04:54:10,668 784     ccm                            DEBUG    cluster.py          :709  | test_copy_to_with_child_process_crashing: node1: Starting scylla-jmx: args=['/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-hkpt7y0u/test/node1/bin/symlinks/scylla-jmx', '-Dapiaddress=127.0.6.1', '-Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder', '-Djava.rmi.server.hostname=127.0.6.1', '-Dcom.sun.management.jmxremote', '-Dcom.sun.management.jmxremote.host=127.0.6.1', '-Dcom.sun.management.jmxremote.port=7199', '-Dcom.sun.management.jmxremote.rmi.port=7199', '-Dcom.sun.management.jmxremote.local.only=false', '-Xmx256m', '-XX:+UseSerialGC', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', '-jar', '/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-hkpt7y0u/test/node1/bin/scylla-jmx-1.0.jar']
04:54:11,076 784     cassandra.policies             INFO     policies.py         :289  | test_copy_to_with_child_process_crashing: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.6.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
04:54:11,090 784     cqlsh_tests.cqlsh_copy_tests   DEBUG    cqlsh_copy_tests.py :1521 | test_copy_to_with_child_process_crashing: Running stress
04:54:18,878 784     cqlsh_tests.cqlsh_copy_tests   DEBUG    cqlsh_copy_tests.py :1529 | test_copy_to_with_child_process_crashing: Exporting to csv file: /tmp/tmp4f1dng5a with {"exit_range": {"start": 0, "end": 5000000000000000000}}
05:04:18,979 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) Command '['/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/3e7ec6cc83da5a31420b79c3c4af8bd2e2638f60/share/cassandra/bin/cqlsh', '127.0.6.1', '9042']' timed out after 600 seconds
05:04:18,979 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> Using 16 child processes
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> 
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> Starting copy of keyspace1.standard1 with columns [key, C0, C1, C2, C3, C4].
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> TypeError: 'NoneType' object cannot be interpreted as an integer
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> OSError: handle is closed
05:04:18,980 784     ccm                            ERROR    cluster.py          :718  | test_copy_to_with_child_process_crashing: node1: (EE) <stdout> 
05:04:19,314 784     errors                         ERROR    conftest.py         :208  | test_copy_to_with_child_process_crashing: test failed: 
...
self = <Popen: returncode: None args: ['/jenkins/workspace/scylla-master/dtest-rele...>
endtime = 4523.978945526, orig_timeout = 600
stdout_seq = [b'Using 16 child processes\n', b'\nStarting copy of keyspace1.standard1 with columns [key, C0, C1, C2, C3, C4].\n', b'OSError: handle is closed\n', b'OSError: handle is closed\n', b'OSError: handle is closed', b'\n', ...]
stderr_seq = [], skip_check_and_raise = False

    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
                       skip_check_and_raise=False):
        """Convenience for checking if a timeout has expired."""
        if endtime is None:
            return
        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,
                    stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/3e7ec6cc83da5a31420b79c3c4af8bd2e2638f60/share/cassandra/bin/cqlsh', '127.0.6.1', '9042']' timed out after 600 seconds

fruch · 2023-08-31T06:52:16Z

this is unique, compare to the others:

TypeError: 'NoneType' object cannot be interpreted as an integer

question now from which part of the code it's coming

fruch · 2023-08-31T07:14:00Z

too bad we didn't had the full backtrace of that error, seem like we would if we used --debug
I'll add that into the test

bhalevy · 2023-08-31T07:14:05Z

this is unique, compare to the others:
TypeError: 'NoneType' object cannot be interpreted as an integer
question now from which part of the code it's coming

I wish it had the full bracktrace

fruch · 2023-08-31T07:14:21Z

this is unique, compare to the others:
TypeError: 'NoneType' object cannot be interpreted as an integer
question now from which part of the code it's coming
I wish it had the full bracktrace

LOL

fruch · 2023-08-31T08:07:39Z

https://github.com/scylladb/scylla-dtest/pull/3538 might help a bit further

fruch · 2023-08-31T18:16:01Z

with the following command lines args --scylla-version=unstable/master:latest --log-cli-level=debug -n 5 --count=15
it's reproduces locally 1-2 to 15 runs.

fruch · 2023-08-31T19:27:37Z

after closing digging into it seems like recv_select is getting to a point it's doing recv() call on a pipe and never return.
Pipe.recv() calls doesn't have timeout, and seems like the other end of the process exited during the recv()

one option would be to replace the whole pipelines with multiprocessing.Queue which might work better for this purpose

fruch · 2023-08-31T19:40:36Z

looks like all those pipes was introduced in 54a7cf9
for solving some windows issue: https://issues.apache.org/jira/browse/CASSANDRA-11701

I'll try to revert it, seeing if it would behave better.

fruch · 2023-08-31T19:58:28Z

looks like all those pipes was introduced in 54a7cf9 for solving some windows issue: https://issues.apache.org/jira/browse/CASSANDRA-11701

I'll try to revert it, seeing if it would behave better.

And the answer is no, it's exactly the same problem.

seems like I didn't went all the way into what the pipes where introduced, and that's in https://issues.apache.org/jira/browse/CASSANDRA-11053

CopyTask is is using ReceivingChannels and SendingChannels which are a list of pipes created by CopyTask and those pipes are being send to a list of subprocesses so the main task can communicate with them. in one dtest that delibratily make once of those child process to break and exit, from time to time we see it getting stuck forever. the reason is the CopyTask process is hanging on `recv` call on one of those pipes, since pipes are copy into the child processes there's one fd open on CopyTask and one fd open on child process when the child process closes the fd, `recv()` doesn't raise EOF since there an open fd that might still send in data. so we need to close the local pipes on CopyTask after all child processes are started Fixes: scylladb#37

fruch · 2023-09-12T20:10:30Z

After banging my head on it again and again, I've found the root cause...

fruch · 2023-09-12T20:10:48Z

and that's the fix: #49

CopyTask is is using ReceivingChannels and SendingChannels which are a list of pipes created by CopyTask and those pipes are being send to a list of subprocesses so the main task can communicate with them. in one dtest that delibratily make once of those child process to break and exit, from time to time we see it getting stuck forever. the reason is the CopyTask process is hanging on `recv` call on one of those pipes, since pipes are copy into the child processes there's one fd open on CopyTask and one fd open on child process when the child process closes the fd, `recv()` doesn't raise EOF since there an open fd that might still send in data. so we need to close the local pipes on CopyTask after all child processes are started Fixes: #37

* tools/cqlsh 66ae7eac...426fa0ea (8): > Updated Scylla Driver[Issue scylladb/scylla-cqlsh#55] > copyutil: closing the local end of pipes after processes starts > setup.py: specify Cython language_level explicitly > setup.py: pass extensions as a list > setup.py: reindent block in else branch > setup.py: early return in get_extension() > reloc: install build==0.10.0 > reloc: add --verbose option to build_reloc.sh Fixes: scylladb/scylla-cqlsh#37

* tools/cqlsh 66ae7eac...426fa0ea (8): > Updated Scylla Driver[Issue scylladb/scylla-cqlsh#55] > copyutil: closing the local end of pipes after processes starts > setup.py: specify Cython language_level explicitly > setup.py: pass extensions as a list > setup.py: reindent block in else branch > setup.py: early return in get_extension() > reloc: install build==0.10.0 > reloc: add --verbose option to build_reloc.sh Fixes: scylladb/scylla-cqlsh#37 Closes #15685

To help understand spurious timeout error we encounter. Refs scylladb/scylla-cqlsh#37 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

bhalevy mentioned this issue May 16, 2023

node: run_cqlsh: add timeout parameter scylladb/scylla-ccm#452

Merged

bhalevy changed the title ~~cqlhs timesout in test_copy_to_with_child_process_crashing dtest~~ cqlhs times out in test_copy_to_with_child_process_crashing dtest May 16, 2023

bhalevy changed the title ~~cqlhs times out in test_copy_to_with_child_process_crashing dtest~~ cqlsh times out in test_copy_to_with_child_process_crashing dtest Jul 6, 2023

bhalevy added a commit to bhalevy/scylla-ccm that referenced this issue Aug 17, 2023

node: run_cqlsh: dump tool's stdout and stderr on timeout

fa2369a

To help understand spurious timeout error we encounter. Refs scylladb/scylla-cqlsh#37 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

bhalevy mentioned this issue Aug 17, 2023

node: run_cqlsh: dump tool's stdout and stderr on timeout scylladb/scylla-ccm#498

Merged

fruch pushed a commit to scylladb/scylla-ccm that referenced this issue Aug 27, 2023

node: run_cqlsh: dump tool's stdout and stderr on timeout

21515bf

To help understand spurious timeout error we encounter. Refs scylladb/scylla-cqlsh#37 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

fruch mentioned this issue Sep 12, 2023

copyutil: closing the local end of pipes after processes starts #49

Merged

fruch closed this as completed in #49 Sep 21, 2023

fruch mentioned this issue Oct 10, 2023

Update tools/cqlsh submodule scylladb/scylladb#15685

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cqlsh times out in test_copy_to_with_child_process_crashing dtest #37

cqlsh times out in test_copy_to_with_child_process_crashing dtest #37

bhalevy commented May 16, 2023

fruch commented Jun 15, 2023

bhalevy commented Aug 17, 2023

bhalevy commented Aug 17, 2023

bhalevy commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

bhalevy commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Sep 12, 2023

fruch commented Sep 12, 2023

cqlsh times out in test_copy_to_with_child_process_crashing dtest #37

cqlsh times out in test_copy_to_with_child_process_crashing dtest #37

Comments

bhalevy commented May 16, 2023

fruch commented Jun 15, 2023

bhalevy commented Aug 17, 2023

bhalevy commented Aug 17, 2023

bhalevy commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

bhalevy commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Aug 31, 2023

fruch commented Sep 12, 2023

fruch commented Sep 12, 2023