Session might not reconnect if all nodes are restarted at once #230
Labels
bug
Something isn't working
Comments
|
I wonder if it's the same thing as #170 - which was supposedly fixed - or maybe it's some different kind of race which only happens when we restart all nodes? |
piodul
added a commit
to piodul/scylla
that referenced
this issue
Jun 16, 2023
The test `test_downgrade_after_successful_upgrade_fails` stops all nodes, reconfigures them to support the test-only feature and restarts them. Unfortunately, it looks like python driver sometimes does not handle this properly and might not reconnect after all nodes are shut down. This commit adds a workaround for scylladb/python-driver#230 - the test re-creates python driver session right after nodes are restarted.
kbr-scylla
added a commit
to scylladb/scylladb
that referenced
this issue
Jun 16, 2023
…ver not reconnecting after full cluster restart' from Piotr Dulikowski The test `test_downgrade_after_successful_upgrade_fails` shuts down the whole cluster, reconfigures the nodes and then restarts. Apparently, the python driver sometimes does not handle this correctly; in one test run we observed that the driver did not manage to reconnect to any of the nodes, even though the nodes managed to start successfully. More context can be found on the python driver issue. This PR works around this issue by using the existing `reconnect_driver` function (which is a workaround for a _different_ python driver issue already) to help the driver reconnect after the full cluster restart. Refs: scylladb/python-driver#230 Closes #14276 * github.com:scylladb/scylladb: tests/topology: work around python driver issue in cluster feature tests test/topology{_raft_disabled}: move reconnect_driver to topology utils
|
Please we need some traction on this, it affects the stability of the tests. |
piodul
added a commit
to piodul/scylla
that referenced
this issue
Feb 12, 2024
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: scylladb#17277
|
@avelanarius - ping |
|
@sylwiaszunejko is currently working on this issue: #295 which seemed more current (before issues reported by @piodul today). I'll try to get to this soon. |
kbr-scylla
pushed a commit
to scylladb/scylladb
that referenced
this issue
Feb 13, 2024
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: #17277 Closes #17278
dgarcia360
pushed a commit
to dgarcia360/scylla
that referenced
this issue
Apr 30, 2024
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: scylladb#17277 Closes scylladb#17278
raphaelsc
added a commit
to raphaelsc/scylla
that referenced
this issue
May 22, 2024
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs scylladb#15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
raphaelsc
added a commit
to raphaelsc/scylla
that referenced
this issue
May 22, 2024
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs scylladb#15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
This was referenced May 27, 2024
mergify bot
pushed a commit
to scylladb/scylladb
that referenced
this issue
May 27, 2024
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs #15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> (cherry picked from commit e724675)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Observed in the following test run: https://jenkins.scylladb.com/job/scylla-master/job/next/6139/artifact/testlog/x86_64/dev/topology.test_cluster_features.2.log
I'm also attaching the logs to the issue: jenkins.scylladb.com_job_scylla-master_job_next_6139_artifact_testlog_x86_64_dev_topology.test_cluster_features.2.log
Link to the source of the failing test: https://github.com/scylladb/scylladb/blob/3a73048bc9a15bebca78dc89143e4e332fb50645/test/topology/test_cluster_features.py#L150
The test
test_downgrade_after_successful_upgrade_failsshuts down all nodes in the cluster, reconfigures them and then starts them again. In the linked run, the test session didn't reconnect after it happened:The driver doesn't reconnect automatically within a minute. IPs of the restarted nodes are the same as before
The text was updated successfully, but these errors were encountered: