-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Session might not reconnect if all nodes are restarted at once #230
Comments
I wonder if it's the same thing as #170 - which was supposedly fixed - or maybe it's some different kind of race which only happens when we restart all nodes? |
The test `test_downgrade_after_successful_upgrade_fails` stops all nodes, reconfigures them to support the test-only feature and restarts them. Unfortunately, it looks like python driver sometimes does not handle this properly and might not reconnect after all nodes are shut down. This commit adds a workaround for scylladb/python-driver#230 - the test re-creates python driver session right after nodes are restarted.
…ver not reconnecting after full cluster restart' from Piotr Dulikowski The test `test_downgrade_after_successful_upgrade_fails` shuts down the whole cluster, reconfigures the nodes and then restarts. Apparently, the python driver sometimes does not handle this correctly; in one test run we observed that the driver did not manage to reconnect to any of the nodes, even though the nodes managed to start successfully. More context can be found on the python driver issue. This PR works around this issue by using the existing `reconnect_driver` function (which is a workaround for a _different_ python driver issue already) to help the driver reconnect after the full cluster restart. Refs: scylladb/python-driver#230 Closes #14276 * github.com:scylladb/scylladb: tests/topology: work around python driver issue in cluster feature tests test/topology{_raft_disabled}: move reconnect_driver to topology utils
Please we need some traction on this, it affects the stability of the tests. |
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: scylladb#17277
@avelanarius - ping |
@sylwiaszunejko is currently working on this issue: #295 which seemed more current (before issues reported by @piodul today). I'll try to get to this soon. |
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: #17277 Closes #17278
Unfortunately, scylladb/python-driver#230 is not fixed yet, so it is necessary for the sake of our CI's stability to re-create the driver session after all nodes in the cluster are restarted. There is one place in test_topology_recovery_basic where all nodes are restarted but the driver session is not re-created. Even though nodes are not restarted at once but rather sequentially, we observed a failure with similar symptoms in a CI run for scylla-enterprise. Add the missing driver reconnect as a workaround for the issue. Fixes: scylladb#17277 Closes scylladb#17278
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs scylladb#15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs scylladb#15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
One source of flakiness is in test_tablet_metadata_propagates_with_schema_changes_in_snapshot_mode due to gossiper being aborted prematurely, and causing reconnection storm. Another is test_tablet_missing_data_repair which is flaky due an issue in python driver that session might not reconnect on rolling restart (tracked by scylladb/python-driver#230) Refs #15356. Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> (cherry picked from commit e724675)
Observed in the following test run: https://jenkins.scylladb.com/job/scylla-master/job/next/6139/artifact/testlog/x86_64/dev/topology.test_cluster_features.2.log
I'm also attaching the logs to the issue: jenkins.scylladb.com_job_scylla-master_job_next_6139_artifact_testlog_x86_64_dev_topology.test_cluster_features.2.log
Link to the source of the failing test: https://github.com/scylladb/scylladb/blob/3a73048bc9a15bebca78dc89143e4e332fb50645/test/topology/test_cluster_features.py#L150
The test
test_downgrade_after_successful_upgrade_fails
shuts down all nodes in the cluster, reconfigures them and then starts them again. In the linked run, the test session didn't reconnect after it happened:The driver doesn't reconnect automatically within a minute. IPs of the restarted nodes are the same as before
The text was updated successfully, but these errors were encountered: