CI: client-routes full node replacement test can fail with HTTP 500 after decommission

[dkropachev]

Failure

Integration test flake seen in PR #913:

• Run: https://github.com/scylladb/python-driver/actions/runs/29257587657
• Job: https://github.com/scylladb/python-driver/actions/runs/29257587657/job/86841926454
• Workflow/job: Integration tests / test asyncio (3.13)
• Merge commit under test: 46abdc63cd03e72369ff52d2cc01bbac0f94d6c9
• Python: CPython 3.13.14
• Scylla: release:2026.1, resolved to 2026.1.8
• Env: EVENT_LOOP_MANAGER=asyncio, PROTOCOL_VERSION=4

Failed test:

tests/integration/standard/test_client_routes.py::TestFullNodeReplacementThroughNlb::test_should_survive_full_node_replacement_through_nlb

Error:

urllib.error.HTTPError: HTTP Error 500: Internal Server Error

Observed sequence

The failure happened during Stage 4, while decommissioning original nodes behind the NLB emulator.

Relevant CI log sequence:

14:35:32.132 Posting 6 routes to http://127.0.0.1:10000/v2/client-routes -> 200
14:35:40.138 Stage 4: Decommissioning original nodes [1, 2, 3]
14:35:46.982 Posting 5 routes to http://127.0.0.3:10000/v2/client-routes -> 200
14:35:47.030 Node 1 decommissioned, 5 nodes still via NLB
14:35:47.030 Decommissioning node 2
14:35:49.371 Posting 4 routes to http://127.0.0.3:10000/v2/client-routes
14:35:50.229 pytest begins teardown after HTTP 500

The failing POST is from:

tests/integration/standard/test_client_routes.py:1247

The helper uses urllib.request.urlopen(req) directly at:

tests/integration/standard/test_client_routes.py:366

Because urllib.error.HTTPError is not caught and its response body is not logged, the CI output does not include the Scylla-side reason for the 500.

Race hypothesis

The test appears to have a topology-change race window. In the decommission loop it currently:

1. calls get_node(node_id).decommission()
2. removes the node from the NLB emulator
3. removes the host id from the route map
4. immediately POSTs the updated route set to /v2/client-routes
5. only afterwards waits for driver metadata/query stability

So the route update can hit a surviving node while Scylla is still settling after the decommission. In this failure it posted to 127.0.0.3:10000 immediately after node2 removal and got a transient-looking 500.

Local reproduction attempt

Targeted reruns of the exact PR merge did not reproduce locally:

SCYLLA_VERSION=release:2026.1 EVENT_LOOP_MANAGER=asyncio PROTOCOL_VERSION=4 \
  uv run pytest -q -s tests/integration/standard/test_client_routes.py::TestFullNodeReplacementThroughNlb::test_should_survive_full_node_replacement_through_nlb

Results:

run 1: passed, 126.60s
run 2: passed, 60.08s
run 3: passed, 91.11s

Local Python was 3.13.5, not CI's 3.13.14.

Even on passing local runs, Scylla emitted topology/rpc noise during this flow, for example:

storage_service - Failed to complete splitting of table ... Read barrier requested by a node outside of the configuration ... retrying after 5000ms
rpc - server connection dropped: Semaphore broken

That supports this being timing-sensitive topology churn, not a deterministic driver failure.

Artifacts

There are no uploaded artifacts for the run:

{"total_count":0,"artifacts":[]}

The workflow also does not have an actions/upload-artifact step, so no cluster logs are available beyond the job log.

Suggested fix

Make this test more robust and diagnosable:

• catch urllib.error.HTTPError in post_client_routes() and log/read the response body
• retry bounded 5xx responses for /v2/client-routes during this replacement test
• consider waiting for topology/query stability before posting updated routes after each decommission, or choose a known stable surviving/new node for the REST POST