Skip to content

Fix DataNode restart readiness race in IoTDBConnectionsIT#18192

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/connections-it-restart-readiness
Jul 13, 2026
Merged

Fix DataNode restart readiness race in IoTDBConnectionsIT#18192
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/connections-it-restart-readiness

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Root cause

testClosedDataNodeGetConnections currently finishes its restart wait once SHOW DATANODES no longer reports an Unknown node. That condition reflects heartbeat readiness, but it does not guarantee that the restarted DataNode's client RPC service is accepting connections.

In the failing Windows run, the restarted DataNode's internal RPC service was ready at 12:06:09.457, while its client RPC service did not start until 12:06:10.945. Subsequent test methods entered that gap and failed with a connection error or inconsistent multi-node results.

A fixed five-second delay added in #17290 previously masked this gap. It was removed by #17423 after Windows metric binding became asynchronous, exposing this separate lifecycle race.

Fix

Cache the stopped DataNode wrapper and, after the existing ConfigNode status check, poll a direct table-dialect JDBC connection to that specific DataNode. The probe uses a write-only test connection so it creates only one physical session and cannot interfere with the connection-count assertions.

The one-second polling interval returns as soon as ClientRPC is ready, while the bounded 30-second timeout remains robust on slower CI workers. This verifies the actual client RPC and login path needed by the following tests instead of relying on another fixed sleep.

Testing

  • IoTDBConnectionsIT with TableClusterIT: 4 tests, 0 failures, 0 errors
  • Integration-test Checkstyle and Spotless checks passed

This PR has:

  • been self-reviewed.
  • added comments explaining the reason for the readiness probe.
  • updated integration-test coverage for the DataNode restart path.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • IoTDBConnectionsIT

@jt2594838 jt2594838 merged commit 5ceb047 into apache:master Jul 13, 2026
41 checks passed
@jt2594838 jt2594838 deleted the fix/connections-it-restart-readiness branch July 13, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants