Skip to content

IGNITE-28877 .NET tests: fail fast when embedded node cannot join topology instead of hanging the suite#13357

Open
anton-vinogradov wants to merge 2 commits into
apache:masterfrom
anton-vinogradov:ignite-28877
Open

IGNITE-28877 .NET tests: fail fast when embedded node cannot join topology instead of hanging the suite#13357
anton-vinogradov wants to merge 2 commits into
apache:masterfrom
anton-vinogradov:ignite-28877

Conversation

@anton-vinogradov

Copy link
Copy Markdown
Contributor

What

Since 2026-07-07 all three "Platform .NET (Windows)" TC suites intermittently die with a 120-minute "Execution timeout", each at a fixed position (1490/1342/78 passed tests). The captured stack of a stuck run shows the embedded JVM spinning forever in ServerImpl.sendJoinRequestMessage / joinTopology inside Ignition.Start: if discovery port 47500 is occupied at bind time (e.g. lingering teardown of the previous fixture, or a half-stopped node), the new node binds 47501+, while the single-port static IP finder of the test configs only knows 47500, and the default joinTimeout=0 retries forever.

This is not a code regression: the same revision both passes and hangs (e.g. builds 9190207 vs 9191076 on ci2), and a hung and a clean run consumed the very same dependency-build artifacts (9193992 vs 9198016).

Fix (test code only)

  • TestUtils.GetStaticDiscovery(): set JoinTimeout = 2 min so a stuck join fails the single test with a clear IgniteSpiException instead of eating the whole suite timeout, and cap MaxAckTimeout at 10 s — otherwise the exponential handshake backoff (up to 600 s per attempt) delays the join-timeout check by ~20 minutes per retry sweep.
  • Same joinTimeout for the XML-configured discovery in app.config / custom_app.config sections.
  • Add the forgotten Localhost = 127.0.0.1 in ComputeTaskSessionTest and the myGrid1 app.config section (every other test config pins it).

Verification

  • All three previously hanging fixtures pass locally (macOS, net6.0 + corretto-17): ComputeTaskSessionTest, IgniteConfigurationSectionTest, ClientClusterDiscoveryTestsNoLocalhost — 12/12.
  • Negative test reproducing the CI condition (a silent process squatting 127.0.0.1:47500): before the fix the node start spins forever (32+ min observed, jstack confirms the sendJoinRequestMessage loop); with the fix the test fails in 2 min 39 s with "Failed to connect to any address from IP finder within join timeout".

🤖 Generated with Claude Code

…ology instead of hanging the suite

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@anton-vinogradov anton-vinogradov mentioned this pull request Jul 10, 2026
7 tasks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@anton-vinogradov

Copy link
Copy Markdown
Contributor Author

CI verification on the affected suites (ci2, branch pull/13357/head):

Suite Before (broken runs) With this PR
Platform .NET (Windows) hung at 1490 passed → 120 min execution timeout SUCCESS, 2039 tests, full run (build 9199018)
Platform .NET (Windows) 2 hung at 1342 full run, 1428 tests (build 9199016); the single failure is IgniteConfigurationTest.TestMulticastIpFinder — a pre-existing flaky (28 failures in the last 40 runs across master and unrelated PRs)
Platform .NET (Windows) 3 hung at 78, then at 285 full run, all 301 tests executed (build 9199026; the second commit was needed — see below): 7 fast failures instead of an endless hang

Two findings from these runs:

  1. A second hang point existed in Windows 3 (matches historical master timeouts at "passed: 277-285", e.g. build 9196929): EntityFrameworkCacheInitializationTest.TestConfigurationAndStartup starts a node through the parameterless IgniteDbConfiguration(), which reads the igniteConfiguration section of the EntityFramework tests' own app.config — and that section had the same copy-pasted pattern: myGrid1 without localhost, while myGrid2 next to it has one. Same for AspNet.Tests/App.config. The second commit adds localhost / joinTimeout there as well; after it, Windows 3 runs to completion.

  2. The root environment problem on the agents is now visible thanks to the fail-fast: the ClientClusterDiscoveryTestsNoLocalhost failures show the node resolving its local addresses to addrs=[10.19.0.202]the loopback interface is missing from the enumeration on the CI Windows agents (healthy runs announce loopback too and interconnect via ::1/127.0.0.1). Such a node does not recognize the IP finder's 127.0.0.1:47500 as itself, sends a join request to its own listener and waits for an answer that never comes: "Failed to send message to address [addr=/127.0.0.1:47500, msg=TcpDiscoveryJoinRequestMessage...]". With the default joinTimeout=0 this retried forever — that was the hang; now it fails in ~2 minutes with a clear message. The remaining 7 red tests in Windows 3 are exactly this environment issue and should disappear once loopback enumeration on publicagent01..03 is fixed (it broke around Jul 6-8) — that part is for INFRA/agent admins, the suites themselves no longer hang either way.

(Note: build 9199026 formally shows "Canceled" — the agent process died during artifact publishing after all 301 tests had executed; yet another symptom of the current agent instability. Re-run queued for a clean status.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant