Summary
CI failure analysis (hack/metrics/ci-failure-analysis.py on branch docs/velocity-metrics of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:
| Test |
Failures |
Job |
Root cause |
TestDurableDirLifecycle |
6× |
e2e-test |
503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING |
TestActorLifecycle |
4× |
e2e-test |
Same: 503 after resume, callActor 30s deadline too tight |
TestMultipleDurableDirLifecycle |
2× |
e2e-test |
Same |
TestSyncer_UpdateWorker_RetryOnVersionConflict |
2× |
run-tests |
wait.PollUntilContextTimeout 5s deadline too tight on loaded CI runners |
TestLoaderConcurrentHandshakes |
2× |
run-tests |
Concurrent symlink rotation — exact failure mode unknown, needs investigation |
Root Cause
e2e demo tests (demo_test.go)
After waitForActorStatus returns STATUS_RUNNING, the atenet-router's xDS route for the actor isn't always ready immediately. callActor retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.
TestSyncer_UpdateWorker_RetryOnVersionConflict
The test asserts via wait.PollUntilContextTimeout(..., 5*time.Second, ...). The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.
TestLoaderConcurrentHandshakes
Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.
Fix
See the linked PR. Changes:
internal/e2e/suites/demo/demo_test.go: callActor deadline 30s → 90s; waitForActorStatus timeout 60s → 120s
cmd/ateapi/internal/controlapi/syncer_test.go: final assertion poll timeout 5s → 15s
Summary
CI failure analysis (
hack/metrics/ci-failure-analysis.pyon branchdocs/velocity-metricsof the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:TestDurableDirLifecycleTestActorLifecyclecallActor30s deadline too tightTestMultipleDurableDirLifecycleTestSyncer_UpdateWorker_RetryOnVersionConflictwait.PollUntilContextTimeout5s deadline too tight on loaded CI runnersTestLoaderConcurrentHandshakesRoot Cause
e2e demo tests (demo_test.go)
After
waitForActorStatusreturnsSTATUS_RUNNING, the atenet-router's xDS route for the actor isn't always ready immediately.callActorretries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.TestSyncer_UpdateWorker_RetryOnVersionConflict
The test asserts via
wait.PollUntilContextTimeout(..., 5*time.Second, ...). The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.TestLoaderConcurrentHandshakes
Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.
Fix
See the linked PR. Changes:
internal/e2e/suites/demo/demo_test.go:callActordeadline 30s → 90s;waitForActorStatustimeout 60s → 120scmd/ateapi/internal/controlapi/syncer_test.go: final assertion poll timeout 5s → 15s