Skip to content

Fix flaky tests: e2e demo suite timeouts and syncer poll timeout #799

Description

Summary

CI failure analysis (hack/metrics/ci-failure-analysis.py on branch docs/velocity-metrics of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:

Test Failures Job Root cause
TestDurableDirLifecycle e2e-test 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING
TestActorLifecycle e2e-test Same: 503 after resume, callActor 30s deadline too tight
TestMultipleDurableDirLifecycle e2e-test Same
TestSyncer_UpdateWorker_RetryOnVersionConflict run-tests wait.PollUntilContextTimeout 5s deadline too tight on loaded CI runners
TestLoaderConcurrentHandshakes run-tests Concurrent symlink rotation — exact failure mode unknown, needs investigation

Root Cause

e2e demo tests (demo_test.go)

After waitForActorStatus returns STATUS_RUNNING, the atenet-router's xDS route for the actor isn't always ready immediately. callActor retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.

TestSyncer_UpdateWorker_RetryOnVersionConflict

The test asserts via wait.PollUntilContextTimeout(..., 5*time.Second, ...). The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.

TestLoaderConcurrentHandshakes

Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.

Fix

See the linked PR. Changes:

  • internal/e2e/suites/demo/demo_test.go: callActor deadline 30s → 90s; waitForActorStatus timeout 60s → 120s
  • cmd/ateapi/internal/controlapi/syncer_test.go: final assertion poll timeout 5s → 15s

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions