Await manager shutdown before stopping envtest

[aponcedeleonch]

Summary

• Operator CI / Operator Tests Integration has a flaky AfterSuite failure: timeout waiting for process kube-apiserver to stop. It accounts for 32 of the last 51 failures of that job, all in mcp-telemetry-config (27) and mcp-group (5).
• Root cause: SuiteEnv.Stop cancelled the suite context, slept 100ms, then tore down the envtest control plane. 100ms is not enough for the controller manager to finish, so its informers still held long-running watch requests when kube-apiserver got SIGTERM. kube-apiserver's graceful shutdown blocks draining those requests, overruns envtest's 20s ControlPlaneStopTimeout, and fails the suite. The CI logs show it directly: reflectors log Failed to watch ... connection refused during teardown, so the manager was demonstrably still running after the API server closed its listener.
• Why only those two suites: they are the only ones with fewer specs than Ginkgo procs (5 specs, 7 procs in CI). BeforeSuite runs on every proc, so the idle procs boot a control plane and cancel the context moments after the informers established their watches — the worst case for the race.
• Fix: wait for Manager.Start to return before stopping the control plane, instead of guessing at a delay. controller-runtime's Informers.Start blocks on waitGroup.Wait() until every informer has stopped, so that return is a real guarantee the watch connections are closed.
• Also bound the manager's own GracefulShutdownTimeout (10s) and the wait (15s) safely under envtest's 20s stop timeout, so a genuinely wedged runnable is reported as a manager shutdown problem rather than mis-attributed to apiserver teardown.

All 11 operator integration suites share this helper, so no per-suite changes are needed.

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests (task test)
• E2E tests (task test-e2e)
• Linting (task lint-fix)
• Manual testing (describe below)

Reproduced the flake locally first, then verified the fix:

	Result
Baseline, ginkgo -p --repeat=6 mcp-telemetry-config	Flaked on attempt 2 — identical 20.121s / same error as CI
Baseline, second --repeat run	Flaked again
Fix, 10 consecutive mcp-telemetry-config runs	10/10 pass
Fix, soak: 15 rounds × (mcp-telemetry-config + mcp-group)	30/30 suite runs pass, 0 timeouts, 11m34s
Fix, full ginkgo -p ./cmd/thv-operator/test-integration/... (11 suites)	Pass, 5m19s

Roughly 40 clean runs of the previously-flaky suites, against a baseline that failed twice in ~17.

task lint-fix reports one pre-existing failure unrelated to this change (cmd/thv/app/upgrade.go:204, gosec G115); confirmed present on the base commit by re-running with the change stashed.

Does this introduce a user-facing change?

No. Test infrastructure only.

Special notes for reviewers

Two deliberate omissions, both open to a different call:

• The idle-proc waste is untouched. mcp-telemetry-config and mcp-group each boot 7 control planes for 5 specs and spend ~38s almost entirely on control-plane startup, because the suites use BeforeSuite (per-proc) rather than SynchronizedBeforeSuite. That inefficiency is what makes this race so easy to hit, but fixing it means sharing one API server across procs and namespace-isolating the specs — worth a follow-up PR, and not needed to stop the flake.
• No tracking issue exists for this flake; I searched and found nothing matching, hence no Fixes #.

The AddReportEntry in the timeout branch is intentional: if the manager ever does hang, teardown still proceeds and the report names the real cause instead of surfacing a second, misleading apiserver timeout.

Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.57%. Comparing base (eedf566) to head (ffd827b).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6179      +/-   ##
==========================================
+ Coverage   72.56%   72.57%   +0.01%     
==========================================
  Files         736      737       +1     
  Lines       76391    76430      +39     
==========================================
+ Hits        55430    55470      +40     
+ Misses      17032    17025       -7     
- Partials     3929     3935       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.