Skip to content

chore: test reliability and speed skill - #133

Merged
sjmiller609 merged 5 commits into
mainfrom
codex/improve-test-reliability-and-speed
Mar 9, 2026
Merged

chore: test reliability and speed skill#133
sjmiller609 merged 5 commits into
mainfrom
codex/improve-test-reliability-and-speed

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Mar 8, 2026

Copy link
Copy Markdown
Collaborator

Note

Medium Risk
Touches runtime paths for ingress binary extraction and default-network discovery, so any bugs could affect multi-process startups or instance networking on Linux. Changes are bounded (file-lock + atomic rename; short self-heal polling) and mainly aimed at reducing CI flakiness.

Overview
Improves Linux integration test reliability by hardening embedded Caddy extraction: ExtractCaddyBinary now uses a per-binary Flock and temp-file + atomic rename writes (including the .sha256) to avoid concurrent ETXTBUSY/partial-write races.

Makes default-network allocation more robust under concurrent tests by adding getDefaultNetworkWithSelfHeal, which re-runs initialization and briefly polls for bridge readiness before failing.

Reduces test flakiness/latency by removing an external S3 curl dependency in TestFirecrackerNetworkLifecycle (replaced with a local probe server), trimming redundant reachability assertions in running-fork tests, and lowering guest exec WaitForAgent timeout in execInInstance to speed up polling-based assertions.

Adds a new skills/test-agent skill with prompt and investigation notes documenting the flake root causes and validation runs.

Written by Cursor Bugbot for commit c94b92d. This will update automatically on new commits. Configure here.

@sjmiller609 sjmiller609 changed the title Codex/improve test reliability and speed chore: test reliability and speed skill Mar 8, 2026
@sjmiller609
sjmiller609 marked this pull request as ready for review March 8, 2026 17:45

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated risk assessment (based on code diff evidence only): Medium-High risk.

Why this is not low-risk:

  • lib/network/allocate.go changes shared allocation behavior by introducing self-heal retry/polling logic in default network retrieval (CreateAllocation and RecreateAllocation paths).
  • lib/ingress/binaries_linux.go changes concurrent binary extraction semantics (cross-process file lock + atomic write/rename path).
  • New instruction files under skills/test-agent/agents/test-agent/PROMPT.md materially change prompt behavior for an automation agent.

Decision:

  • Code review required.
  • No self-approval for Medium-High risk.
  • Reviewer assignment not changed because 2 reviewers are already assigned (rgarcia, hiroTamada) and max is 2.

Additional checks:

  • No existing approvals were present, so no approval dismissal action was needed.
  • No CODEOWNERS file was found in-repo, so codeowners-specific blocking could not be inferred from repository files.

Open in Web View Automation 

@hiroTamada hiroTamada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good -- solid flake fixes and test speed improvements. left one minor suggestion on the skill docs.

description: Improve repository test quality by eliminating flakes first, then reducing slow-test redundancy, and finally optimizing longest tests while preserving coverage. Use when asked to repeatedly run CI-equivalent no-cache test cycles, diagnose flaky tests, and document findings in this skill's notes.
---

Follow the workflow and constraints documented in [agents/test-agent/PROMPT.md](agents/test-agent/PROMPT.md). Do not modify this file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider adding a getting-started section covering environment setup (e.g. required OS, root access, firecracker/qemu binaries, bridge networking prereqs) and how to generate any tokens or credentials needed to run the test suite. right now someone picking up this skill for the first time would need to reverse-engineer the setup from the test code and CI config.

@sjmiller609
sjmiller609 merged commit b4d0073 into main Mar 9, 2026
7 checks passed
@sjmiller609
sjmiller609 deleted the codex/improve-test-reliability-and-speed branch March 9, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants