Skip to content

fix nightly CI: point at run-swe-rollouts, port example to the Proxy API, add sharding#124

Merged
Meirtz merged 2 commits into
masterfrom
fix/nightly-run-swe-rollouts
Jul 2, 2026
Merged

fix nightly CI: point at run-swe-rollouts, port example to the Proxy API, add sharding#124
Meirtz merged 2 commits into
masterfrom
fix/nightly-run-swe-rollouts

Conversation

@Meirtz

@Meirtz Meirtz commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Problem

Nightly CI still targeted examples/eval-cc-swe, which has been deleted — every scheduled run failed. The successor example examples/run-swe-rollouts also referenced a removed module API (agentix.bridge.anthropic), so it could not run either.

What this does

Workflow (.github/workflows/nightly-ci.yml)

  • Point build/deploy/evaluate/upload steps at examples/run-swe-rollouts; build a named bundle tar and deploy it explicitly.
  • Deploy step now runs under set -euo pipefail and fails fast when the bundle ref comes back empty/null — previously a failed agentix deploy piped into jq still exited 0 and all 20 shards ran for hours against an empty BUNDLE.

Example (examples/run-swe-rollouts/main.py)

  • Port ClaudeCodeAgent to agentix.bridge.Proxy + AnthropicFromOpenAIClient (current plugin surface).
  • Register /abridge before the sandbox connection exists: the runner's dataset.setup() already connects the lazily-created RuntimeClient, and register_namespace only works pre-connect — solve() now recycles the connection with sandbox.aclose() first. Without this, every agent-path rollout raised RuntimeError (the nightly --ground-truth path never exercises it).
  • Extract the diff before tunnel teardown, and log (not raise) stop-phase errors, so teardown can't discard a computed patch or mask the agent's real error.
  • --num-shards / --shard-index round-robin sharding for CI; an empty shard (shards > rows) exits 0 and writes an empty summary.json so the artifact upload still finds its path.
  • Validate --limit (None or >= 1; a negative value used to silently select nearly the whole shard); apply it per shard while slicing lazily; reject --instance-id combined with sharding instead of silently ignoring the shard flags.
  • Commit the example's uv.lock, matching the other cookbook examples.

Bridge plugin (plugins/abridge)

  • AnthropicFromOpenAIClient.aclose() releases the upstream AsyncOpenAI connection pool; Proxy.stop()/session() already close any client that implements aclose(), so per-rollout clients no longer leak httpx pools at high concurrency. Covered by a new test.

Verification

  • Selection semantics verified against a local 10-row parquet: shard partition/disjoint-cover, per-shard --limit, ground-truth vs agent defaults, --instance-id ordering + unknown-id error, and all exit-code paths (empty shard → 0; bad limit / bad shard args / instance-id+sharding / missing key → 2) — 20/20.
  • uv sync + python main.py --help + import check in the example dir.
  • Example API usage cross-checked against the real plugin sources (ClaudeCodeInput fields, ClaudeCodeResult.returncode, swe.prepare_env/swe.score signatures, bash.run(...).stdout) since examples are outside pyright/pytest coverage.
  • pytest plugins/abridge/tests 44/44; uv run pyright (CI invocation) 0 errors; ruff check examples/run-swe-rollouts .github clean.
  • The workflow's build→deploy→evaluate path needs Docker; locally only YAML-parse/dry checks were run.

🤖 Generated with Claude Code

Meirtz and others added 2 commits July 2, 2026 05:26
Nightly CI still targeted examples/eval-cc-swe, which was deleted — every
run failed at checkout of the working directory. Point the workflow at the
successor example and bring that example up to the current plugin surface:

- port ClaudeCodeAgent from the removed agentix.bridge.anthropic module API
  to agentix.bridge.Proxy + AnthropicFromOpenAIClient
- register /abridge before the sandbox connection exists: dataset.setup()
  already connected the lazily-created RuntimeClient, and a host namespace
  can only join before connect, so solve() recycles the connection with
  sandbox.aclose() first
- extract the diff before tunnel teardown and log (not raise) stop-phase
  errors so teardown can't discard a computed patch or mask the agent's
  real error
- give AnthropicFromOpenAIClient an aclose() so Proxy.stop()/session()
  releases the upstream AsyncOpenAI pool (with test)
- add --num-shards/--shard-index round-robin sharding; an empty shard
  exits 0 (writing an empty summary.json) so shards may outnumber rows;
  --limit is validated (None or >= 1) and applies per shard lazily;
  --instance-id is rejected in combination with sharding
- deploy step: set pipefail and fail fast on an empty/null bundle ref
  instead of letting 20 shards run against an empty BUNDLE
- commit the example's uv.lock, matching the other cookbook examples

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`ruff check .` from the repo root classifies `datasets` with the
first-party block (as on master); checking the example dir directly
does not flag it, which is how this slipped past local verification.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Meirtz Meirtz merged commit 8060b18 into master Jul 2, 2026
5 checks passed
@Meirtz Meirtz deleted the fix/nightly-run-swe-rollouts branch July 2, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant