fix nightly CI: point at run-swe-rollouts, port example to the Proxy API, add sharding#124
Merged
Merged
Conversation
Nightly CI still targeted examples/eval-cc-swe, which was deleted — every run failed at checkout of the working directory. Point the workflow at the successor example and bring that example up to the current plugin surface: - port ClaudeCodeAgent from the removed agentix.bridge.anthropic module API to agentix.bridge.Proxy + AnthropicFromOpenAIClient - register /abridge before the sandbox connection exists: dataset.setup() already connected the lazily-created RuntimeClient, and a host namespace can only join before connect, so solve() recycles the connection with sandbox.aclose() first - extract the diff before tunnel teardown and log (not raise) stop-phase errors so teardown can't discard a computed patch or mask the agent's real error - give AnthropicFromOpenAIClient an aclose() so Proxy.stop()/session() releases the upstream AsyncOpenAI pool (with test) - add --num-shards/--shard-index round-robin sharding; an empty shard exits 0 (writing an empty summary.json) so shards may outnumber rows; --limit is validated (None or >= 1) and applies per shard lazily; --instance-id is rejected in combination with sharding - deploy step: set pipefail and fail fast on an empty/null bundle ref instead of letting 20 shards run against an empty BUNDLE - commit the example's uv.lock, matching the other cookbook examples Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`ruff check .` from the repo root classifies `datasets` with the first-party block (as on master); checking the example dir directly does not flag it, which is how this slipped past local verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Nightly CI still targeted
examples/eval-cc-swe, which has been deleted — every scheduled run failed. The successor exampleexamples/run-swe-rolloutsalso referenced a removed module API (agentix.bridge.anthropic), so it could not run either.What this does
Workflow (
.github/workflows/nightly-ci.yml)examples/run-swe-rollouts; build a named bundle tar and deploy it explicitly.set -euo pipefailand fails fast when the bundle ref comes back empty/null— previously a failedagentix deploypiped intojqstill exited 0 and all 20 shards ran for hours against an emptyBUNDLE.Example (
examples/run-swe-rollouts/main.py)ClaudeCodeAgenttoagentix.bridge.Proxy+AnthropicFromOpenAIClient(current plugin surface)./abridgebefore the sandbox connection exists: the runner'sdataset.setup()already connects the lazily-createdRuntimeClient, andregister_namespaceonly works pre-connect —solve()now recycles the connection withsandbox.aclose()first. Without this, every agent-path rollout raisedRuntimeError(the nightly--ground-truthpath never exercises it).--num-shards/--shard-indexround-robin sharding for CI; an empty shard (shards > rows) exits 0 and writes an emptysummary.jsonso the artifact upload still finds its path.--limit(None or >= 1; a negative value used to silently select nearly the whole shard); apply it per shard while slicing lazily; reject--instance-idcombined with sharding instead of silently ignoring the shard flags.uv.lock, matching the other cookbook examples.Bridge plugin (
plugins/abridge)AnthropicFromOpenAIClient.aclose()releases the upstreamAsyncOpenAIconnection pool;Proxy.stop()/session()already close any client that implementsaclose(), so per-rollout clients no longer leak httpx pools at high concurrency. Covered by a new test.Verification
--limit, ground-truth vs agent defaults,--instance-idordering + unknown-id error, and all exit-code paths (empty shard → 0; bad limit / bad shard args / instance-id+sharding / missing key → 2) — 20/20.uv sync+python main.py --help+ import check in the example dir.ClaudeCodeInputfields,ClaudeCodeResult.returncode,swe.prepare_env/swe.scoresignatures,bash.run(...).stdout) since examples are outside pyright/pytest coverage.pytest plugins/abridge/tests44/44;uv run pyright(CI invocation) 0 errors;ruff check examples/run-swe-rollouts .githubclean.🤖 Generated with Claude Code