Add live multi-config integration test via the real job.yml composite actions#64
Open
Tim-Pohlmann wants to merge 16 commits into
Open
Add live multi-config integration test via the real job.yml composite actions#64Tim-Pohlmann wants to merge 16 commits into
Tim-Pohlmann wants to merge 16 commits into
Conversation
4 tasks
Tim-Pohlmann
added a commit
that referenced
this pull request
Jul 7, 2026
… .github/actions Consolidates the duplicated checkout-of-download-rix.sh logic across job.yml's two jobs into download-rix itself, so it works standalone from any workflow. job.yml still needs its own checkout for the local `uses: ./...` composite-action references (run-rix-job, submit-rix-job) to resolve, so that checkout is narrowed to .github/actions instead of removed. Ported from test/live-job-yml-plumbing (PR #64), where this was originally applied - it belongs here since it's about the composite actions this PR introduces, not the live test job #64 adds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
Tim-Pohlmann
added a commit
that referenced
this pull request
Jul 9, 2026
… .github/actions Consolidates the duplicated checkout-of-download-rix.sh logic across job.yml's two jobs into download-rix itself, so it works standalone from any workflow. job.yml still needs its own checkout for the local `uses: ./...` composite-action references (run-rix-job, submit-rix-job) to resolve, so that checkout is narrowed to .github/actions instead of removed. Ported from test/live-job-yml-plumbing (PR #64), where this was originally applied - it belongs here since it's about the composite actions this PR introduces, not the live test job #64 adds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tim-Pohlmann
force-pushed
the
infra/job-yml-composite-actions
branch
from
July 9, 2026 17:41
ef18627 to
4e29404
Compare
… actions Runs the actual .github/actions/run-rix-job composite action (the same one job.yml's `run` job uses) against this repo, across several agent/model configs with no API key: opencode's free default model expects success, every other combination expects an auth-shaped failure. This exercises the real end-to-end plumbing - CLI parsing, agent CLI install, BuildInvocation, the actual agent process - instead of a hand-copied mirror of it that can silently drift (as happened when the real claude CLI started requiring --verbose but a bats-based mirror of its args didn't need updating). The success config also chains into submit-rix-job with a deliberately invalid write-token, proving live that SubmitRunner's zero-PR short-circuit never touches it when the agent queues no PRs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An absent API key made opencode silently fall back to its own free default model instead of erroring on the requested openai/gpt-4o, so the "expect: failure" assumption never held. Passing an invalid OPENAI_API_KEY instead forces a real authenticated call that fails. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ution Relative `uses: ./...` action references require the repo to already be checked out at github.workspace - GitHub Actions has no mechanism to resolve them against "the reusable workflow's own repo" without that. My prior commit removed job.yml's checkout entirely on a mistaken assumption; restoring it here, scoped to just .github/actions since download-rix now self-provisions its own download-rix.sh copy. Also adds a temporary workflow_dispatch caller to empirically verify this fix by invoking job.yml for real (not just via ci.yml's direct composite-action calls, which never exercise job.yml itself) - to be removed once confirmed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…efault-branch registration) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Its job is done: it confirmed the checkout fix in the previous commit actually resolves job.yml's local composite-action references end-to-end (not just via ci.yml's direct composite-action calls, which never exercise job.yml itself). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
status="failure" alone can mean an agent-exit failure, a setup failure, or a later delivery-stage failure - a regression in any of those could masquerade as the expected auth failure. Expose the result's error field as a step output and require it match "agent failed: exited with code ..." for the failure matrix legs.
… a release agent-plumbing previously downloaded the latest published release, which lags behind this PR's own changes - e.g. it was still exercising a pre-fix --verbose bug and never surfaced the new stderr diagnostic. A dedicated build-rix job now publishes the binary once and agent-plumbing's matrix legs share it via artifact.
steps.run.outputs.error is now real agent diagnostic text (spaces, quotes,
colons) rather than an always-short "exited with code N" string, and inline
${{ }} interpolation into a run: block substitutes before bash parses the
script - so a message containing shell metacharacters was executed as extra
commands instead of being treated as data (observed live: "command not
found" errors on 3 of 4 agent-plumbing legs once the stderr-diagnostic fix
started populating this field with real content). Pass status/error through
env instead, matching GitHub's recommended pattern for untrusted values.
Now that agent-plumbing runs this PR's own binary and its diagnostic field carries real content (verified in CI: "Not logged in · Please run /login" for claude, "Incorrect API key provided" for opencode), pin each failure leg to its expected reason via an expect-diagnostic substring check. Previously any same-exit-code agent failure (bad key, bad flag, bad model - anything) would satisfy these legs equally; now an auth failure can't be confused with some other agent-exit failure.
…tention - Hoist the repeated rix-bin path into a job-level RIX_BIN_PATH env var instead of duplicating the literal in two composite-action calls. - Flatten the case/esac exit-shape check to a single [[ ]] prefix match, matching the style of the diagnostic check right below it. - Tune the rix-bin artifact upload: it's a self-contained ~100MB binary needed only for this run's own lifetime, so skip zip compression (an already-compiled executable won't compress well) and cap retention at 1 day instead of the repo default.
The matrix's failure-mode legs asserted on the agent CLI's own prose error text (e.g. "Not logged in", "Incorrect API key"), which doesn't scale - every new failure mode meant hardcoding more vendor wording into workflow YAML, and the jq/case-based structural checks needed to make that robust don't belong embedded in a workflow script. agent-plumbing now only proves the plumbing itself works end-to-end (checkout, build, composite actions, output capture) via the one config that needs no credentials: opencode's free default model. Real per-failure-mode assertions move to a dedicated rix-job-e2e test next, where they can do proper JSON parsing instead of workflow-embedded jq. This also lets run-rix-job's "Run rix job" step drop back to a plain invocation - the stdout-capture/jq-parsed status+error outputs it grew existed only to feed the assertions removed here, and are unused now that rix always writes result.json (see #66) rather than only on success.
agent-plumbing (trimmed to the happy path in the previous commit) no longer covers auth/key failure modes. This adds a deeper layer that invokes the built rix binary directly - bypassing the composite-action layer - and does real jq-based JSON parsing in a proper bats test file instead of embedding that logic in workflow YAML. Rather than hardcoding either agent CLI's error wording (which doesn't scale and isn't ours to pin), each failure-mode test asserts that rix's error field's embedded diagnostic parses as JSON: both opencode's and claude's own structured error/result protocols always emit one for a real runtime failure (verified against this PR's own CI logs), while a regression that breaks agent invocation (e.g. a missing required CLI flag) surfaces as plain, non-JSON usage-error text instead - so this still catches that class of regression without tracking exact prose. Knowingly a bit brittle: neither agent CLI's npm package is version-pinned (see EnsureInstalledViaNpmAsync), so upstream changes to either could still shift this test's behavior - accepted for now.
Tim-Pohlmann
force-pushed
the
test/live-job-yml-plumbing
branch
from
July 9, 2026 17:49
796ac13 to
3194e07
Compare
Tim-Pohlmann
changed the base branch from
infra/job-yml-composite-actions
to
feat/always-write-result-json
July 9, 2026 17:49
Contributor
There was a problem hiding this comment.
Pull request overview
Adds live CI integration coverage for rix job by building the PR’s rix binary in CI and exercising (a) the real run-rix-job/submit-rix-job composite actions for a happy-path run, and (b) deeper end-to-end failure-shape assertions via a new Bats test that inspects result.json.
Changes:
- Add
build-rixjob to publish a Linuxrixbinary and share it via an artifact for downstream CI jobs. - Add
agent-plumbingjob that invokes the real.github/actions/run-rix-jobcomposite action against this repo, then runssubmit-rix-jobwith a deliberately invalid write token. - Add
rix-job-e2ejob plustests/e2e/rix-job-e2e.batsto runrix jobdirectly and assert structured diagnostic behavior for auth-like failures.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
tests/e2e/rix-job-e2e.bats |
New Bats e2e tests that run rix job and validate success/failure outcomes and diagnostic JSON shape. |
.github/workflows/ci.yml |
Adds CI jobs to build rix, run composite-action plumbing validation, and run the new Bats e2e suite. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Verified against the real CLI locally (invalid key -> real API 401, no credentials -> client-side rejection) that api_error_status differs between the two auth-failure paths but the "error" field is stable across both, so it's safe to pin without reintroducing prose-matching.
Temporary - will be reverted once we see the actual field value from a real CI-installed claude CLI.
Real CI evidence (job 86234738010) showed the diagnostic rix actually captures is the CLI's last stdout line (its final "type":"result" line), which has no top-level "error" field at all - that field only appears on an earlier "assistant" message line that rix's diagnostic capture never sees. is_error:true is what's actually present and stable across both real auth-failure paths.
…t wrap
- opencode's invalid-key test now asserts the diagnostic is a JSON
object, not just any JSON value (jq empty alone accepts a bare null).
- Fix a comment that was split mid-word ("per-failure" / "-mode").
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent-plumbinginvokes the real.github/actions/run-rix-jobcomposite action - the same onejob.yml'srunjob uses - directly against this repo, with opencode's free default model (no API key needed), then chains intosubmit-rix-jobwith a deliberately invalid write-token to prove live thatSubmitRunner's zero-PR short-circuit is safe (it never touches the write-token because the prompt guarantees no PRs are queued).rix-job-e2e(viatests/e2e/rix-job-e2e.bats) invokes the builtrixbinary directly across a matrix of (agent, model) configs with no API key, asserting via the job's ownstatusfield (success/failure/setupFailure) and structured diagnostic parsing - opencode's free default model is expected to succeed; every other combination (opencode with an explicit paid model, claude with/without an explicit model) is expected to fail with an auth-shaped error.Why this replaces the bats approach
The original bats-based draft invoked the real agent CLIs directly with hand-copied argument lists mirroring
ClaudeAgent/OpenCodeAgent. That mirror silently drifted from the real code once (a--verboseflag needed by the realclaudeCLI), which is exactly the class of bug live tests are meant to catch.agent-plumbingnow runs the actual composite action instead, so there is no separate copy of the argument-building logic to keep in sync;rix-job-e2estill invokes the binary directly (to do real JSON parsing on failure diagnostics) but sources its arguments from the CLI's own parsing, not a hand-copied mirror.Test plan
agent-plumbingandrix-job-e2erix-job-e2e's failure cases fail for an auth reason, not a CLI/flag rejection (check job logs)agent-plumbing's submit step succeeds with zero PRs opened