feat: add typed capability effect evidence to agent runners - #2755
Merged
Conversation
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:16 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds optional capability-effect evidence validation to the shared runner library and exposes normalized evidence through Claude, Codex, Cursor, and Gemini reusable workflows. Updates tests, contracts, integration guidance, observability documentation, and worker metadata. ChangesCapability evidence contract
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant ReusableRunnerWorkflow
participant RunnerLibrary
participant GitHubOutputs
Caller->>ReusableRunnerWorkflow: provide optional evidence inputs
ReusableRunnerWorkflow->>RunnerLibrary: run normalize-evidence
RunnerLibrary->>GitHubOutputs: write normalized evidence
ReusableRunnerWorkflow->>Caller: return validated workflow outputs
Possibly related issues
Suggested labels: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
stranske
force-pushed
the
codex/issue-2752-capability-effect-evidence
branch
from
July 11, 2026 00:18
11f3194 to
3e166bc
Compare
Contributor
Automated Status SummaryHead SHA: c5d7f87
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScope
Context for AgentRelated Issues/PRsTasks
Acceptance criteria
|
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:19 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11f31947fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the agent-runner output contract to optionally carry typed, provider-neutral capability/effect evidence, and wires that evidence through the shared runner library into the four registry-backed runner reusable workflows while updating the relevant documentation and tests.
Changes:
- Add
CapabilityEffectEvidence+normalize_capability_effect_evidence()validation (plusnormalize-evidenceCLI) to emit bounded, all-or-none evidence outputs. - Extend
reusable-{codex,claude,cursor,gemini}-run.ymlto accept optional evidence inputs, validate them pre-run, and surface validated values asworkflow_calloutputs. - Update runner contract + integration/observability docs and add unit coverage for normalization + CLI output writing.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/scripts/test_runner_lib.py |
Adds unit tests for evidence normalization and the new normalize-evidence CLI behavior. |
scripts/runner_lib/core.py |
Introduces evidence schema/validation, GitHub outputs emission, and a new CLI subcommand. |
scripts/runner_lib/__init__.py |
Exports the new evidence dataclass and normalization helper from the package surface. |
docs/keepalive/Observability_Contract.md |
Documents the new optional evidence outputs in the keepalive observability contract. |
docs/INTEGRATION_GUIDE.md |
Notes the optional evidence outputs and adds them to the reusable runner outputs catalog entries. |
docs/guides/AGENT_RUNNER_IMPLEMENTATION.md |
Adds implementation guidance for validating and relaying typed evidence via the runner library CLI. |
docs/contracts/agent-runner-output.md |
Bumps contract version to 1.1 and defines the optional evidence output schema and semantics. |
docs/ci/WORKFLOW_OUTPUTS.md |
Updates the canonical workflow outputs catalog to include the new runner outputs. |
.github/workflows/reusable-gemini-run.yml |
Adds optional evidence inputs + outputs and validates them via scripts.runner_lib normalize-evidence. |
.github/workflows/reusable-cursor-run.yml |
Adds optional evidence inputs + outputs and validates them via scripts.runner_lib normalize-evidence. |
.github/workflows/reusable-codex-run.yml |
Adds optional evidence inputs + outputs and validates them via scripts.runner_lib normalize-evidence. |
.github/workflows/reusable-claude-run.yml |
Adds optional evidence inputs + outputs and validates them via scripts.runner_lib normalize-evidence. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/reusable-claude-run.yml:
- Around line 449-467: The workflow’s post-validation steps must not run when
capability evidence validation fails. Add a condition requiring
steps.capability_evidence.outcome != 'failure' to the always()-based Run Claude
and Commit and push changes steps, preserving always() while preventing agent
execution and commits after malformed evidence.
- Around line 94-127: Forward capability_id, effect_fingerprint,
evidence_artifact_ref, supervision_mode, capability_evidence_status, and
terminal_disposition from the consumer workflow’s reusable-claude-run
invocation, using the corresponding workflow inputs and preserving their
defaults, so the new fields are available end-to-end.
In @.github/workflows/reusable-cursor-run.yml:
- Around line 466-484: The workflow currently allows Run Cursor and Commit and
push changes to proceed when capability_evidence fails. Update both steps’ if
conditions to include steps.capability_evidence.outcome != 'failure' alongside
their existing conditions, ensuring invalid or partial evidence blocks agent
execution and pushing changes.
In @.github/workflows/reusable-gemini-run.yml:
- Around line 474-492: Gate the “Run Gemini” and “Commit and push changes”
workflow steps using conditions that retain their existing always-run behavior
while also requiring steps.capability_evidence.outcome != 'failure'. This must
prevent agent execution and repository mutation/push when the
capability_evidence validation fails.
In `@docs/ci/WORKFLOW_OUTPUTS.md`:
- Line 250: Insert a blank line immediately before the “Provider-neutral
optional capability evidence” heading to satisfy markdownlint MD022.
In `@docs/INTEGRATION_GUIDE.md`:
- Around line 154-160: Update docs/INTEGRATION_GUIDE.md so the quick index
consistently documents all registry-backed runners: add six capability-evidence
output rows for Cursor and Gemini, matching the entries for Codex and Claude and
the definitions in docs/ci/WORKFLOW_OUTPUTS.md. Alternatively, narrow the
paragraph’s claim to only the runners currently documented, but prefer adding
the missing Cursor and Gemini rows.
In `@scripts/runner_lib/core.py`:
- Around line 87-112: Validate CapabilityEffectEvidence during direct
construction via __post_init__ or a shared validator: allow the completely empty
legacy instance, but require every field when any capability field is populated
and reject invalid values. Ensure github_outputs() can only emit validated
evidence while preserving the public dataclass API.
- Line 41: Update CAPABILITY_ID_RE to enforce valid lowercase-kebab identifiers:
require each segment to start and end with an alphanumeric character and allow
only single hyphens between segments, while preserving the existing length
constraints and capability: prefix.
In `@tests/scripts/test_runner_lib.py`:
- Around line 61-73: The capability-ID test currently exercises the
partial-record validation instead of invalid ID validation. In
test_capability_effect_evidence_rejects_partial_records, create a complete valid
six-field evidence fixture, then override only capability_id with
capability:Consumer_Sync and assert the expected invalid-ID error; keep the
existing partial-record cases unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 651cd2bd-5d64-4b84-9f44-b86650048c2a
📒 Files selected for processing (12)
.github/workflows/reusable-claude-run.yml.github/workflows/reusable-codex-run.yml.github/workflows/reusable-cursor-run.yml.github/workflows/reusable-gemini-run.ymldocs/INTEGRATION_GUIDE.mddocs/ci/WORKFLOW_OUTPUTS.mddocs/contracts/agent-runner-output.mddocs/guides/AGENT_RUNNER_IMPLEMENTATION.mddocs/keepalive/Observability_Contract.mdscripts/runner_lib/__init__.pyscripts/runner_lib/core.pytests/scripts/test_runner_lib.py
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:26 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Owner
Author
|
Addressed all currently actionable review threads in commits |
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Owner
Author
|
Runner dispatch state for autofix on PR #2755. Do not edit. |
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:34 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
github-actions
Bot
added
the
autofix
Contributor
|
Autofix updated these files:
|
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/reusable-codex-run.yml (1)
1412-1414: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winAdd the capability-evidence gate to Codex commit/push.
Commit and push changesstill runs underif: always(), so a failedcapability_evidencestep can still mutate and push the branch. Use the same guard as the other reusable agent workflows.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/reusable-codex-run.yml around lines 1412 - 1414, The Commit and push changes step currently uses if: always(), allowing pushes after capability_evidence fails. Replace this condition with the same capability-evidence success guard used by the other reusable agent workflows, while preserving the step’s existing commit/push behavior.
♻️ Duplicate comments (2)
docs/ci/WORKFLOW_OUTPUTS.md (1)
250-250: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd a blank line before the heading.
markdownlintreports MD022 because the new heading is not separated from the preceding content. Insert one blank line before## Provider-neutral optional capability evidence.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/ci/WORKFLOW_OUTPUTS.md` at line 250, Insert a blank line immediately before the “## Provider-neutral optional capability evidence” heading in WORKFLOW_OUTPUTS.md to satisfy markdownlint MD022.Source: Linters/SAST tools
docs/INTEGRATION_GUIDE.md (1)
154-160: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winKeep the quick index consistent with the “all runners” claim.
The guide says all registry-backed runners expose these outputs, but the quick index lists rows only for Codex and Claude. Add the six corresponding Cursor and Gemini rows, or narrow the claim to the runners listed here.
docs/ci/WORKFLOW_OUTPUTS.mdalready documents all four.Also applies to: 252-285
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/INTEGRATION_GUIDE.md` around lines 154 - 160, The quick index is inconsistent with the claim that all registry-backed runners expose the six capability evidence outputs. Update the index section in INTEGRATION_GUIDE.md to add rows for Cursor and Gemini covering capability-id, effect-fingerprint, evidence-artifact-ref, supervision-mode, capability-evidence-status, and terminal-disposition, matching the existing Codex and Claude entries and WORKFLOW_OUTPUTS.md.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/runner_lib/core.py`:
- Around line 103-109: The validation around the evidence reference must detect
credential-like prefixes throughout namespaced references, not only at the
beginning. Update the checks in the validation function containing lowered_ref
and SECRET_LIKE_EVIDENCE_PREFIXES to reject matching markers at token boundaries
or anywhere appropriate in the full reference, while preserving legitimate
references; add regression tests for examples such as artifact:ghp_example and
github-actions:owner/repo:123:sk-example.
- Around line 88-90: Update _validate_capability_effect_evidence_values to
validate every value is a string before applying the empty-record
any(values.values()) check, raising the established validation error for None,
booleans, and other non-string inputs; preserve absence semantics only for valid
empty strings. Add regression coverage for constructing CapabilityEffectEvidence
with non-string fields and verify github_outputs() is not allowed to emit
invalid values.
In `@templates/consumer-repo/docs/contracts/agent-runner-output.md`:
- Line 116: Replace the `effect-fingerprint` example in the documentation with a
`sha256:` prefix followed by exactly 64 lowercase hexadecimal characters,
matching the implementation’s required format.
---
Outside diff comments:
In @.github/workflows/reusable-codex-run.yml:
- Around line 1412-1414: The Commit and push changes step currently uses if:
always(), allowing pushes after capability_evidence fails. Replace this
condition with the same capability-evidence success guard used by the other
reusable agent workflows, while preserving the step’s existing commit/push
behavior.
---
Duplicate comments:
In `@docs/ci/WORKFLOW_OUTPUTS.md`:
- Line 250: Insert a blank line immediately before the “## Provider-neutral
optional capability evidence” heading in WORKFLOW_OUTPUTS.md to satisfy
markdownlint MD022.
In `@docs/INTEGRATION_GUIDE.md`:
- Around line 154-160: The quick index is inconsistent with the claim that all
registry-backed runners expose the six capability evidence outputs. Update the
index section in INTEGRATION_GUIDE.md to add rows for Cursor and Gemini covering
capability-id, effect-fingerprint, evidence-artifact-ref, supervision-mode,
capability-evidence-status, and terminal-disposition, matching the existing
Codex and Claude entries and WORKFLOW_OUTPUTS.md.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8092cba2-104a-48d5-bbc6-67511186c5dd
📒 Files selected for processing (13)
.github/workflows/reusable-claude-run.yml.github/workflows/reusable-codex-run.yml.github/workflows/reusable-cursor-run.yml.github/workflows/reusable-gemini-run.ymldocs/INTEGRATION_GUIDE.mddocs/ci/WORKFLOW_OUTPUTS.mddocs/contracts/agent-runner-output.mddocs/guides/AGENT_RUNNER_IMPLEMENTATION.mddocs/keepalive/Observability_Contract.mdscripts/runner_lib/__init__.pyscripts/runner_lib/core.pytemplates/consumer-repo/docs/contracts/agent-runner-output.mdtests/scripts/test_runner_lib.py
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 00:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 02:03 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
had a problem deploying
to
agent-standard
July 11, 2026 02:03 — with
GitHub Actions
Failure
GitHub Actions
Failure
stranske-keepalive
Bot
added
agent:needs-attention
12 tasks
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 02:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Provider Comparison ReportProvider Summary
📋 Full Provider Details (click to expand)openai
anthropic
Agreement
DisagreementNo major disagreements detected. Unique Insights
🔍 LangSmith Traces |
Owner
Author
|
Verifier CONCERNS audit: current Workflows main contains the optional runner-output validation/emission contract, but the source issue requires adapting accepted/rejected evidence into Orchestrator’s completion-event and capability-ledger lifecycle, including replay idempotency and counterexample preservation. No such adapter or end-to-end replay fixture is present in this repository’s current main. This is a real completion gap, not a verifier false positive; initiating the bounded verifier follow-up lane. |
stranske
temporarily deployed
to
agent-high-privilege
July 11, 2026 05:27 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
55 tasks
Owner
Author
|
📋 Follow-up issue created: #2758 Verification concerns have been analyzed and structured into a follow-up issue. Next steps:
|
Closed
47 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2752
Automated Status Summary
Scope
Context for Agent
Related Issues/PRs
Tasks
Acceptance criteria
Summary by CodeRabbit