Skip to content

Issue #2741: Add keepalive capability bundle contract - #2744

Merged
stranske merged 17 commits into
mainfrom
codex/issue-2741-capability-bundle
Jul 10, 2026
Merged

Issue #2741: Add keepalive capability bundle contract#2744
stranske merged 17 commits into
mainfrom
codex/issue-2741-capability-bundle

Conversation

@stranske

@stranske stranske commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Source: Issue #2741

Closes #2741

Automated Status Summary

Scope

Workflows correctly owns the GitHub event loop and agent-agnostic routing (docs/keepalive/Agents.md:68-75; GoalsAndPlumbing.md:140-151), but it has no neutral contract for portable Orchestrator capabilities. Keepalive metrics currently return iteration/action/error/duration/task counts without applied capability/gate IDs (docs/keepalive/METRICS_SCHEMA.md:1-36).

The research backplane is a separate activation-debt proving case. This issue establishes the neutral capability-bundle contract only; participant emission and Workflows registry transitions are split into dependent repo-specific issues.

Tasks

  • Add a versioned capability-bundle/v1 schema under Workflows contract/schema paths with capability ID/version/hash, deterministic selector predicates, owner, task/acceptance fragments, gate/playbook refs, expiry, and rollback; forbid local weights/credentials/raw prompts.
  • Add registry validation and deterministic selection helpers used by .github/scripts/keepalive_prompt_composer.js and related Keepalive runner code without changing label-to-runner authority.
  • Extend Keepalive metrics/state with matched/applied capability IDs, bundle/hash, gate versions, rejection reasons, and downstream result.
  • Update templates/consumer-repo, sync manifests, conformance tests, and freshness tests from the Workflows source of truth.

Acceptance criteria

  • Workflows contract tests reject a bundle with an unknown capability ID, mismatched content hash, expired selector, unsafe inline prompt, or local routing weight.
  • A Keepalive fixture deterministically applies a matching capability fragment and returns its exact ID/hash/gate version in metrics; a nonmatching fixture applies none.
  • Root and consumer-template freshness tests fail when bundle schema/runner integration diverges.
  • Deliberate break: in .github/scripts/__tests__/capability-bundle-contract.test.js::validBundle, change only content_hash to sha256:deadbeef; run exact test .github/scripts/__tests__/capability-bundle-contract.test.js::hash mismatch blocks dispatch via node --test .github/scripts/__tests__/capability-bundle-contract.test.js --test-name-pattern='hash mismatch blocks dispatch'; observe AssertionError: invalid capability hash reached prompt composition; revert and show the exact test passes.

Summary by CodeRabbit

  • New Features
    • Added deterministic capability bundle support with SHA-256 integrity checks, safe-field blocking, and selector-based loading/selection.
    • Updated keepalive prompt composition to prepend rendered applied bundle fragments and include applied bundle evidence.
    • Extended keepalive metrics with applied bundle IDs/hashes, gate/playbook versions, and structured rejection reasons (including invalid JSON).
  • Documentation
    • Added Capability Bundle v1 contract docs and schema; updated metrics schema accordingly.
  • Tests
    • Added contract, parsing, selection/composition, and metrics coverage.
  • Chores
    • Synced new contract assets and consumer templates/helpers.

@stranske stranske added the agent:codex Agent-created issues from Codex label Jul 10, 2026
Copilot AI review requested due to automatic review settings July 10, 2026 03:08
@stranske stranske added agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation agent:codex Agent-created issues from Codex labels Jul 10, 2026
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:08 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

Capability Bundle v1 adds deterministic bundle validation and selection, integrates matching fragments into keepalive prompts, records bundle evidence in metrics, defines JSON Schema and documentation contracts, synchronizes consumer templates, and adds contract tests.

Capability Bundle v1

Layer / File(s) Summary
Bundle contract and validation
.github/scripts/capability_bundle.js, templates/consumer-repo/.github/scripts/capability_bundle.js, docs/contracts/..., templates/consumer-repo/docs/contracts/..., .github/sync-manifest.yml
Adds schema enforcement, stable hashing, unsafe-field rejection, expiration checks, bundle loading, and synchronized contract artifacts.
Bundle selection and prompt composition
.github/scripts/keepalive_prompt_composer.js, templates/consumer-repo/.github/scripts/keepalive_prompt_composer.js
Matches bundles against context selectors, renders applied fragments, and adds capability content and selection metadata to composed prompts.
Metrics propagation and contract coverage
.github/scripts/keepalive_loop.js, templates/consumer-repo/.github/scripts/keepalive_loop.js, .github/scripts/__tests__/capability-bundle-contract.test.js, .github/scripts/__tests__/keepalive-prompt-composer.test.js, docs/keepalive/METRICS_SCHEMA.md, langsmith-fleet-worker-attempt.json
Adds applied bundle IDs, hashes, gate versions, and rejection reasons to metrics, parses bundle workflow inputs, records attempt metadata, and tests validation, loading, prompt composition, selection, and metrics output.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant KeepalivePromptComposer
  participant selectCapabilityBundles
  participant renderCapabilityFragments
  participant KeepaliveMetrics
  KeepalivePromptComposer->>selectCapabilityBundles: select bundles for context
  selectCapabilityBundles-->>KeepalivePromptComposer: applied and rejected bundles
  KeepalivePromptComposer->>renderCapabilityFragments: render applied fragments
  renderCapabilityFragments-->>KeepalivePromptComposer: capability prompt text
  KeepalivePromptComposer->>KeepaliveMetrics: record bundle IDs, hashes, gates, and rejection reasons
Loading

Suggested labels: codex-automation

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The added langsmith-fleet-worker-attempt.json file appears unrelated to the capability-bundle contract scope. Remove the metadata file or split it into a separate PR unless it is required for this contract change.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding the keepalive capability bundle contract.
Linked Issues check ✅ Passed The PR covers the new v1 contract, validation, selection, metrics, docs, tests, and consumer-template sync required by #2741.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/issue-2741-capability-bundle

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:09 — with GitHub Actions Inactive
@agents-workflows-bot

agents-workflows-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🤖 Keepalive Loop Status

PR #2744 | Agent: Codex | Iteration 0/12

Current State

Metric Value
Iteration progress [----------] 0/12
Action run (agent-run-failed)
Agent status ❌ AGENT FAILED
Gate success
Tasks 1/8 complete
Timeout 45 min (default)
Timeout usage 13m elapsed (30%, 32m remaining)
Keepalive ✅ enabled
Autofix ❌ disabled

Last Codex Run

Result Value
Status ❌ AGENT FAILED
Reason agent-run-failed
Exit code unknown
Failures 1/3 before pause

To retry immediately:

  • Add the agent:retry label to this PR

Or wait for the next successful Gate run to automatically retry.

🔍 Failure Classification

| Error type | infrastructure |
| Error category | transient |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

⚠️ Failure Tracking

| Consecutive failures | 1/3 |
| Reason | agent-run-failed |

@agents-workflows-bot

agents-workflows-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor
Keepalive Work Log (click to expand)
# Time (UTC) Agent Action Result Files Tasks Progress Commit Gate
0 2026-07-10 03:09:24 Codex wait (gate-cancelled-transient-transient) skipped 0 0/8 cancelled
0 2026-07-10 03:10:16 Codex run (agent-run-skipped) retry skipped 0 0/8 cancelled
0 2026-07-10 03:12:57 Codex run (agent-run-failed) retry failure 1 file(s) 0 0/8
0 2026-07-10 03:15:14 Codex run (agent-run-failed) failure 1 file(s) 0 0/8 success
0 2026-07-10 03:27:07 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 03:27:44 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 03:28:17 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 03:30:35 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 03:34:01 Codex skip (needs-human) retry skipped 0 0/8 success
0 2026-07-10 03:37:10 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8 success
0 2026-07-10 03:37:47 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 03:38:24 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 03:38:59 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 03:42:22 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 04:05:25 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 04:06:07 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 04:06:44 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 04:07:37 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 04:11:01 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 04:20:25 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 04:21:11 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 04:21:45 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 04:22:21 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 04:25:52 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 05:06:54 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 05:07:45 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 05:08:27 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 05:09:18 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 05:12:41 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 05:19:41 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 05:20:31 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 05:21:20 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 05:22:07 Codex skip (needs-human) skipped 0 0/8 cancelled
0 2026-07-10 05:24:45 Codex skip (needs-human) skipped 0 0/8 success
0 2026-07-10 05:31:51 Codex run (agent-run-failed-repeat) retry failure 1 file(s) 0 0/8
0 2026-07-10 05:32:42 Codex skip (needs-human) skipped 0 0/8
0 2026-07-10 05:33:30 Codex run (agent-run-skipped) retry skipped 0 0/8
0 2026-07-10 05:36:04 Codex run (agent-run-failed) failure 2 file(s) +1 1/8 f22870a cancelled
0 2026-07-10 05:38:24 Codex run (agent-run-failed) failure 2 file(s) 0 1/8 b360f9c cancelled
0 2026-07-10 05:40:56 Codex run (agent-run-failed-repeat) failure 2 file(s) 0 1/8 4c00995 cancelled
0 2026-07-10 05:41:39 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 05:42:13 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 05:42:51 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 05:44:21 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 05:45:55 Codex skip (needs-human) retry skipped 0 1/8
0 2026-07-10 05:46:40 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 06:07:40 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 1/8 b871fd6
0 2026-07-10 06:08:30 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 06:09:07 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 06:14:53 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 06:16:18 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 06:27:52 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 1/8 cce2502
0 2026-07-10 06:28:43 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 06:29:27 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 06:30:25 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 07:04:53 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 1/8 760b013
0 2026-07-10 07:05:33 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 07:06:21 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 07:07:09 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 07:12:12 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 07:14:22 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 07:37:21 Codex skip (needs-human) retry skipped 0 1/8 success
0 2026-07-10 08:06:58 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 1/8 c2d6124
0 2026-07-10 08:07:41 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 08:08:26 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 08:09:23 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 08:09:57 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 08:11:44 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 09:06:04 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 1/8 61caafd
0 2026-07-10 09:06:51 Codex skip (needs-human) skipped 0 1/8
0 2026-07-10 09:07:29 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 09:08:11 Codex skip (needs-human) skipped 0 1/8 cancelled
0 2026-07-10 09:10:28 Codex skip (needs-human) skipped 0 1/8 success
0 2026-07-10 09:33:44 Codex wait (gate-pending-transient) skipped 0 1/8
0 2026-07-10 09:46:41 Codex run (agent-run-failed) failure 1 file(s) 0 1/8 1616f70 success

@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:09 — with GitHub Actions Inactive
@stranske stranske added the agent:retry Add to trigger agent retry after rate limit or pause label Jul 10, 2026
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2744. Do not edit.

@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:10 — with GitHub Actions Inactive
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:10 — with GitHub Actions Inactive
@stranske-keepalive stranske-keepalive Bot removed the agent:retry Add to trigger agent retry after rate limit or pause label Jul 10, 2026
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:10 — with GitHub Actions Inactive
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:12 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the new capability-bundle/v1 contract to Workflows (and consumer templates) and introduces initial JS helpers to validate/select bundles, inject their fragments into keepalive prompt composition, and surface bundle evidence in keepalive metrics.

Changes:

  • Introduces capability-bundle/v1 schema + contract docs (root + consumer template).
  • Adds capability_bundle.js (root + consumer template) and integrates bundle selection/rendering into keepalive_prompt_composer.js.
  • Extends keepalive metrics shape to include applied bundle IDs/hashes and rejection reasons; adds contract tests and sync-manifest entries.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
docs/contracts/schemas/capability-bundle-v1.schema.json Adds the JSON Schema definition for capability-bundle/v1.
docs/contracts/capability-bundle-v1.md Documents the contract fields and hashing rules for bundles.
docs/keepalive/METRICS_SCHEMA.md Documents new keepalive metrics fields for capability bundle evidence.
.github/scripts/capability_bundle.js Implements bundle hashing, validation, selector matching, and fragment rendering.
.github/scripts/keepalive_prompt_composer.js Applies selected capability bundle fragments into composed keepalive prompts and returns selection results.
.github/scripts/keepalive_loop.js Extends metrics record construction to emit capability-bundle-related fields.
.github/scripts/__tests__/capability-bundle-contract.test.js Adds tests for bundle validation, hash mismatch rejection, prompt composition, and metrics emission behavior.
.github/sync-manifest.yml Declares new scripts/docs as centrally managed consumer-facing assets.
templates/consumer-repo/docs/contracts/schemas/capability-bundle-v1.schema.json Consumer-template copy of the schema.
templates/consumer-repo/docs/contracts/capability-bundle-v1.md Consumer-template copy of the contract docs.
templates/consumer-repo/.github/scripts/capability_bundle.js Consumer-template copy of bundle hashing/validation/selection helper.
templates/consumer-repo/.github/scripts/keepalive_prompt_composer.js Consumer-template copy of prompt composer integration.
templates/consumer-repo/.github/scripts/keepalive_loop.js Consumer-template copy of keepalive metrics field emission changes.

Comment thread .github/scripts/keepalive_loop.js
Comment thread .github/scripts/capability_bundle.js Outdated
Comment thread .github/scripts/capability_bundle.js
Comment thread docs/keepalive/METRICS_SCHEMA.md
Comment thread templates/consumer-repo/.github/scripts/keepalive_loop.js
Comment thread templates/consumer-repo/.github/scripts/capability_bundle.js Outdated
Comment thread templates/consumer-repo/.github/scripts/capability_bundle.js

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c098902e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/keepalive_prompt_composer.js
Comment thread .github/scripts/capability_bundle.js Outdated
Comment thread .github/scripts/capability_bundle.js Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/capability_bundle.js:
- Around line 71-126: Update validateCapabilityBundle to require non-empty owner
and rollback fields, alongside the existing fragments and gates checks,
rejecting bundles that omit either documented field. Add validation tests
covering bundles missing owner and rollback, and ensure valid bundles still
pass.

In @.github/scripts/keepalive_prompt_composer.js:
- Around line 34-47: Update the compose function to distinguish an omitted
params.capabilityBundles from an explicitly provided empty array. Use a presence
check (such as Array.isArray(params.capabilityBundles) or an own-property check)
so [] is passed to selectCapabilityBundles as the per-call override, while
absent or invalid values continue using the composer-level capabilityBundles
default.

In `@templates/consumer-repo/.github/scripts/capability_bundle.js`:
- Around line 128-134: loadCapabilityBundles currently drops plain object
documents when they lack a bundles property. Update its parsed-object handling
to either treat parsed itself as a single bundle or explicitly throw a
validation error, rather than passing parsed.bundles through asArray and
returning an empty list; preserve array and {bundles: [...]} support and use
validateCapabilityBundle for accepted single objects.
- Around line 71-126: Align validateCapabilityBundle with the companion schema
by validating capabilityId against ^[a-z0-9][a-z0-9._/-]*$, validating the
normalized version against ^v?[0-9]+(\.[0-9]+){0,2}$, and ensuring every item in
bundle.gates is a non-empty string after normalization. Preserve the existing
required-field and gate-array checks while rejecting values that violate these
constraints.
- Around line 30-41: Protect the capability bundle template and its rendered
fragment inputs from unauthorized edits: update the repository’s CODEOWNERS or
equivalent path-level access controls to require trusted owner review for this
bundle path and related fragments.task/fragments.acceptance files. Ensure
changes to these files cannot be merged without the designated owner approval,
rather than relying solely on computeCapabilityBundleHash and
validateCapabilityBundle.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4507bf44-69a7-4b1b-a556-44cced2f9809

📥 Commits

Reviewing files that changed from the base of the PR and between cb498d7 and 7c09890.

📒 Files selected for processing (13)
  • .github/scripts/__tests__/capability-bundle-contract.test.js
  • .github/scripts/capability_bundle.js
  • .github/scripts/keepalive_loop.js
  • .github/scripts/keepalive_prompt_composer.js
  • .github/sync-manifest.yml
  • docs/contracts/capability-bundle-v1.md
  • docs/contracts/schemas/capability-bundle-v1.schema.json
  • docs/keepalive/METRICS_SCHEMA.md
  • templates/consumer-repo/.github/scripts/capability_bundle.js
  • templates/consumer-repo/.github/scripts/keepalive_loop.js
  • templates/consumer-repo/.github/scripts/keepalive_prompt_composer.js
  • templates/consumer-repo/docs/contracts/capability-bundle-v1.md
  • templates/consumer-repo/docs/contracts/schemas/capability-bundle-v1.schema.json

Comment thread .github/scripts/keepalive_prompt_composer.js
Comment thread templates/consumer-repo/.github/scripts/capability_bundle.js
Comment thread templates/consumer-repo/.github/scripts/capability_bundle.js
Comment thread templates/consumer-repo/.github/scripts/capability_bundle.js
@stranske stranske added the agent:retry Add to trigger agent retry after rate limit or pause label Jul 10, 2026
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:24 — with GitHub Actions Inactive
@stranske
stranske temporarily deployed to agent-standard July 10, 2026 03:24 — with GitHub Actions Inactive
@stranske-keepalive stranske-keepalive Bot removed the agent:retry Add to trigger agent retry after rate limit or pause label Jul 10, 2026
@stranske-keepalive
stranske-keepalive Bot temporarily deployed to agent-high-privilege July 10, 2026 04:21 Inactive
@stranske stranske added agent:retry Add to trigger agent retry after rate limit or pause and removed agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Jul 10, 2026
@stranske

Copy link
Copy Markdown
Owner Author

Closer review-fix pushed in c3691657.

Addressed the current unresolved review set:

  • capability_bundle.js now enforces schema-required owner and rollback, and loadCapabilityBundles() accepts a standalone capability-bundle/v1 document as well as arrays / {bundles:[...]} wrappers.
  • keepalive_prompt_composer.js now respects an explicit empty per-call capabilityBundles: [] override instead of falling back to default bundles.
  • keepalive_loop.js now carries selected capability bundle metadata into metrics through capability_bundles_json / capabilityBundlesJson, and includes both gate refs and playbook refs in capability_gate_versions.
  • Root and templates/consumer-repo script copies were kept in sync; metrics docs/examples now describe gate refs plus playbook refs.

Validation:

  • node --test .github/scripts/__tests__/keepalive-loop.test.js .github/scripts/__tests__/capability-bundle-contract.test.js .github/scripts/__tests__/keepalive-prompt-composer.test.js -> 125 passed.
  • python -m pytest -q tests/scripts/test_validate_workflow_yaml.py tests/workflows/test_sync_manifest_delivery.py tests/workflows/test_workflow_agents_consolidation.py -> 73 passed.
  • python scripts/validate_template_completeness.py -> passed.
  • git diff --check -> passed.

I also removed stale automation blocker labels (needs-human, agent:needs-attention) and added agent:retry; fresh checks are expected after this push.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/capability_bundle.js:
- Around line 77-134: Tighten validateCapabilityBundle to match the schema by
rejecting unknown top-level properties, enforcing the schema’s capability_id and
version patterns, and expanding FORBIDDEN_KEY_PATTERN to detect
command/control/exec-style inline fields. Use the existing bundle validation and
walkForbiddenKeys logic, preserving current error handling and hash validation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6816e1ad-91fa-4811-a08d-421fe62832d2

📥 Commits

Reviewing files that changed from the base of the PR and between 7c09890 and 6b39895.

📒 Files selected for processing (15)
  • .github/scripts/__tests__/capability-bundle-contract.test.js
  • .github/scripts/__tests__/keepalive-prompt-composer.test.js
  • .github/scripts/capability_bundle.js
  • .github/scripts/keepalive_loop.js
  • .github/scripts/keepalive_prompt_composer.js
  • .github/sync-manifest.yml
  • docs/contracts/capability-bundle-v1.md
  • docs/contracts/schemas/capability-bundle-v1.schema.json
  • docs/keepalive/METRICS_SCHEMA.md
  • langsmith-fleet-worker-attempt.json
  • templates/consumer-repo/.github/scripts/capability_bundle.js
  • templates/consumer-repo/.github/scripts/keepalive_loop.js
  • templates/consumer-repo/.github/scripts/keepalive_prompt_composer.js
  • templates/consumer-repo/docs/contracts/capability-bundle-v1.md
  • templates/consumer-repo/docs/contracts/schemas/capability-bundle-v1.schema.json

Comment thread .github/scripts/capability_bundle.js
@stranske

Copy link
Copy Markdown
Owner Author

Closer update for the latest CodeRabbit validation thread:

  • Tightened .github/scripts/capability_bundle.js and the consumer-template copy to reject unknown top-level fields, enforce the schema capability_id and version patterns, require non-empty gate refs, and catch command/control/exec-style inline fields through the forbidden-key walker.
  • Added focused contract coverage for schema-pattern drift, unknown top-level fields, nested unsafe prompt/exec fields, and blank gates.

Validation passed locally on head 564399c2:

  • node --test .github/scripts/__tests__/capability-bundle-contract.test.js .github/scripts/__tests__/keepalive-prompt-composer.test.js .github/scripts/__tests__/keepalive-loop.test.js -> 127 passed
  • python -m pytest -q tests/scripts/test_validate_workflow_yaml.py tests/workflows/test_sync_manifest_delivery.py tests/workflows/test_workflow_agents_consolidation.py -> 73 passed
  • python scripts/validate_template_completeness.py -> passed
  • git diff --check -> passed

The remaining blocker should be fresh async GitHub checks/review settling on the new head.

@github-actions

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.4 CONCERNS 82% The merged code appears strong on the core contract work: it adds a versioned capability-bundle v1 schema and docs, extends agent registry/selection logic, integrates deterministic capability selec...
anthropic claude-sonnet-4-6 CONCERNS 72% The PR adds the core capability-bundle infrastructure (schema JSON, capability_bundle.js, contract tests, consumer-repo mirrors, metrics schema update) that addresses the acceptance criteria direct...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.4
  • Verdict: CONCERNS
  • Confidence: 82%
  • Scores:
    • Correctness: 8.0/10
    • Completeness: 6.0/10
    • Quality: 8.0/10
    • Testing: 8.0/10
    • Risks: 7.0/10
  • Summary: The merged code appears strong on the core contract work: it adds a versioned capability-bundle v1 schema and docs, extends agent registry/selection logic, integrates deterministic capability selection into keepalive prompt composition and loop handling, and adds solid contract-focused tests for invalid IDs, hash mismatch, unsafe prompt/local routing restrictions, and matching/nonmatching behavior. Code quality and test coverage for the core bundle contract look good. However, the PR does not clearly satisfy the full acceptance scope around expanded Keepalive metrics/state and explicit freshness tests for divergence between root and consumer-template copies. Because the central feature is implemented but some acceptance criteria appear only partially addressed, the best verdict is CONCERNS rather than PASS.
  • Concerns:
    • The PR substantially implements the new neutral capability-bundle contract, schema, validation helpers, deterministic selection, and runner/prompt-composer integration, but the acceptance criteria appear only partially complete.
    • The task/acceptance criteria require Keepalive metrics/state to include matched/applied capability IDs, bundle/hash, gate versions, rejection reasons, and downstream result. The diff summary shows only a small metrics schema doc change and limited keepalive loop/composer changes; based on the changed files, this looks underimplemented relative to the full metrics/state extension requirement.
    • Freshness/divergence protection for root vs consumer template is only partially evidenced. The template files and sync manifest were updated, but there is no clear indication from the changed files that dedicated freshness tests were added to ensure failures when schema/runner integration diverges.
    • The acceptance criterion mentioning a deliberate hash-mismatch break/revert is represented by tests for hash validation, but a post-merge code review cannot verify the required observed failure demonstration itself; only the presence of the test logic is evident.
    • There is some risk of duplication drift because capability bundle logic was copied into templates/consumer-repo rather than imported from a shared source; sync-manifest helps, but long-term maintenance risk remains.

anthropic

  • Model: claude-sonnet-4-6
  • Verdict: CONCERNS
  • Confidence: 72%
  • Scores:
    • Correctness: 6.0/10
    • Completeness: 5.0/10
    • Quality: 7.0/10
    • Testing: 6.0/10
    • Risks: 6.0/10
  • Summary: The PR adds the core capability-bundle infrastructure (schema JSON, capability_bundle.js, contract tests, consumer-repo mirrors, metrics schema update) that addresses the acceptance criteria directionally. However, three of four task checkboxes remain unchecked in both the PR and issue, an unrelated file (langsmith-fleet-worker-attempt.json) was included, and the freshness-test failure guarantee (AC3) is not clearly demonstrated in the diff. The deliberate-break test (AC4) appears structurally present but cannot be fully verified without the complete file contents. The implementation is partially complete with meaningful gaps in completeness verification.
  • Concerns:
    • Three of four tasks remain unchecked in both the PR and issue checklists (schema addition, metrics extension, consumer-template freshness), yet the diff appears to include files for them - the disconnect between task status and diff content creates ambiguity about whether implementations are complete or partial stubs.
    • langsmith-fleet-worker-attempt.json is added with no clear relation to the capability-bundle contract scope; this appears to be an accidental or out-of-scope file inclusion.
    • AC3 (freshness tests fail when bundle schema/runner integration diverges) is only partially addressed: sync-manifest.yml is updated but no explicit test asserting failure-on-divergence is clearly present in the diff summary.
    • AC1 requires rejection of 'local routing weight' and 'unsafe inline prompt' - without seeing the full capability_bundle.js implementation, it's unclear whether all five rejection cases are implemented and tested.
    • The deliberate-break AC4 scenario cannot be fully verified from the diff alone - the test file exists and references the correct test name, but the exact assertion message 'invalid capability hash reached prompt composition' must be present in the implementation.
    • METRICS_SCHEMA.md updated (+5/-1) but it's unclear if matched/applied capability IDs, bundle/hash, gate versions, rejection reasons, and downstream result are all captured as required by AC2.
    • keepalive-model-profile-contract.test.js (104 lines) was added but is not referenced in the acceptance criteria - its relationship to the scope is unclear and may indicate scope creep.

Agreement

  • Verdict: CONCERNS (all providers)
  • Completeness: scores within 1 point (avg 5.5/10, range 5.0-6.0)
  • Quality: scores within 1 point (avg 7.5/10, range 7.0-8.0)
  • Risks: scores within 1 point (avg 6.5/10, range 6.0-7.0)

Disagreement

Dimension openai anthropic
Correctness 8.0/10 6.0/10
Testing 8.0/10 6.0/10

Unique Insights

  • openai: The PR substantially implements the new neutral capability-bundle contract, schema, validation helpers, deterministic selection, and runner/prompt-composer integration, but the acceptance criteria appear only partially complete.; The task/acceptance criteria require Keepalive metrics/state to include matched/applied capability IDs, bundle/hash, gate versions, rejection reasons, and downstream result. The diff summary shows only a small metrics schema doc change and limited keepalive loop/composer changes; based on the changed files, this looks underimplemented relative to the full metrics/state extension requirement.; Freshness/divergence protection for root vs consumer template is only partially evidenced. The template files and sync manifest were updated, but there is no clear indication from the changed files that dedicated freshness tests were added to ensure failures when schema/runner integration diverges.; The acceptance criterion mentioning a deliberate hash-mismatch break/revert is represented by tests for hash validation, but a post-merge code review cannot verify the required observed failure demonstration itself; only the presence of the test logic is evident.; There is some risk of duplication drift because capability bundle logic was copied into templates/consumer-repo rather than imported from a shared source; sync-manifest helps, but long-term maintenance risk remains.
  • anthropic: Three of four tasks remain unchecked in both the PR and issue checklists (schema addition, metrics extension, consumer-template freshness), yet the diff appears to include files for them - the disconnect between task status and diff content creates ambiguity about whether implementations are complete or partial stubs.; langsmith-fleet-worker-attempt.json is added with no clear relation to the capability-bundle contract scope; this appears to be an accidental or out-of-scope file inclusion.; AC3 (freshness tests fail when bundle schema/runner integration diverges) is only partially addressed: sync-manifest.yml is updated but no explicit test asserting failure-on-divergence is clearly present in the diff summary.; AC1 requires rejection of 'local routing weight' and 'unsafe inline prompt' - without seeing the full capability_bundle.js implementation, it's unclear whether all five rejection cases are implemented and tested.; The deliberate-break AC4 scenario cannot be fully verified from the diff alone - the test file exists and references the correct test name, but the exact assertion message 'invalid capability hash reached prompt composition' must be present in the implementation.; METRICS_SCHEMA.md updated (+5/-1) but it's unclear if matched/applied capability IDs, bundle/hash, gate versions, rejection reasons, and downstream result are all captured as required by AC2.; keepalive-model-profile-contract.test.js (104 lines) was added but is not referenced in the acceptance criteria - its relationship to the scope is unclear and may indicate scope creep.

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a neutral capability bundle contract to Workflows/Keepalive

3 participants