Issue #2742: Propagate keepalive execution profiles - #2746
Merged
Conversation
stranske
added
agent:codex
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 4 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (11)
📝 WalkthroughWalkthroughAdds Codex execution profiles to agent registries, validates and resolves selected profiles during keepalive evaluation, and propagates profile and model fallback data through the keepalive and reusable Codex workflows with contract tests. ChangesExecution profile routing
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 04:11 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
🤖 Keepalive Loop StatusPR #2746 | Agent: Codex | Iteration 0/12 Current State
🔍 Failure Classification| Error type | infrastructure |
|
Contributor
Keepalive Work Log (click to expand)
|
Contributor
|
Runner dispatch state for codex on PR #2746. Do not edit. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b29a48a06c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 04:16 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Propagates a registry-backed “execution profile” through the keepalive evaluation and into the Codex reusable runner so that the requested worker model (and fallback chain) can be validated early and forwarded into the remote Codex invocation, while also clarifying that config/model_registry.json / config/llm_slots.json are evaluator/judge catalogs (not worker profiles).
Changes:
- Add
execution_profilesto the agent registry plus JS helpers to validate/resolve them, and plumb the resolved profile through keepalive evaluation outputs. - Forward
execution_profile,codex_model, andcodex_fallback_modelsintoreusable-codex-run.yml, and expose worker model/profile telemetry as reusable-workflow outputs. - Add contract/unit tests covering profile resolution and keepalive→reusable forwarding, and annotate model/slot JSON configs with “purpose” fields.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
.github/agents/registry.yml |
Introduces execution_profiles (codex-default / codex-fast) in the canonical registry. |
.github/scripts/agent_registry.js |
Adds validation + resolveExecutionProfile() to load and enforce profile contracts. |
.github/scripts/keepalive_loop.js |
Normalizes execution_profile, resolves/validates it during keepalive evaluation, and surfaces it in results. |
.github/workflows/agents-keepalive-loop.yml |
Adds workflow input and forwards resolved profile/model/fallback outputs into the Codex reusable call. |
.github/workflows/reusable-codex-run.yml |
Adds inputs for fallback models + execution profile ID and exposes worker telemetry outputs; wires fallback models into model-resolution env. |
.github/scripts/__tests__/keepalive-model-profile-contract.test.js |
Adds a contract test to prevent accidental removal of the new forwarding. |
.github/scripts/__tests__/agent-registry.test.js |
Adds tests for profile resolution and unknown-profile rejection. |
config/model_registry.json |
Adds a “purpose” field clarifying it is an auxiliary judge/evaluator catalog. |
config/llm_slots.json |
Adds a “purpose” field clarifying slots are evaluator/judge-only. |
templates/consumer-repo/.github/agents/registry.yml |
Mirrors registry execution profile additions in the consumer template. |
templates/consumer-repo/.github/scripts/agent_registry.js |
Mirrors profile validation/resolution helpers in the consumer template. |
templates/consumer-repo/.github/scripts/keepalive_loop.js |
Mirrors keepalive execution-profile normalization/resolution in the consumer template. |
templates/consumer-repo/config/model_registry.json |
Mirrors the “purpose” clarification in the consumer template. |
templates/consumer-repo/config/llm_slots.json |
Mirrors the “purpose” clarification in the consumer template. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/reusable-codex-run.yml (1)
727-759: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winFallback model chain is dropped for any profile besides
codex-default.
candidatesonly picks up$FALLBACK_CODEX_MODELSwhen"$model" = "$DEFAULT_CODEX_MODEL"(the hardcodedgpt-5.5) or when the model matches*-codex*. For the registry'scodex-fastprofile (model=gpt-5.4,fallback_model=gpt-5.5),codex_modelresolves togpt-5.4, which matches neither branch, socandidatesstays just"gpt-5.4"— the profile's fallback model is never tried if the primary model is unavailable. This silently defeats the fallback propagation this PR is meant to add for any non-default execution profile.🐛 Proposed fix: always append the resolved fallback chain
candidates="$model" - if [ "$model" = "$DEFAULT_CODEX_MODEL" ]; then - candidates="$DEFAULT_CODEX_MODEL $FALLBACK_CODEX_MODELS" - fi + if [ -n "${FALLBACK_CODEX_MODELS:-}" ]; then + candidates="$model $FALLBACK_CODEX_MODELS" + fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/reusable-codex-run.yml around lines 727 - 759, Always append FALLBACK_CODEX_MODELS to candidates after resolving the requested or default model, rather than only when model equals DEFAULT_CODEX_MODEL or matches the unsupported *-codex* branch. Update the candidates construction in the “Resolve Codex run model” step so profiles such as codex-fast retain their primary model followed by the configured fallback chain, while preserving the specialized-model normalization behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/agents/registry.yml:
- Around line 8-25: validateExecutionProfile() must verify that each execution
profile’s model and fallback_model exists in config/model_registry.json instead
of only checking non-empty values. Load or reuse the model registry during
validation, reject unknown model identifiers with a clear validation error, and
preserve the existing runner and other field checks.
In @.github/scripts/__tests__/keepalive-model-profile-contract.test.js:
- Around line 14-53: Add a runtime-focused test alongside the existing workflow
contract tests that exercises the “Resolve Codex run model” candidate-building
logic in reusable-codex-run.yml, covering a non-default requested model with
configured fallback models and asserting the fallback chain is applied. Reuse
the workflow’s actual shell logic or extract it into a testable helper, and
ensure the test would fail if fallbacks are only applied when the request equals
the hardcoded default.
In @.github/scripts/keepalive_loop.js:
- Around line 2840-2858: Resolve and validate the execution profile only for
actions that dispatch work, such as run, fix, or conflict, rather than
unconditionally for every keepalive action. Update the execution-profile block
around requestedExecutionProfile and the dispatch logic near its later use so
skip, wait, stop, defer, and review paths bypass it. Replace the hard throw in
the profile-resolution catch with graceful degradation consistent with the
existing agent-routing try/catch, logging the error and returning the
appropriate non-dispatch result instead of escaping evaluateKeepaliveLoop.
In @.github/workflows/agents-keepalive-loop.yml:
- Around line 34-38: Add the execution_profile workflow input to the consumer
template’s agents-keepalive-loop workflow, matching the definition in the source
workflow: description, optional required flag, codex-default value, and string
type.
In `@templates/consumer-repo/.github/scripts/keepalive_loop.js`:
- Around line 2840-2858: Update the execution-profile handling around
requestedExecutionProfile and resolveExecutionProfile so it only runs for run,
fix, and conflict actions; skip profile resolution and validation for skip,
wait, and stop paths. Make profile errors non-fatal for those non-execution
actions, and apply the identical control flow and error handling used to fix the
main keepalive_loop.js copy so both implementations remain synchronized.
---
Outside diff comments:
In @.github/workflows/reusable-codex-run.yml:
- Around line 727-759: Always append FALLBACK_CODEX_MODELS to candidates after
resolving the requested or default model, rather than only when model equals
DEFAULT_CODEX_MODEL or matches the unsupported *-codex* branch. Update the
candidates construction in the “Resolve Codex run model” step so profiles such
as codex-fast retain their primary model followed by the configured fallback
chain, while preserving the specialized-model normalization behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 17940123-00f4-4d97-b551-a38d2eb41fa4
📒 Files selected for processing (14)
.github/agents/registry.yml.github/scripts/__tests__/agent-registry.test.js.github/scripts/__tests__/keepalive-model-profile-contract.test.js.github/scripts/agent_registry.js.github/scripts/keepalive_loop.js.github/workflows/agents-keepalive-loop.yml.github/workflows/reusable-codex-run.ymlconfig/llm_slots.jsonconfig/model_registry.jsontemplates/consumer-repo/.github/agents/registry.ymltemplates/consumer-repo/.github/scripts/agent_registry.jstemplates/consumer-repo/.github/scripts/keepalive_loop.jstemplates/consumer-repo/config/llm_slots.jsontemplates/consumer-repo/config/model_registry.json
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 04:27 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Owner
Author
|
Closer review-fix update for #2746 pushed in commits Addressed review findings:
Validation run locally:
|
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 04:31 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 04:32 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
agents-workflows-bot
Bot
added
agent:needs-attention
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 05:06 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
🤖 Bot Comment Handler
The agent has been assigned to this PR to address the bot review comments. Instructions for agent
The bot comment handler workflow has prepared context in the artifacts. |
stranske
added
agent:retry
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 05:17 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
agents-workflows-bot
Bot
added
agent:needs-attention
stranske
removed
needs-human
stranske
temporarily deployed
to
agent-high-privilege
July 10, 2026 05:31 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Closed
10 tasks
stranske
had a problem deploying
to
agent-standard
July 10, 2026 05:32 — with
GitHub Actions
Failure
GitHub Actions
Failure
agents-workflows-bot
Bot
added
agent:needs-attention
Contributor
Provider Comparison ReportProvider Summary
📋 Full Provider Details (click to expand)openai
anthropic
Agreement
Disagreement
Unique Insights
🔍 LangSmith Traces |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2742
Automated Status Summary
Scope
Workflows is close to supporting remote model choice but does not wire it through Keepalive. On verified
origin/main,.github/workflows/reusable-codex-run.yml:55-64acceptscodex_model, resolves/falls back it at:700-746, runs Codex with--modelat:925-979, and emits selected-model details at:990-1000. The Keepalive caller at.github/workflows/agents-keepalive-loop.yml:687-715does not pass that input. Routing remains agent-label only (docs/keepalive/MULTI_AGENT_ROUTING.md:10-21), and.github/agents/registry.ymlhas no registry-backed execution profile.Missing behavior: a local provider/model decision cannot reach the remote coding worker or return causally valid requested/resolved worker identity; evaluator telemetry can currently be mistaken for worker identity.
Tasks
.github/agents/registry.ymlor a companion versioned registry, including exact model, runner, capacity-pool references, safety defaults, and lifecycle state..github/workflows/agents-keepalive-loop.ymland pass the corresponding exact model into.github/workflows/reusable-codex-run.yml; reject unknown profiles before worker execution.langsmith-fleet/v1artifact to emit operation roleworker, profile ID, requested model, selected/resolved model, fallback reason, and runner/CLI version separately from evaluator/verifier traces.config/model_registry.jsonandconfig/llm_slots.jsondocumentation so auxiliary judge models are not confused with coding-worker profiles.templates/consumer-repoworkflow/registry copies and Workflows freshness/sync tests from the canonical source.Acceptance criteria
reusable-codex-run.ymland its exact model reaches the Codex command..github/workflows/agents-keepalive-loop.yml:687-715, delete only the newexecution_profile/codex_modelforwarding line; run exact test.github/scripts/__tests__/keepalive-model-profile-contract.test.js::profile reaches codex runnervianode --test .github/scripts/__tests__/keepalive-model-profile-contract.test.js --test-name-pattern='profile reaches codex runner'; observeAssertionError: expected selected profile model in reusable runner inputs; revert and show the exact test passes.Summary by CodeRabbit
New Features
Bug Fixes
Documentation