feat(llm): run pipeline stages on a ChatGPT plan via the Codex CLI - #67
Merged
Conversation
Provider routing assumed every provider authenticates through one environment
variable. Codex authenticates from a file-backed CLI login instead, so the
question "what does this model need before it can run?" becomes a tagged
ProviderRequirement union rather than another nullable string. Every caller
reads that union; there is no second copy of the check.
ensureManagedCodexHome exists because `-c 'mcp_servers={}'` does not work: -c
merges into the config table rather than replacing it, so an operator's MCP
servers still boot and spam transport errors on every call. A private
CODEX_HOME is the only lever that isolates config, and it symlinks the
operator's auth.json so the existing login is reused without this repo ever
reading the token.
Widening LlmProvider forced two exhaustive Records open. PROVIDER_LABEL gains
its codex entry and ModelReadiness widens envVar to string | null, both of
which the readiness phase needed anyway. Fingerprints for non-codex workspaces
are unchanged, so no existing verification run goes stale.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@openai/codex-sdk is pinned exact because it pins its own @openai/codex, and the two must agree for the vendored binary to resolve. It is declared in both workspaces, matching how the Anthropic and OpenAI SDKs are declared. serverExternalPackages because the SDK resolves a native binary relative to its own package directory, which bundling would move it away from. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The six ids are the subset the ChatGPT plan actually serves through Codex, read from ~/.codex/models_cache.json. It is narrower than the API catalog and moves independently, so the list is named and tested rather than inferred. Entries are generated from the OpenAI rows rather than hand-copied, so a Codex mirror can never drift from the price it estimates. The existing "prices every catalog entry" invariant could not have caught that drift: PRICES is built from LLM_CATALOG, so it is self-consistent by construction. The new test compares a mirror against the entry it mirrors, which is a claim the old one could not make. The migration ships with the catalog because the dropdown values are Postgres enum labels. Offering an option the database rejects on save is not a shippable state. ADD VALUE IF NOT EXISTS keeps it idempotent against a dev-push race, and down is a no-op because Postgres cannot drop an enum value. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One thread per call against the local Codex CLI, in a read-only sandbox with a scratch working directory and approvals disabled. The SDK import is dynamic so mock-mode processes and unit tests never load it. Two measured behaviours drive the shape. An item of type error appears in items even on fully successful turns, so only the outcome of run() decides success. Supplying env to the Codex constructor replaces the child environment rather than extending it, so the whole of process.env is spread in alongside CODEX_HOME. Callers compose their own JSON instruction; the adapter adds none, because llm.ts and brandVoiceExtract.ts each already carry one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The provider ternary existed twice, in completeJSON and again inside createLlmClient. Adding a third provider to both copies would have invited them to drift, so they collapse into one exhaustive switch. parseJsonReply stays outside the Codex try/catch. The Anthropic and OpenAI catches only re-map known SDK error classes, but the Codex one is a catch-all, so wrapping the parse would stamp the stage into the message twice. loadStageModels branches on the requirement kind rather than interpolating one template, because "needs `codex login` set" is not a sentence. The env message is unchanged to the byte. The login check is memoised as a single promise, so six codex stages cost one process and mock mode costs none. A Codex login deliberately does not flip mock mode to live. Mock is the local default and a dev machine that happens to carry a login must not start making paid calls because of it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
extractionMockMode asked whether the model had an API key, which made every
codex model look unconfigured and silently mock. It now asks the requirement
what it needs. It still never throws, so a keyless dev environment keeps working.
Readiness gains needsCodexLogin, kept out of `missing` because that list is
environment variable names and a login is not one. The config fingerprint now
hashes the login state; it previously read a nonexistent `codex-login` variable
and so never changed when the login did. A pinned hash proves the fingerprint
for a workspace with no codex stage is byte-identical, so no existing
verification run goes stale.
Admin renders use the file check rather than spawning the CLI per request. The
authoritative check runs once per pipeline run.
Also fixes a pre-existing rendering bug in the banner. missing.join('</code>,
<code>') inside JSX is escaped rather than parsed, so two missing variables
rendered as literal markup. This change makes the banner appear in more cases,
which is a poor moment to leave that in place.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every figure is measured rather than estimated. The per-call overhead of about 15k input tokens is stated plainly because it is the strongest reason to keep API keys as the default and Codex as an opt-in. The managed CODEX_HOME is documented with its reason. Without it an operator's own Codex config applies to every call, which on the machine this was built on meant 11 MCP servers booting and a notify hook launching a desktop app per call. The trusted-host boundary is stated rather than implied. OpenAI has not said whether embedding Codex with a ChatGPT login in a hosted product is permitted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0790eabf9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
extractionMockMode treated the presence of auth.json as activation, so with MOCK_MODE unset a brand-guide upload made a live call and billed plan quota while pipeline config and workspace readiness both still reported mock. The rule this repo already states is that a Codex login is not consent to spend. config.ts and modeFromEnv both honour it. This one did not, and the disagreement was the bug. Extraction now needs MOCK_MODE=false explicitly, like every other live path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
runtime.missing holds environment variable names, and a login is not one, so a workspace blocked only by a missing Codex login produced "Configure the required environment variables: ." in three actions. runtime.blockers carries everything unmet in the words an operator acts on. missing keeps its meaning for the callers that want variable names, including the fingerprint and the run snapshot. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Operators pay per token for every pipeline stage. Many already hold a ChatGPT plan that includes Codex, so this adds a third provider that spends that plan instead of an API key.
The prefix carries the routing. A model id of
codex/gpt-5.6-terraselects the provider, which preserves the module's stated invariant that the id alone decides the provider. The alternative, a separate provider column on the catalog, adds a second thing to keep in sync with the id.Scope
cms/src/lib/llmProvider.tsgains a thirdLlmProviderand aProviderRequirementtagged union. The old shape assumed every provider authenticates through one environment variable, which Codex does not.requirementForModelis now the single branch point;envVarNameForModelandapiKeyForModelread its result and returnundefinedfor Codex.cms/src/lib/codexAuth.tsis new.checkCodexLoginshellscodex login status.ensureManagedCodexHomebuilds a privateCODEX_HOMEholding a minimal config plus a symlink to the operator'sauth.json. Nothing reads the token.cms/src/lib/codexCompletion.tsis new. One thread per call, read-only sandbox, scratch working directory, approvals off, SDK imported dynamically.pipeline/src/llm.tscollapses the provider ternary that existed twice, atcompleteJSONand insidecreateLlmClient, into one exhaustivecompleteJSONLiveswitch, and addscompleteJSONCodex.pipeline/src/models.tsbranches the preflight on requirement kind. The environment message is unchanged to the byte.pipeline/src/config.tsaccepts a Codex login as a third credential but does not let one flip mock mode to live.cms/src/lib/llmCatalog.tsgenerates sixcodex/entries from the OpenAI rows so a mirror cannot drift from the price it estimates. Migration20260902_210000_codex_model_optionsadds those six labels to seven Postgres enums.cms/src/lib/brandVoiceExtract.ts,workspaceReadiness.tsand the runtime banner learn the new requirement. Out of scope: hosted multi-tenant use, and any storage of credentials.Tradeoffs
Codex is an agent runtime, not a completion endpoint. A trivial prompt billed 15,767 input tokens because it loads its own preamble every call. Roughly 15k of overhead per stage is the price of the plan billing, so API keys stay the default and this is opt-in.
Cost rows are estimates at API rates, marked
provider = codex. The plan bills in its own included usage, so the dollar figure is indicative. The alternative, logging zero, would have made the reports lie by omission.Each call runs in a managed
CODEX_HOMErather than the operator's own.-c 'mcp_servers={}'was measured and does not work, because-cmerges into the config table rather than replacing it.Blast Radius
LlmProvideris shared by both workspaces and by the CMS content-run job, so widening it touched two exhaustiveRecordtypes. Existing Anthropic and OpenAI paths are untouched at runtime; the only change to them is that one duplicated ternary became one switch.The migration only adds enum labels. Postgres cannot drop them, so
downis a no-op.ADD VALUE IF NOT EXISTSkeeps it idempotent.The readiness fingerprint is the one silent risk, since a change would stale every prior verification run. A pinned hash test proves a workspace with no Codex stage fingerprints byte-identically.
Verification
Typecheck and lint clean across both workspaces, 0 errors.
621 pipeline unit tests and 305 CMS integration tests pass, including new suites for provider routing, the auth helpers, the adapter, the client switch, the preflight gate, and readiness.
End to end against the real product, not the harness. Saving
codex/gpt-5.6-terrainto the Models global succeeds, which exercises the enum migration. The preflight with no login throws before any stage runs and namescodex login. Mock mode with acodex/model returns the fixture and never loads the SDK.costUsd('codex/gpt-5.6-terra', 1e6, 1e6)returns 14.Two live probes against the real CLI confirmed the managed home. With the operator's own config, stderr carried repeated
rmcp::transport::workerfailures. With the managed home, stderr was clean, nonotifyprocess launched, and the agent returned exactly{"ok":true}.Live, against the operator's ChatGPT plan.
All six catalog ids answer and return valid JSON. Input-token overhead measured per call: sol 17322, terra 17487, luna 15920, 5.5 16527, 5.4 15172, 5.4-mini 14824.
A full
completeJSONLoggedcall oncodex/gpt-5.6-lunawrote this row, which is the single call site every stage routes through:The cost is exact against the catalog rate, 15955 at $0.2/1M plus 17 at $1.2/1M. No
notifyprocess launched.The web-search path returned
webSearchRequests: 1with a real source URL, confirmingweb_searchitems are counted andwebSearchModereaches the CLI.