Skip to content

feat(llm): run pipeline stages on a ChatGPT plan via the Codex CLI - #67

Merged
snowopsdev merged 9 commits into
mainfrom
codex-provider
Sep 2, 2026
Merged

feat(llm): run pipeline stages on a ChatGPT plan via the Codex CLI#67
snowopsdev merged 9 commits into
mainfrom
codex-provider

Conversation

@snowopsdev

@snowopsdev snowopsdev commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Why

Operators pay per token for every pipeline stage. Many already hold a ChatGPT plan that includes Codex, so this adds a third provider that spends that plan instead of an API key.

The prefix carries the routing. A model id of codex/gpt-5.6-terra selects the provider, which preserves the module's stated invariant that the id alone decides the provider. The alternative, a separate provider column on the catalog, adds a second thing to keep in sync with the id.

Scope

cms/src/lib/llmProvider.ts gains a third LlmProvider and a ProviderRequirement tagged union. The old shape assumed every provider authenticates through one environment variable, which Codex does not. requirementForModel is now the single branch point; envVarNameForModel and apiKeyForModel read its result and return undefined for Codex.

cms/src/lib/codexAuth.ts is new. checkCodexLogin shells codex login status. ensureManagedCodexHome builds a private CODEX_HOME holding a minimal config plus a symlink to the operator's auth.json. Nothing reads the token.

cms/src/lib/codexCompletion.ts is new. One thread per call, read-only sandbox, scratch working directory, approvals off, SDK imported dynamically.

pipeline/src/llm.ts collapses the provider ternary that existed twice, at completeJSON and inside createLlmClient, into one exhaustive completeJSONLive switch, and adds completeJSONCodex.

pipeline/src/models.ts branches the preflight on requirement kind. The environment message is unchanged to the byte. pipeline/src/config.ts accepts a Codex login as a third credential but does not let one flip mock mode to live.

cms/src/lib/llmCatalog.ts generates six codex/ entries from the OpenAI rows so a mirror cannot drift from the price it estimates. Migration 20260902_210000_codex_model_options adds those six labels to seven Postgres enums.

cms/src/lib/brandVoiceExtract.ts, workspaceReadiness.ts and the runtime banner learn the new requirement. Out of scope: hosted multi-tenant use, and any storage of credentials.

Tradeoffs

Codex is an agent runtime, not a completion endpoint. A trivial prompt billed 15,767 input tokens because it loads its own preamble every call. Roughly 15k of overhead per stage is the price of the plan billing, so API keys stay the default and this is opt-in.

Cost rows are estimates at API rates, marked provider = codex. The plan bills in its own included usage, so the dollar figure is indicative. The alternative, logging zero, would have made the reports lie by omission.

Each call runs in a managed CODEX_HOME rather than the operator's own. -c 'mcp_servers={}' was measured and does not work, because -c merges into the config table rather than replacing it.

Blast Radius

LlmProvider is shared by both workspaces and by the CMS content-run job, so widening it touched two exhaustive Record types. Existing Anthropic and OpenAI paths are untouched at runtime; the only change to them is that one duplicated ternary became one switch.

The migration only adds enum labels. Postgres cannot drop them, so down is a no-op. ADD VALUE IF NOT EXISTS keeps it idempotent.

The readiness fingerprint is the one silent risk, since a change would stale every prior verification run. A pinned hash test proves a workspace with no Codex stage fingerprints byte-identically.

Verification

Typecheck and lint clean across both workspaces, 0 errors.

621 pipeline unit tests and 305 CMS integration tests pass, including new suites for provider routing, the auth helpers, the adapter, the client switch, the preflight gate, and readiness.

End to end against the real product, not the harness. Saving codex/gpt-5.6-terra into the Models global succeeds, which exercises the enum migration. The preflight with no login throws before any stage runs and names codex login. Mock mode with a codex/ model returns the fixture and never loads the SDK. costUsd('codex/gpt-5.6-terra', 1e6, 1e6) returns 14.

Two live probes against the real CLI confirmed the managed home. With the operator's own config, stderr carried repeated rmcp::transport::worker failures. With the managed home, stderr was clean, no notify process launched, and the agent returned exactly {"ok":true}.

Live, against the operator's ChatGPT plan.

All six catalog ids answer and return valid JSON. Input-token overhead measured per call: sol 17322, terra 17487, luna 15920, 5.5 16527, 5.4 15172, 5.4-mini 14824.

A full completeJSONLogged call on codex/gpt-5.6-luna wrote this row, which is the single call site every stage routes through:

stage=generate  provider=codex  model=codex/gpt-5.6-luna
input_tokens=15955  output_tokens=17  cost_usd=0.0032114
response={"ok": true, "title": "Clemson vs LSU"}

The cost is exact against the catalog rate, 15955 at $0.2/1M plus 17 at $1.2/1M. No notify process launched.

The web-search path returned webSearchRequests: 1 with a real source URL, confirming web_search items are counted and webSearchMode reaches the CLI.

snowopsdev and others added 7 commits September 2, 2026 17:03
Provider routing assumed every provider authenticates through one environment
variable. Codex authenticates from a file-backed CLI login instead, so the
question "what does this model need before it can run?" becomes a tagged
ProviderRequirement union rather than another nullable string. Every caller
reads that union; there is no second copy of the check.

ensureManagedCodexHome exists because `-c 'mcp_servers={}'` does not work: -c
merges into the config table rather than replacing it, so an operator's MCP
servers still boot and spam transport errors on every call. A private
CODEX_HOME is the only lever that isolates config, and it symlinks the
operator's auth.json so the existing login is reused without this repo ever
reading the token.

Widening LlmProvider forced two exhaustive Records open. PROVIDER_LABEL gains
its codex entry and ModelReadiness widens envVar to string | null, both of
which the readiness phase needed anyway. Fingerprints for non-codex workspaces
are unchanged, so no existing verification run goes stale.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@openai/codex-sdk is pinned exact because it pins its own @openai/codex, and
the two must agree for the vendored binary to resolve. It is declared in both
workspaces, matching how the Anthropic and OpenAI SDKs are declared.

serverExternalPackages because the SDK resolves a native binary relative to its
own package directory, which bundling would move it away from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The six ids are the subset the ChatGPT plan actually serves through Codex, read
from ~/.codex/models_cache.json. It is narrower than the API catalog and moves
independently, so the list is named and tested rather than inferred.

Entries are generated from the OpenAI rows rather than hand-copied, so a Codex
mirror can never drift from the price it estimates. The existing "prices every
catalog entry" invariant could not have caught that drift: PRICES is built from
LLM_CATALOG, so it is self-consistent by construction. The new test compares a
mirror against the entry it mirrors, which is a claim the old one could not make.

The migration ships with the catalog because the dropdown values are Postgres
enum labels. Offering an option the database rejects on save is not a shippable
state. ADD VALUE IF NOT EXISTS keeps it idempotent against a dev-push race, and
down is a no-op because Postgres cannot drop an enum value.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One thread per call against the local Codex CLI, in a read-only sandbox with a
scratch working directory and approvals disabled. The SDK import is dynamic so
mock-mode processes and unit tests never load it.

Two measured behaviours drive the shape. An item of type error appears in items
even on fully successful turns, so only the outcome of run() decides success.
Supplying env to the Codex constructor replaces the child environment rather
than extending it, so the whole of process.env is spread in alongside CODEX_HOME.

Callers compose their own JSON instruction; the adapter adds none, because
llm.ts and brandVoiceExtract.ts each already carry one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The provider ternary existed twice, in completeJSON and again inside
createLlmClient. Adding a third provider to both copies would have invited
them to drift, so they collapse into one exhaustive switch.

parseJsonReply stays outside the Codex try/catch. The Anthropic and OpenAI
catches only re-map known SDK error classes, but the Codex one is a catch-all,
so wrapping the parse would stamp the stage into the message twice.

loadStageModels branches on the requirement kind rather than interpolating one
template, because "needs `codex login` set" is not a sentence. The env message
is unchanged to the byte. The login check is memoised as a single promise, so
six codex stages cost one process and mock mode costs none.

A Codex login deliberately does not flip mock mode to live. Mock is the local
default and a dev machine that happens to carry a login must not start making
paid calls because of it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
extractionMockMode asked whether the model had an API key, which made every
codex model look unconfigured and silently mock. It now asks the requirement
what it needs. It still never throws, so a keyless dev environment keeps working.

Readiness gains needsCodexLogin, kept out of `missing` because that list is
environment variable names and a login is not one. The config fingerprint now
hashes the login state; it previously read a nonexistent `codex-login` variable
and so never changed when the login did. A pinned hash proves the fingerprint
for a workspace with no codex stage is byte-identical, so no existing
verification run goes stale.

Admin renders use the file check rather than spawning the CLI per request. The
authoritative check runs once per pipeline run.

Also fixes a pre-existing rendering bug in the banner. missing.join('</code>,
<code>') inside JSX is escaped rather than parsed, so two missing variables
rendered as literal markup. This change makes the banner appear in more cases,
which is a poor moment to leave that in place.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every figure is measured rather than estimated. The per-call overhead of about
15k input tokens is stated plainly because it is the strongest reason to keep
API keys as the default and Codex as an opt-in.

The managed CODEX_HOME is documented with its reason. Without it an operator's
own Codex config applies to every call, which on the machine this was built on
meant 11 MCP servers booting and a notify hook launching a desktop app per call.

The trusted-host boundary is stated rather than implied. OpenAI has not said
whether embedding Codex with a ChatGPT login in a hosted product is permitted.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-02T21:42:43.301418Z 0790eab PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0790eabf9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cms/src/lib/brandVoiceExtract.ts
Comment thread cms/src/lib/workspaceReadiness.ts
snowopsdev and others added 2 commits September 2, 2026 19:05
extractionMockMode treated the presence of auth.json as activation, so with
MOCK_MODE unset a brand-guide upload made a live call and billed plan quota
while pipeline config and workspace readiness both still reported mock.

The rule this repo already states is that a Codex login is not consent to
spend. config.ts and modeFromEnv both honour it. This one did not, and the
disagreement was the bug. Extraction now needs MOCK_MODE=false explicitly,
like every other live path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
runtime.missing holds environment variable names, and a login is not one, so a
workspace blocked only by a missing Codex login produced "Configure the
required environment variables: ." in three actions.

runtime.blockers carries everything unmet in the words an operator acts on.
missing keeps its meaning for the callers that want variable names, including
the fingerprint and the run snapshot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@snowopsdev
snowopsdev merged commit 62090b8 into main Sep 2, 2026
6 of 8 checks passed
@snowopsdev
snowopsdev deleted the codex-provider branch September 2, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant