Route image and video renders per surface: a render-defaults registry plus per-record provider and model overrides

[atomantic]

Problem / Goal

PortOS can render images on four backends (local, codex, grok, agy) and video on two (local, grok), but only the Image Gen page can actually choose which one, and which model, for a given render. Every other creative surface — Universe Builder batch render, universe character sheets, Series / Creative Director first-pass, Sprites, Music Video, LoRA dataset generation, the creative agent tool — either silently inherits the single install-wide settings.imageGen.mode pin or exposes a bare backend picker with no model selection.

Two concrete consequences:

1. You cannot say "this Universe renders on Codex, that one renders on Agy." There is one global mode and a per-request body.mode, but no per-record persisted choice and no way to pin the cloud model.
2. There is no way to express per-surface defaults — e.g. "Universe Bible renders default to Codex; Music Video renders default to Agy" — so changing the Image Gen default mode changes it for everything at once.
3. Agy image renders have no cheap-tier default, so an unpinned render drives the session on whatever agy's own config selects — potentially claude-opus-4-6-thinking. Codex was given exactly this pin (CODEX_IMAGEGEN_DEFAULT_MODEL) for exactly this reason; agy never got the equivalent.

Goal: a render-target defaults registry (install-wide default provider + model per creative surface, set in Settings) plus per-record provider + model overrides on the records that own a render (Universe, Series, Sprite, Music Video project), threaded through the existing resolveCloudProviderConfig plumbing.

Context

The plumbing already exists — it's just only wired at one call site

server/services/imageGen/cloudProviderConfig.js is the shared cloud-provider resolver. CLOUD_PROVIDER_SPECS already carries per-provider supportsModelOverride (codex: true, agy: true, grok: false — grok's image_gen runs on a fixed xAI backend with no model knob) and resolveCloudProviderConfig(settings, mode, { model }) already applies an override with override → saved default → provider default precedence.

The client mirror is already in place too: client/src/lib/imageGenBackends.js exports MODEL_OVERRIDE_CAPABLE_MODES / supportsCloudModelOverride(mode), and client/src/components/imageGen/ImageGenControls.jsx:53-120 already renders a cloudModel picker with a "Settings default (…)" sentinel option.

But of the nine server call sites that resolve a cloud provider, only two pass a model override:

Call site	Surface	Passes { model }?
server/routes/imageGen.js:412	Image Gen page	✅
server/services/imageGen/index.js:120	dispatcher	✅
server/services/universeBuilderRender.js:76	Universe batch render	❌
server/services/universeCharacterSheet.js:445	Universe character sheet	❌
server/services/creativeDirector/firstPassGen.js:76	Series / CD first pass	❌
server/services/sprites/reference.js:543	Sprites reference	❌
server/services/pipeline/visualStageHelpers.js:254	Pipeline visual stages	❌
server/services/loraDatasetGenerate.js:348	LoRA dataset gen	❌
server/services/creative/tools/media.js:179	Creative agent tool	❌

How each surface picks its backend today

• Universe Builder — server/services/universeBuilderRender.js:62: const mode = body.mode || settings.imageGen?.mode || IMAGE_GEN_MODE.EXTERNAL. body.mode comes from transient renderOpts UI state in client/src/components/universeBuilder/RenderTab.jsx — nothing is persisted on the universe record, and no cloud model can be pinned.
• Series / Creative Director — server/services/creativeDirector/firstPassGen.js:75: settings.imageGen?.mode only. No per-request or per-series override at all.
• Sprites — has a page-owned backend picker (client/src/components/sprites/ReferenceWorkflow.jsx:200,441-442, ForkSpriteModal.jsx:129) so you can choose agy/codex/grok, but there is no model selection when agy or codex is chosen, and the choice isn't persisted on the sprite record.
• Video — server/services/videoGen/modes.js resolveVideoMode(requested, settings) defaults to LOCAL and its own comment flags the gap: "There is no settings.videoGen.mode install-wide pin today; when one lands it belongs as the second candidate here." Grok video also has no model knob (VIDEO_GEN_MODE only has LOCAL/GROK).

Prior art to follow

server/lib/creativeCommissionValidation.js:50,89-90,99,114-116 is the reference shape: a COMMISSION_RENDER_BACKEND_AUTO = 'auto' sentinel plus imageMode/imageModelId and videoMode/videoModelId fields on the record, with the client mirror at client/src/components/creative-commission/commissionForm.js:36-46,84-89. Pipeline issues do the same (server/routes/pipeline/issues.js:107, server/services/pipeline/issuesShared.js:364-378) — though note imageModelId there is deliberately cleared for non-local modes, which is exactly the limitation this issue removes.

The Agy model list is not wrong — it's a different axis

The report that Settings → Image Gen → Agy CLI should list imagen-3 / imagen-3-fast / gemini-2.0-flash / gemini-2.5-flash / gemini-3.5-flash instead of gemini-3.6-flash-high / claude-sonnet-4-6 / … rests on a misreading of that field, and the code already says so at client/src/components/settings/ImageGenTab.jsx:932-945.

Verified against the installed CLI, two ways:

1. The tool has no model parameter. agy's built-in generate_image takes exactly Prompt (req), ImageName (req), toolSummary (req), toolAction (req), AspectRatio (opt), ImagePaths (opt).

2. --model rejects image-model ids outright. Agy itself will claim you can pass an image model headlessly (agy --model imagen-3-fast --print "…"). You cannot — it fails before generating anything:

$ agy --dangerously-skip-permissions --model imagen-3-fast --print "A close-up photograph of a dew-covered spider web, macro lens…"
Error: invalid model selection (--model "imagen-3-fast" --effort ""): model imagen-3-fast is not recognized as a known model or custom model in settings
Available models:
  Gemini 3.6 Flash (High) … Gemini 3.5 Flash (Low) … Claude Opus 4.6 (Thinking) … GPT-OSS 120B (Medium)

No image was produced and no file was written. The "Available models" list agy prints is the same 11 agent models agy models returns — which is why client/src/hooks/useAgyModels.js drives the picker off a live probe (server/services/imageGen/agy.js:97 listModels) rather than a baked-in list. Grepping the agy binary finds none of the five imagen-* / gemini-N.N-flash ids.

One hook worth noting: the error says "not recognized as a known model or custom model in settings", so agy supports custom model registration in its own config. That is an Antigravity-side setup step outside PortOS's control — it is the only conceivable --model route, and PortOS must not assume it exists on a user's install.

3. The prompt-directive channel also fails — settled, no knob to build. Agy next suggests naming the image model in the prompt instead (agy --model gemini-3.5-flash-low --print "Use your generate_image tool to … using the imagen-3-fast model."). That was run end to end; it produced an image and then reported on itself:

Actual Image Model: The image was produced using the default backend model, Imagen 3 (specifically imagen-3.0-generate-002).
Honoring the imagen-3-fast Request: I was not able to honor the request for imagen-3-fast. The generate_image tool does not expose a parameter or option to select specific model variants (like imagen-3-fast vs. standard imagen-3). It automatically utilizes the default production Imagen 3 model configured on the server side.

So all three channels are closed (tool param, --model, prompt directive). Agy's image model is fixed server-side at imagen-3.0-generate-002 and is not selectable by PortOS through any means. Usefully, it is knowable — so the Settings row and the render sidecar can name the concrete model instead of an "unknown/vendor-chosen" placeholder.

Incidental finding, no action needed: with no active agy workspace, agy ignored the relative save path and wrote to ~/.gemini/antigravity-cli/scratch/web.png. PortOS is already immune — buildAgyPrompt directs an absolute stagingPath and generateImage verifies the bytes at that path. Worth knowing before anyone "simplifies" that path to a relative one.

So the underlying image model on the Agy path is chosen by Antigravity, not by PortOS. The remaining real defects are UI clarity — a text input labelled "Default agent model" with a model datalist under a provider called "Agy CLI Imagegen" reads as an image-model picker — and the missing cheap-tier session pin below. Both are Phase 1.

Agy has no cheap-tier session default (the Codex pin was never mirrored)

CLOUD_PROVIDER_SPECS[AGY].modelId (cloudProviderConfig.js) resolves to override || a.model || ANTIGRAVITY_CONFIGURED_DEFAULT, and resolveCliModel (server/lib/providerModels.js:52) maps that sentinel to null — so appendAntigravityModelAndEffort (server/lib/antigravity.js) appends no --model at all and agy runs the session on whatever its own config selects. agy models on this install offers claude-opus-4-6-thinking and gemini-3.1-pro-high alongside the flash tiers, so an unpinned PortOS image render can silently drive a reasoning-heavy tier just to relay one generate_image call.

Codex already solved this. server/services/imageGen/modes.js:54-64 pins CODEX_IMAGEGEN_DEFAULT_MODEL = 'gpt-5.6-luna' + CODEX_IMAGEGEN_DEFAULT_EFFORT = 'low' as a code-level default (not a settings migration) precisely so "every media-pipeline render pays the light path by default" on every install and federated peer, while an explicit imageGen.codex.model in Settings still wins. Agy needs the same treatment: the driving agent does no creative work on the image — it relays a prompt into one tool call and writes a file — so the cheapest/fastest model that reliably issues that call is the correct shipped default.

Proposed approach

Ship in phases, each its own PR referencing this epic (Refs #<this>, not Closes).

Phase 1 — Agy: pin a cheap session model, and name the fixed image model honestly

1. Add AGY_IMAGEGEN_DEFAULT_MODEL next to the Codex pins in server/services/imageGen/modes.js:54-64, and use it as the last fallback in CLOUD_PROVIDER_SPECS[AGY].modelId/.params (override || a.model || AGY_IMAGEGEN_DEFAULT_MODEL) so an unpinned agy render stops inheriting agy's own possibly-heavy default. Follow the Codex precedent exactly: a code-level default, no settings migration, so it reaches every install and federated peer; an explicit imageGen.agy.model in Settings still wins.
   • Shipped value: gemini-3.5-flash-low. Rationale: the driving agent only needs to relay one generate_image tool call and write a file, so the cheapest flash tier suffices; gemini-3.5-flash-low is directly verified to run and follow tool-oriented instructions on this install, whereas gemini-3.6-flash-low is newer and unverified for generate_image. Note agy bakes the reasoning ladder into the model id (-low/-medium/-high), so no separate --effort pin is needed for the gemini tiers.
   • Confirm reliability over a handful of renders. gemini-3.5-flash-low is verified to drive generate_image to a completed image once (see Context). If it proves flaky at issuing the tool call, escalate exactly one rung to gemini-3.5-flash-medium and record why in the constant's comment. Reliability beats cost — a tier that silently declines to generate produces the AGY_NO_IMAGE_HINT path (agy.js:133), which is worse than paying one rung up.
   • Mirror the constant into client/src/lib/imageGenBackends.js alongside CODEX_IMAGEGEN_DEFAULT_EFFORT so the Settings placeholder can read "Leave blank to use gemini-3.5-flash-low" instead of today's "Agy's own configured default" — which will no longer be true.
2. Relabel the Settings field to name its axis unambiguously (e.g. "Agent model (drives the session)") and render the image-model fact as its own explicit read-only row naming the concrete model — "Image model: Imagen 3 (imagen-3.0-generate-002) — fixed server-side by Antigravity, not selectable" — so the two axes can't be conflated. Keep the live agy models probe as the source for the agent list.
3. Do not build an image-model selector. All three channels are empirically closed (see Context): the tool exposes no parameter, --model imagen-* errors out, and the prompt directive is explicitly not honored. Ship no imageGen.agy.imageModel setting.
   • Record the three negative results as a comment next to the agy provider (and in docs/ alongside the other provider notes), including the fact that agy itself will claim both the --model and prompt-directive routes work. That false claim is the reason to write it down — it is cheap to re-litigate and costs a render each time.
   • Add imagen-3.0-generate-002 as a constant so the render sidecar can record the image model that actually ran, rather than the agent model, for honest provenance.

Phase 2 — renderDefaults: an install-wide per-surface defaults registry

Add a settings.renderDefaults map keyed by render target, each entry { imageMode, imageModel, videoMode, videoModel } with 'auto'/null meaning "fall through to the global settings.imageGen.mode". Define the target keys in one new leaf module (server/lib/renderTargets.js, barrelled + catalogued per the Module Organization rule) so the Settings UI, validation, and every resolver derive from one alphabet rather than hand-copying strings. Initial targets, drawn from the call-site table above:

universe-bible, universe-character-sheet, series-first-pass, sprite-reference, pipeline-visual, music-video, lora-dataset, creative-agent

Then a single resolver — resolveRenderTarget(target, settings, overrides) in server/services/imageGen/cloudProviderConfig.js's neighbourhood — that layers per-record override → settings.renderDefaults[target] → settings.imageGen.mode → provider default and hands back both the mode and the { model } bundle. Every one of the seven ❌ call sites above switches to it, so a new surface is one renderTargets entry plus one resolver call.

New Settings → Image Gen section ("Render defaults") lists each target with a backend picker + a model picker that only appears when supportsCloudModelOverride(mode) — reusing the existing ImageGenControls cloud-model control rather than a new one.

Phase 3 — Per-record overrides on Universe, Series, Sprite, Music Video

Persist imageMode / imageModelId (and videoMode / videoModelId where the record renders video) on each record, following the creativeCommissionValidation.js field shape and its 'auto' sentinel:

• Universe — on the universe record; surfaced in RenderTab.jsx as the default for this universe, with renderOpts still able to override for one batch.
• Series — on the series record; firstPassGen.js reads it (this surface has no override path today at all).
• Sprite — persist the existing page-owned picker onto the sprite record and add the model control next to it (this is the gap called out in the report: backend selectable, model not).
• Music Video project — on the project record, defaulting from renderDefaults['music-video'].

Each needs the usual PortOS plumbing for a new persisted field: schema parity per server/CLAUDE.md, a migration in scripts/migrations/ + data.reference/ seed if the on-disk shape changes, a server/lib/schemaVersions.js bump if the sync payload shape changes, and the client mirror.

Phase 4 — Video parity

Add the settings.videoGen.mode install-wide pin that server/services/videoGen/modes.js already anticipates, wire resolveVideoMode to consult renderDefaults[target].videoMode as its second candidate, and surface video backend + model on the same per-record controls. Grok video keeps supportsModelOverride: false until xAI exposes a model knob — the control must not be offered for it.

Acceptance criteria

• AGY_IMAGEGEN_DEFAULT_MODEL exists next to the Codex pins in server/services/imageGen/modes.js, is the last fallback in CLOUD_PROVIDER_SPECS[AGY], and an unpinned agy render appends an explicit --model <cheap tier> instead of no --model at all.
• An explicit imageGen.agy.model in Settings still overrides the new code-level default (Codex-precedent behaviour), and a per-render cloudModel still overrides both.
• The Settings placeholder names the actual shipped default rather than "Agy's own configured default", which stops being true.
• The chosen default tier is confirmed to reliably issue the generate_image call over a handful of renders — none landing on the AGY_NO_IMAGE_HINT path — or the constant is escalated one rung with the reason recorded in its comment. (gemini-3.5-flash-low is already verified to generate successfully once.)
• Settings → Image Gen → Agy CLI makes the agent-model vs image-model distinction unambiguous, and names the fixed image model (imagen-3.0-generate-002) as a read-only fact. No field implies PortOS can pick imagen-*.
• No imageGen.agy.imageModel setting is added — all three selection channels are already proven closed.
• The three negative results are written down next to the provider, explicitly noting that agy misreports its own capability here.
• Agy render sidecars record imagen-3.0-generate-002 as the image model (distinct from the agent model), so provenance doesn't conflate the two.
• server/lib/renderTargets.js enumerates every render target; a test fails if a resolveCloudProviderConfig call site resolves a mode without going through the shared render-target resolver.
• settings.renderDefaults is settable per target in Settings, and picking e.g. universe-bible → codex / music-video → agy changes those surfaces' defaults without changing settings.imageGen.mode.
• All seven ❌ call sites in the table pass a model override through resolveCloudProviderConfig.
• Universe, Series, Sprite, and Music Video records each persist a backend + model choice that overrides the install default, visible and editable on that record's page.
• A model control is never shown for a backend whose spec says supportsModelOverride: false (grok image, grok video).
• The effective mode + model that actually rendered is recorded in the render sidecar / job params for every surface, so provenance doesn't lie when a default is in play.
• settings.videoGen.mode exists and resolveVideoMode consults it; grok video still renders only at 6s/10s with no model knob.
• Migration in scripts/migrations/ + data.reference/ seed for any new persisted field; schemaVersions.js bump if a sync payload shape changes; server + client tests for the resolver's precedence ladder (override → target default → global → provider default).

Out of scope

• Adding new image or video providers. This is about routing to the four image / two video backends that already exist.
• Changing how any provider renders (prompt construction, cleaners, aspect handling, C2PA/denoise).
• Per-render-step granularity below the record level (e.g. a different backend per individual canon entry within one universe).
• Making grok accept an image or video model — it has no such knob.
• Selecting Antigravity's image model by any means. All three channels are proven closed; it is fixed server-side at imagen-3.0-generate-002.
• Registering custom models in Antigravity's own settings to make --model imagen-3-fast resolve. That is a user-side Antigravity setup step, not something PortOS can provision or depend on.

[atomantic]

Phase 1 shipped in PR #3232 (merged): AGY_IMAGEGEN_DEFAULT_MODEL cheap-tier pin, AGY_IMAGEGEN_IMAGE_MODEL sidecar provenance, and the Settings agent-model/image-model split. Phases 2–4 (renderDefaults registry, per-record overrides, video parity) remain.