feat(images): add Grok image bridge for non-OpenAI models#424
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an Image Bridge for routed image-generation requests. It detects hosted image tools, plans xAI execution, stores generated images as local artifacts, fulfills intercepted calls, continues the model conversation through bounded SSE iterations, and documents configuration and limitations. ChangesImage Bridge
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant ResponsesCore
participant runWithImageBridge
participant xAI
participant Artifacts
Client->>ResponsesCore: image_generation request
ResponsesCore->>runWithImageBridge: execute routed image turn
runWithImageBridge->>xAI: generate or edit image
xAI-->>runWithImageBridge: image data or URL
runWithImageBridge->>Artifacts: save image artifact
Artifacts-->>runWithImageBridge: local path
runWithImageBridge-->>Client: continued SSE response
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fb94938 to
976da38
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (3)
docs-site/src/content/docs/guides/image-bridge.md (1)
15-18: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winCorrect the xAI authentication guidance.
Document the
xaiprovider preset withhttps://api.x.ai/v1and theopenai-chatadapter. Explain thatauthMode: "oauth"uses the stored/auto-refreshed bearer token, whileauthMode: "key"uses the configured API key. Do not claim OAuth is less reliable without evidence;forwardonly relays the curated ChatGPT header allowlist and is not a substitute for xAI bridge credentials.Also applies to: 60-63
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/guides/image-bridge.md` around lines 15 - 18, Update the xAI authentication guidance in the image-bridge guide to document the xAI preset using https://api.x.ai/v1 with the openai-chat adapter, and distinguish authMode: "oauth" (stored/auto-refreshed bearer token) from authMode: "key" (configured API key). Remove unsupported reliability claims and clarify that forward only relays the curated ChatGPT header allowlist, not xAI bridge credentials.Source: Path instructions
src/server/responses/core.ts (2)
1452-1473: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winAdd focused handler-level regression coverage.
Cover a non-OpenAI routed request that injects
image_genand calls the bridge, OpenAI passthrough bypass, web-search precedence, and the non-streaming behavior above.As per path instructions, “A behavior change in src/ should come with a focused regression test near the existing tests for that subsystem.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 1452 - 1473, Add focused regression tests near the existing response-handler tests covering a non-OpenAI routed request that injects image_gen and invokes runWithImageBridge, OpenAI passthrough bypass, web-search taking precedence, and non-streaming responses returning the bridged body with preserved status and headers. Use the handler symbols planImageBridge, runWithImageBridge, and the surrounding response flow to verify each behavior without broad refactoring.Source: Path instructions
1457-1471: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winDo not return SSE for non-streaming image-bridge requests.
This branch ignores
parsed.stream, whilerunWithImageBridgeforces every provider iteration tostream: trueand returns the bridged stream. A request with"stream": falsecan therefore receivetext/event-streaminstead of a normal Responses JSON object.Collect/serialize a non-streaming result, or reject it explicitly before invoking the bridge.
Proposed guard
if (imgPlan && !wsPlan) { + if (!parsed.stream) { + return formatErrorResponse( + 400, + "invalid_request_error", + "Image bridge requires stream=true", + ); + } parsed.context.tools = [...(parsed.context.tools ?? []), buildImageTool()];🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/responses/core.ts` around lines 1457 - 1471, Update the image-bridge branch around runWithImageBridge to honor parsed.stream: reject non-streaming requests before invoking the bridge, or collect and serialize the bridged result as a normal Responses JSON object instead of returning its SSE body. Preserve the existing trackStreamLifetime Response path only for streaming requests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/images/loop.ts`:
- Around line 147-150: Update the image bridge dependency construction in the
responses core flow to forward config.images.maxRounds whenever it is defined,
including 0, instead of using a truthy check. Preserve omission when the value
is undefined so runWithImageBridge can apply its default only when no
configuration was provided.
In `@src/images/synthetic-tool.ts`:
- Around line 21-44: Update the function-tool branch in
extractHostedImageGeneration to read the tool name from the flat obj.name field
instead of nested obj.function.name, while retaining the isImageGenName check
and existing tool tracking. Update the corresponding synthetic-tool test fixture
to use the flat Responses shape with type "function" and name.
---
Duplicate comments:
In `@docs-site/src/content/docs/guides/image-bridge.md`:
- Around line 15-18: Update the xAI authentication guidance in the image-bridge
guide to document the xAI preset using https://api.x.ai/v1 with the openai-chat
adapter, and distinguish authMode: "oauth" (stored/auto-refreshed bearer token)
from authMode: "key" (configured API key). Remove unsupported reliability claims
and clarify that forward only relays the curated ChatGPT header allowlist, not
xAI bridge credentials.
In `@src/server/responses/core.ts`:
- Around line 1452-1473: Add focused regression tests near the existing
response-handler tests covering a non-OpenAI routed request that injects
image_gen and invokes runWithImageBridge, OpenAI passthrough bypass, web-search
taking precedence, and non-streaming responses returning the bridged body with
preserved status and headers. Use the handler symbols planImageBridge,
runWithImageBridge, and the surrounding response flow to verify each behavior
without broad refactoring.
- Around line 1457-1471: Update the image-bridge branch around
runWithImageBridge to honor parsed.stream: reject non-streaming requests before
invoking the bridge, or collect and serialize the bridged result as a normal
Responses JSON object instead of returning its SSE body. Preserve the existing
trackStreamLifetime Response path only for streaming requests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1c432e5a-32fd-4e62-aa6d-2b080fad526f
📒 Files selected for processing (17)
docs-site/src/content/docs/guides/image-bridge.mdsrc/images/artifacts.tssrc/images/fulfill.tssrc/images/index.tssrc/images/loop.tssrc/images/plan.tssrc/images/synthetic-tool.tssrc/images/types.tssrc/images/xai-client.tssrc/responses/parser.tssrc/server/responses/core.tssrc/types.tstests/images/fulfill.test.tstests/images/loop.test.tstests/images/plan.test.tstests/images/synthetic-tool.test.tstests/images/xai-client.test.ts
976da38 to
38d4112
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
38d4112 to
d4b9ac5
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
d4b9ac5 to
79da2f9
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
79da2f9 to
db48e7c
Compare
🔒 Under maintainer review — detailed feedback incoming@lidge-jun (maintainer) has this PR in an active review pass. Please do not merge or close This PR is not part of the current integration batch. It is held back deliberately — Heads-up on rebasing: No action needed from you until then. Thanks for the contribution and for your patience. Review tracker: |
|
Thank you for building out the Grok image bridge. There are two merge-blocking issues in the current implementation. First, Second, The outbound fetch is a network security boundary, so this PR also needs explicit maintainer security review after the fixes. I appreciate the substantial work here; these two paths need to be closed before we can safely expose it. Please also rebase onto current Reviewed as part of a maintainer integration pass. |
db48e7c to
99fa843
Compare
|
Follow-up review at head The paid bridge defaults to on. if (config.images?.bridgeEnabled === false) return undefined;so merely having xAI configured is enough to route prompts into a paid generation path. A cross-provider feature that spends money on the operator's account should require them to say yes: if (config.images?.bridgeEnabled !== true) return undefined;Update The credential destination is not pinned. Resolve the provider through Status: held for explicit security review under One coordination note: #355 also adds Reviewed as part of a maintainer PABCD pass. Adds findings not present in the earlier pass. |
통합(dev push 완료): #437 803807a, #460 74ddd96, #431 82a47db, #405 be16c1d, #468 후속 테스트 bcaf029. close: 통합 4건 + 통합 없이 1건(#459, 설계 충돌). #457은 OPEN 유지. 이슈 close 0건 — dev에서 고쳐졌는데 안 닫힌 이슈가 없었다. 무효화: WP4(#466)와 WP6(#467)은 동료가 직접 머지했고, 우리 분석이 지적한 결함을 동료 커밋이 같은 방향으로 해소했다. 보안 보류 5건에 신규 발견 게시: #429 재분류, #355 OAuth 토큰 목적지, #424 유료 브리지 기본 ON, #408 설치 락 해제, #403 4분할 요청. A-gate는 사이클마다 3~5라운드를 돌았다. 반복된 원인은 계획서에 테스트나 코드를 적으면서 실물 소스를 확인하지 않은 것이었다.
When a non-OpenAI model receives an image_generation hosted tool from Codex,
OpenAI's server-side execution is unavailable. This intercepts the tool call,
routes it to xAI Grok Imagine, materializes the image to ~/.opencodex/artifacts/,
and feeds the result back to the model — all inside a streaming SSE loop.
Architecture mirrors the proven web-search sidecar pattern:
- parser.ts: extract hosted image_generation tool, stash into parsed._imageGeneration
- plan.ts: decide whether to activate (xAI provider + token available, non-OpenAI route)
- loop.ts: agentic loop (max 3 rounds) — intercept image tool calls, fulfill via xAI,
inject results, re-call model, bridge to SSE
- synthetic-tool.ts: buildImageTool() injection + isImageGenName() detection
- xai-client.ts: xAI /images/generations and /images/edits API client
- fulfill.ts: execute single image call, materialize to disk (never throws)
- artifacts.ts: extended with downloadImageToArtifact() + magic byte format detection
New files:
src/images/{types,xai-client,synthetic-tool,fulfill,plan,loop,index}.ts
tests/images/{xai-client,plan,fulfill,synthetic-tool,loop}.test.ts
docs-site/src/content/docs/guides/image-bridge.md
Modified files:
src/types.ts: +_imageGeneration stash, +OcxTool.imageGeneration flag, +OcxImagesConfig fields
src/responses/parser.ts: extractHostedImageGeneration extraction
src/server/responses/core.ts: image bridge trigger point
src/images/artifacts.ts: +downloadImageToArtifact, +guessExtFromMagic, shared helpers
99fa843 to
b3f2b49
Compare
2 participants
Summary
When Codex routes to a non-OpenAI model (Claude, Gemini, Grok, etc.), the
image_generationhosted tool stops working because it requires OpenAI's server-side execution. This PR adds an Image Bridge that:image_generationin the request's tools arrayimage_gen) the routed model can call/images/generations)~/.opencodex/artifacts/with magic-byte format detectionThe architecture mirrors the proven
src/web-search/sidecar pattern — same async-generator loop,bridgeToResponsesSSE, heartbeat, and abort-signal linkage.New Files
src/images/types.tsImageBridgePlan,ImageCallResult)src/images/synthetic-tool.tssrc/images/xai-client.ts/images/generationsand/images/editsclientsrc/images/fulfill.tssrc/images/plan.tssrc/images/loop.tssrc/images/index.tstests/images/*.test.tsdocs-site/.../image-bridge.mdModified Files
src/types.ts—_imageGenerationstash field,OcxTool.imageGenerationflag,OcxImagesConfigbridge fieldssrc/responses/parser.ts—extractHostedImageGeneration()+ stash into parsed resultsrc/server/responses/core.ts— image bridge trigger (after web-search block, before standard path)src/images/artifacts.ts— extended withdownloadImageToArtifact()+guessExtFromMagic()(shared helpers extracted)Verification
bun x tsc --noEmitpasses (0 errors)bun test tests/images/passes (53 tests, 0 failures)bun run privacy:scanpassesChecklist
Summary by CodeRabbit
image_generationon non-OpenAI providers using an interception/synthetic tool and bounded multi-round tool-call looping.