Desktop: assisted-first setup for ChatGPT/Claude cloud connectors (field-copy overlay card)#8826
Conversation
… connectors ChatGPT and Claude cloud connector setup no longer attempts autonomous browser automation (AX/OCR of other apps' web UIs proved too brittle across browsers and machines). "Do it for me" now deterministically opens the provider's connector page and shows a movable on-screen card with one copy button per field (secrets masked on screen, real value copied), and the sheet auto-expands the step-by-step instructions. No Accessibility or Screen Recording permission needed. The browserAutonomous routing is kept unmapped as the slot for a future DOM-perception rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/cloud-connectors-roadmap.md: why autonomous setup is parked, the assisted-first Phase 1 shipped here, Phase 2 (DOM-fidelity automation via embedded WKWebView), and Phase 3 (Anthropic connector directory + OpenAI Apps SDK listings). Linked from integrations-philosophy.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…osophy doc The ChatGPT setup guidance pointed at OAuth client "omi" with the user's MCP key as client secret — prod registers `omi-chatgpt-prod` as a PUBLIC PKCE client whose token endpoint rejects any secret, so sign-in failed with "Unknown OAuth client". The card and manual steps now use the registered client id per backend (prod/dev) and tell the user to leave the secret blank (rendered as a non-copyable "leave blank" row). Remaining backend gaps (per-connector redirect allowlist, missing Claude client) are tracked in issue #8824. Also: field-copy card is draggable (isMovableByWindowBackground), automation state reports the live panel frame, and the philosophy doc gains §2b (assisted-first), probe-the-whole-chain, and backend-config-is-provider- knowledge prescriptions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8714fd3fa4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// endpoint rejects any client secret for it, so setup must leave the | ||
| /// secret blank. Dev registers `omi-chatgpt-dev`. | ||
| static var chatgptOAuthClientID: String { | ||
| mcpBaseURL.contains("api.omi.me") ? "omi-chatgpt-prod" : "omi-chatgpt-dev" |
There was a problem hiding this comment.
For beta/dev builds DesktopBackendEnvironment.pythonBaseURL() points at api.omiapi.com, so this hardcoded branch tells users to enter omi-chatgpt-dev with a blank secret. I checked the dev backend wiring in backend/charts/backend-listen/dev_omi_backend_listen_values.yaml and backend/charts/backend-secrets/dev_omi_backend_secrets_values.yaml: it only supplies the legacy MCP_OAUTH_CHATGPT_CLIENT_ID/SECRET/REDIRECT_URIS, not MCP_OAUTH_PUBLIC_CLIENT_ID or a public-client config, and backend/database/mcp_oauth.py only treats omi-chatgpt-prod as a built-in public ChatGPT client. In that environment the OAuth authorize/token flow will reject the displayed client as unknown or confidential, so beta/dev users following the new assisted card cannot complete ChatGPT setup.
Useful? React with 👍 / 👎.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
6 issues found across 9 files
Confidence score: 3/5
- In
desktop/macos/Desktop/Sources/CloudConnectorGuidanceOverlay.swift,ForEachkeys fields bylabel, so duplicate labels can trigger a runtime crash when rendering the guidance form; switch to a guaranteed-uniqueid(or index-based key) before merging. - In
desktop/macos/Desktop/Sources/CloudConnectorGuidanceOverlay.swift,isSecretcurrently masks only labels containing "secret", which can expose values like API keys/tokens/passwords in the UI and recordings; broaden masking rules or make sensitivity explicit per field before merge. - In
desktop/macos/Desktop/Sources/MemoryExportService.swift, returningomi-chatgpt-devfor non-prod appears misaligned with backend OAuth client registration, so dev/staging auth flows may fail as-is; align the client ID with configured backend clients (or register it) before merging. - In
desktop/macos/Desktop/Sources/MemoryExportService.swift, the guided setup tool JSON omits a schema-required argument, so the firstfill_cloud_connector_formcall can fail validation and block setup; include the required argument (empty for PKCE if needed) and add a regression check before merging.
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@cursor address review comments if valid, and harden logic so that we can't accidentally run into these types of footguns again |
|
All six review items were valid. I addressed them in commit Overlay hardening (
|
- Use explicit CloudConnectorCopyField ids (never label-derived) with debug uniqueness check; broaden secret masking to key/token/password labels - Extract shared card header; size field-copy card from subtitle length - Include empty oauth_client_secret in ChatGPT guided-setup tool JSON - Register omi-chatgpt-dev as built-in public PKCE client on backend and wire dev helm public OAuth env vars; add regression tests Co-authored-by: David Zhang <Git-on-my-level@users.noreply.github.com>
2 participants
What
Makes the assisted flow the primary "Do it for me" path for the two connectors where autonomous browser automation has been unreliable (ChatGPT and Claude cloud custom connectors):
chatgpt.com/#settings/Connectors) and shows a movable on-screen card with one Copy button per field — secrets masked on screen, real value copied. The connector sheet auto-expands the step-by-step instructions.MCPExecuteKind.browserAutonomousstays in the code but is unmapped — parked as the routing slot for a future DOM-perception rebuild.omi-chatgpt-prodon prod,omi-chatgpt-develsewhere) with the Client Secret left blank — the previous guidance ("omi" + MCP key as secret + client_secret_post) failed with "Unknown OAuth client" and would have failed token exchange. Remaining backend gaps (per-connector redirect allowlist, missing Claude client registration) are tracked in MCP OAuth: ChatGPT/Claude cloud connector sign-in fails against prod (client registry + redirect allowlist gaps) #8824.desktop/macos/docs/cloud-connectors-roadmap.md(Phase 1 assisted-first → Phase 2 DOM-fidelity automation via WKWebView → Phase 3 provider directory listings), andintegrations-philosophy.mdgains §2b (assisted-first), probe-the-whole-chain, and backend-config-is-provider-knowledge prescriptions.Testing
swift test --filter BrowserAutomationTargetTests: 28/28 pass.omi-omi-assisted-overlay) and the automation bridge:POST /execute-exportfor both destinations opens the browser page and presents the card (Claude 4 rows, ChatGPT 7 rows); overlay state and window captures verified. David manually verified the Claude card end-to-end (copy buttons, dragging) and walked the ChatGPT connector flow live — which is what surfaced the OAuth wiring bug fixed here.rm -rf .build && xcrun swift build -c release --triple arm64-apple-macosxcurrently fails on a pre-existing manifest conflict (omi-lib min macOS 10.13 vs AudioKit 11.0), unrelated to this diff; debug build and tests are green.🤖 Generated with Claude Code