feat: Headless platform-destination sync via cqpd_ token (no cloudquery login)#22992
Merged
Conversation
…ownload When CQ_PLATFORM_TOKEN is set, sync/migrate use the pre-minted cqpd_ token as the plugin-download credential and read the owning team from its `tm` claim — no `cloudquery login`. managedplugin then hits the team-scoped DownloadPluginAssetByTeam endpoint, so downloads are authorized + recorded against the tenant's cloud team. Adds platform.TeamFromToken (mirrors apiURLFromToken) and exports EnvPlatformToken. Requires the token to carry tm (platform#3082, cloud#8642).
disq
changed the title
tm claim)tm claim)
The headless-cqpd_-vs-login download-credential block was verbatim in sync and migrate. Move it to platform.DownloadAuth(ctx, logger, sources, destinations, transformers) (token, team, err) — one place reads CQ_PLATFORM_TOKEN and the tm claim, killing the drift risk. sync/migrate now call it; both drop their now -unused auth/os imports.
stoovon
approved these changes
Jun 25, 2026
DownloadAuth now detects a cqpd_ token on the resolved credential (so it works via the standard CLOUDQUERY_API_KEY env, not just CQ_PLATFORM_TOKEN) and resolves the team from its tm claim locally instead of calling GetTeamForToken, which would hit cloud and fail for a syncs-scoped key.
One platformToken() (CQ_PLATFORM_TOKEN, else a cqpd_ in CLOUDQUERY_API_KEY) used by DownloadAuth, MaybeInjectDestination and DetectTenant — so a cqpd_ in either env behaves identically for download, injection and tenant detection. Removes the half-wired post-resolution prefix check; CLOUDQUERY_API_KEY=cqpd_ now also triggers injection/detection, not just download.
When a cqpd_ token is used (no session mint) and CQ_PLATFORM_PLUGIN_VERSION isn't set, fetch the recommended destination plugin version from the tenant's platform whoami (reached via the token's api_url, authed with the token) and pin the injected plugin to it. Env override still wins; best-effort — any whoami failure falls back to the CLI default. Needs platform#3162 (whoami returns plugin_version).
Match the platform default floor — v1.0.1 derives api_url from the token, so the CLI's no-injected-api_url behavior works when neither the env override nor whoami supplies a version.
Log each branch of recommendedVersionFromWhoami at debug (no api_url, request build, transport error, non-200, decode/nil, and the pinned version) so a headless run that falls back to the default is diagnosable.
Source/destination plugins validate premium tables + report usage against cloud using CLOUDQUERY_API_KEY from their own env (inherited from the CLI in local runs). A cqpd_ supplied only via CQ_PLATFORM_TOKEN was invisible to them, so premium-table syncs failed with 'run cloudquery login'. PropagatePluginCredential exports the headless cqpd_ as CLOUDQUERY_API_KEY (only when unset, so a user's own key wins); sync and migrate call it after DownloadAuth.
disq
changed the title
tm claim)
disq
commented
Jun 29, 2026
| // user who set their own (e.g. a team key) keeps it for plugin auth. | ||
| func PropagatePluginCredential(token string) { | ||
| if strings.HasPrefix(token, cqpdPrefix) && os.Getenv("CLOUDQUERY_API_KEY") == "" { | ||
| _ = os.Setenv("CLOUDQUERY_API_KEY", token) |
Member
Author
There was a problem hiding this comment.
This looks scary at first glance but it's only done for cqpd_ tokens, where the user has manually created and copy-pasted this into their terminal/env.
erezrokah
approved these changes
Jun 29, 2026
kodiakhq Bot
pushed a commit
that referenced
this pull request
Jun 30, 2026
🤖 I have created a release *beep* *boop* --- ## [6.38.0](cli-v6.37.0...cli-v6.38.0) (2026-06-30) ### Features * Headless platform-destination sync via cqpd_ token (no cloudquery login) ([#22992](#22992)) ([4f55c23](4f55c23)) ### Bug Fixes * **deps:** Update go module directive to v1.26.4 ([#22996](#22996)) ([4c6a6b1](4c6a6b1)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq Bot
pushed a commit
that referenced
this pull request
Jul 6, 2026
When authenticated against a CloudQuery Platform tenant, `init` and `validate-config` now use the platform's **pinned** plugin versions and **recommended tables**, and handle source-only platform specs — instead of treating everything as a plain cloud-hub sync. Reuses existing `/external-syncs/*` endpoints (the same data that powers the sync-time gate); all changes here are CLI-side. Follows the platform-destination line (#22945, #22991, #22992). **Depends on cloudquery/platform#3342** (the `/external-syncs/recommended-tables` endpoint) for the recommended-tables feature — best-effort, so the CLI degrades to `tables: ['*']` until that ships. > **Also touches `sync`** (despite the title): `MaybeInjectDestination`'s mint-failure handling changes from fail-open to hard-error — see the `sync` section. ## `init` - Resolves the tenant **once** (one `activeTenants` lookup + one mint) via `DetectTenantForInit`, returning a `TenantInit` that carries the tenant URL, the pinned source versions, and a session reused for the recommended-tables lookup — no extra mint. - Scaffolds the platform-**pinned** source version instead of cloud-latest. - Scaffolds the platform's **recommended tables** for the source (`GET /external-syncs/recommended-tables`) in place of `tables: ['*']`, so a sync populates the tables the platform ingests. Only the wildcard is replaced — a curated example config is left alone. Best-effort → falls back to `['*']`. - If a tenant is detected but its session can't be minted, `init` **fails** (pointing at `--disable-platform`) rather than scaffolding a spec that would break at sync time. ## `validate-config` - **Source-version gate**: for sources opting into `platform`, hard-fails on a version outside the platform's supported window — the same rejection `sync` hits at `CreateExternalSync`. - **Source-only specs**: mirrors `sync` — reads without requiring a destination, injects the `platform` destination, then validates. No longer fails with `expecting at least one destination`. - **Plugin-spawn path**: resolves the download credential + team via `DownloadAuth` and runs `PropagatePluginCredential`, so a headless `cqpd_` token works and spawned plugins authenticate premium-table validation as under `sync`. ## `sync` - `MaybeInjectDestination` returns the mint error instead of silently skipping injection — an explicit `platform` opt-in shouldn't be dropped into a sync whose source targets a destination that was never injected (same stance as the ambiguous-tenant case, #22991). ## Safety Every platform code path is gated on a source opting into `platform` (`destinations: [platform]`). A non-platform sync/validate returns before any tenant/mint/gate/tables call — structurally impossible to affect. Covered by `TestInject_NonPlatformSync_MintErrorIrrelevant`. ## Auth flows Headless (`CQ_PLATFORM_TOKEN` / `cqpd_` `CLOUDQUERY_API_KEY`) reads the token's `u` claim; logged-in mints a session for the resolved tenant. ## Known / accepted - `validate-config` mints twice for a platform spec (version-gate lookup + injection) where `sync` mints once. Low priority — `validate-config` isn't hot. ## Tests `internal/platform`: one-pass tenant detection (`DetectTenantForInit` — direct-token, login-mint, mint-fail→error, pins-unavailable→best-effort, multi-tenant), `RecommendedTables` (session reuse, non-200→nil), support-window table, `GateSources`, `PropagatePluginCredential`, non-platform safety. `cmd`: `TestValidateConfig_PlatformSourceOnly` (hermetic), `Test_withRecommendedTables` (wildcard variants, curated-untouched, parseable output). `go build` / `go vet` / gofmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets
cloudquery sync/migraterun against a CloudQuery Platform tenant using only a pre-mintedcqpd_token — nocloudquery login— for the full flow: plugin download, destination auto-injection, premium-table auth, and version pinning. Everything the CLI needs (tenant, API URL, owning team) rides in the token's claims (t/u/tm).How a cqpd_ token is supplied
Either env works, and they behave identically (one
platformToken()resolver):CQ_PLATFORM_TOKEN=cqpd_…— explicit, separable platform-destination credential.CLOUDQUERY_API_KEY=cqpd_…— the standard credential env, when it holds a cqpd_.What the CLI does with it
DownloadAuth): uses the cqpd_ as the plugin-download credential and the team from thetmclaim →managedpluginhits the team-scopedDownloadPluginAssetByTeam, so downloads are authorized + recorded against the tenant's cloud team. The cqpd_ is syncs-scoped (can't enumerate teams), so the team comes from the claim, not a cloud call.MaybeInjectDestination): auto-injects theplatformdestination from the token alone — no cloud login / tenant discovery / session mint.api_urlcomes from theuclaim (not injected into the spec).DetectTenant):initreports the target tenant from theuclaim.PropagatePluginCredential): exports the cqpd_ asCLOUDQUERY_API_KEYfor spawned plugins (which validate premium tables / report usage against cloud from their inherited env) — only when unset, so a user's own key wins.…/external-syncs/whoami(viau, authed with the token) and pins the injected plugin. Precedence:CQ_PLATFORM_PLUGIN_VERSIONenv > whoami-recommended > CLI default (v1.0.1). Best-effort with debug logging on every branch.Non-headless (logged-in) flow is unchanged — it still resolves the token/team via
cloudquery loginand gets the version from the session mint.Cross-repo dependencies
tmclaim + admin mint (3082, merged), platform-direct minttm(3117), whoamiplugin_version+ defaultv1.0.1(3162)tminto the mint (8642)api_urlfrom the token (cloudquery-private#12858, merged) — needs av1.0.1release for the default download to resolve.Tests
DownloadAuth(CQ_PLATFORM_TOKEN + CLOUDQUERY_API_KEY paths),TeamFromToken, direct-token inject (both envs), whoami version-pin (httptest),PropagatePluginCredential(set-when-unset / never-clobber / non-cqpd no-op),DetectTenantmulti-tenant. Build/vet/go test ./internal/platform/green.