feat: competitor intelligence ingest + seedance2 fixes#1
Merged
Conversation
Adds the competitor-intel data model per docs/growth/07 §4 (+ 09 §3's segmentPlan addition): CompetitorCreative caches AppGrowing ad metadata and AI analysis, idempotent on (orgId, source, externalId); RemixJob is a queryable ledger for turning competitor creatives into our-own-IP assets (not wired to an engine yet — Phase 2). Purely additive, both relate to Organization like the rest of the schema. Migration generated via `prisma migrate diff --from-schema <base> --to-schema prisma/schema.prisma --script` (no local Postgres available in this worktree — see .claude/rules/schema.md).
POST /api/ingest/competitor?org=<orgId> — clones the ingest/scenes HMAC + idempotent-upsert pattern for competitor ad creatives pushed by the local creative-pipeline/AppGrowing bridge. Accepts a single item or an array; mediaUrl already on our GCS bucket is linked directly, an external mediaUrl is fetched once and uploaded to GCS. Per-item created/updated/failed result. Connector parsing/mapping logic lives in src/lib/platforms/appgrowing.ts so the ingest source is isolated from the route, per docs/growth/07 §5. GET /api/competitors — session-scoped listing (requireAuthWithOrg, never trusts a client orgId), filterable by appName/level, orderable by adDays/impressions. Ref: docs/growth/09-pipeline-adex-integration.md §3
Real-world doubao-seedance-2-0-260128 task responses return the generated video URL at content[].video_url.url, not output.video_url — add resolveVideoUrl() that prefers content, falls back to output for older/other model shapes. Also round the requested duration to whole seconds before submitting to Ark; a fractional duration gets a 400 InvalidParameter. Found via first real Seedance2 calls in the local creative-pipeline tooling (commit e06e215/1224f1c). Ref: docs/growth/09-pipeline-adex-integration.md §3
Register→login→GET /api/orgs→POST /api/ingest/competitor round trip: HMAC-signed ingest creates then idempotently updates a CompetitorCreative row, a bad signature 401s, GET /api/competitors requires a session and reflects the upsert. playwright.config.ts's webServer.env gains INVITE_CODES_DISABLED + INGEST_WEBHOOK_SECRET so the spec can self-provision a user/org and sign requests without a PlatformAuth row. Verified the code path against a build with no DATABASE_URL (fails at the expected `DATABASE_URL is not set` boundary, confirming registration/ingest wiring is correct) — no live Postgres was available in this worktree to run the full assertions end-to-end.
Adds the design docs this PR implements: 07 (competitor-intel + remix proposal, incl. the CompetitorCreative/RemixJob model proposal), 08 (creative pipeline stages), 09 (this PR's scope — pipeline↔adex integration, PR #1 boundaries).
WHY: product decision — the whole pipeline lands on adex, not local tooling. Control plane stays in the Next.js service; minute-scale audio/video batch work moves to a Cloud Run Jobs worker whose image packages the already-verified creative-pipeline scripts. Adds per-stage design table, PR roadmap (#1-oratis#4), cost + model-tier policy.
WHY: verified against live task cgt-20260710153030-s5t5x — real
doubao-seedance-2-0-260128 returns content as an OBJECT {video_url:string}
(not an array) and duration at TOP LEVEL. Prior resolveVideoUrl threw
TypeError (content.find is not a function), leaving Assets stuck
'generating' — the PR's core fix did not work against the target model.
WHAT: resolveVideoUrl handles object/array/legacy shapes (array form skips
echoed reference_video inputs); resolveDuration reads top-level→output→
usage; generate route rounds duration before the Int? persist (was a 500
after a paid Ark job started).
WHY: code review — the competitors list filtered level via a rawMeta JSON path that nothing writes (dead filter), while its sibling segmentPlan got a real column. level is the primary routing axis (09 §3), so it warrants a first-class indexed column, not an untyped blob key.
WHY/WHAT (xhigh review findings): - SSRF: ingest fetched caller-supplied mediaUrl with no validation → new storage.uploadFromUrl blocks non-http(s) + private/link-local hosts. - OOM/timeout: whole video buffered, unbounded batch → 100MB cap + 50-item batch cap (413). - competitors list masked all errors as empty 200 → returns 500 on failure; level filter now hits the real column. - parseHumanNumber fixes impressions/adDays silently dropping '710.4K' / '1,942' (the ranking signals); mergeRawMeta no longer corrupts array payloads; asset type prefers content-type over extension; isOwnGcsUrl + gcsPublicPrefix consolidated into storage.ts (was duplicated).
WHY: register is rate-limited 5/hour/IP; registering per-test flaked across retries/reruns. Register once in beforeAll and share the authed context; the anon test uses the default unauthenticated request fixture (config baseURL, no hardcoded localhost).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements PR #1 of the competitor-intel roadmap (docs/growth/09-pipeline-adex-integration.md §3):
CompetitorCreative+RemixJobPrisma models (docs/growth/07 §4 + 09'ssegmentPlanaddition), purely additive, migration20260710180000_competitor_intel.POST /api/ingest/competitor?org=— HMAC-authed (clonesingest/scenes's pattern), idempotent upsert on(orgId, source, externalId), GCS media handling (link directly if already on our bucket, fetch+upload otherwise).GET /api/competitors— session-scoped listing (requireAuthWithOrg, never trusts clientorgId), filter byappName/level, order byadDays/impressions.seedance2.tsfixes found via real Ark calls — read the generated video URL fromcontent[].video_url.urlfirst (was assumingoutput.video_url), and rounddurationto whole seconds (Ark 400s on a fractional value).e2e/competitor-ingest.spec.ts— HMAC happy path (create then idempotent update), bad-signature 401,GET /api/competitorsauth requirement + listing.docs/growth/07-competitor-intel-remix.md,08-competitor-creative-pipeline.md,09-pipeline-adex-integration.md(this PR's spec).Verification results
npm run lint— 0 errors, 34 pre-existing warnings (none from this diff).npx tsc --noEmit— clean.npm run build— succeeds;/api/ingest/competitorand/api/competitorsboth registered as dynamic routes.npx prisma validate— schema valid.prisma migrate diff --from-schema <pre-change schema> --to-schema prisma/schema.prisma --script(no local Postgres available in the worktree — see.claude/rules/schema.md's Cloud SQL/shadow-DB constraints). Pure additive DDL (2CREATE TABLE, indexes, FKs toOrganization).e2e/smoke.spec.ts— 14/14 pass locally (no regression).e2e/competitor-ingest.spec.ts— could not run to completion locally (noDATABASE_URL/live Postgres in this environment). Ran it anyway to confirm wiring: it fails at the expectedDATABASE_URL is not setboundary inside/api/auth/register, and the one test that doesn't need a DB (GET /api/competitors requires a logged-in session→ 401) passes. Needs a real run against a live DB before merge sign-off.Scope notes
Per docs/growth/09 §3, explicitly not in this PR: daily-sync cron (Phase 3), the Remix engine API (
/api/creatives/remix, Phase 2), competitor panel UI (next PR).🤖 Generated with Claude Code