feat(service): Google Sheets workflow — 3 parallel loops, generic read activity

[tonyxiao]

Summary

• Refactor googleSheetPipelineWorkflow to use 3 parallel loops (liveEventLoop, reconcileLoop, writeLoop) matching the pipelineWorkflow pattern
• Single GoogleSheetWorkflowState object replaces scattered booleans (setupDone, readComplete, pendingWrites)
• derivePipelineStatus() + setState() pattern for consistent status reporting
• EOF-based completion detection replaces deepEqual
• Rename readGoogleSheetsIntoQueue → readIntoQueue (now generic — reads from source into Kafka)
• Move row key/number augmentation from read to write activity (only write is Sheets-specific)
• Extract immutable mergeStateMessage() utility for consistent state merge across activities
• Rename RowIndex → RowIndexMap, rowAssignments → rowIndexMap for clarity
• Rename connector type google-sheets → google_sheets (snake_case on the wire)

Test plan

• pnpm build passes
• pnpm test — all 23 service tests pass
• Workflow tests cover: backfill → ready, live events, pause/resume, delete/teardown, catalog discovery

🤖 Generated with Claude Code

[tonyxiao]

Closing in favor of #253 which targets v2

[Copilot]

Pull request overview

This PR introduces a broader “engine + service + dashboard + supabase integration” structure and refactors the Google Sheets Temporal workflow to follow the shared pipeline workflow pattern, with queue-backed reads/writes and improved workflow state handling.

Changes:

• Refactors the Google Sheets Temporal workflow into 3 parallel loops (live events, reconcile/read, write) with unified workflow state + status derivation.
• Adds Supabase edge functions for Stripe webhook ingestion and cron-driven sync, plus bundling/tests for edge function code.
• Adds new engine/service/dashboard apps, CLI entrypoints, Docker targets, and supporting utilities/tests/config.

Reviewed changes

Copilot reviewed 126 out of 640 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
apps/supabase/src/lib.ts	Updates re-export to use ESM .js extension.
apps/supabase/src/index.ts	Adds a barrel entrypoint exporting package modules.
apps/supabase/src/edge-functions/stripe-webhook.ts	Adds Stripe webhook handler edge function using sync connectors.
apps/supabase/src/edge-functions/stripe-sync.ts	Adds cron-like sync edge function with PG-backed state and auth.
apps/supabase/src/edge-functions/deno.json	Updates Deno import map to use pg.
apps/supabase/src/edge-function-code.ts	Adds raw-bundled edge function source exports for deployment.
apps/supabase/src/tests/supabase.e2e.test.ts	Adds Supabase E2E tests covering webhook + backfill flows.
apps/supabase/src/tests/install.e2e.test.ts	Adds install/uninstall E2E test harness (not CI-enabled).
apps/supabase/src/tests/edge-runtime.smoke.test.ts	Adds dockerized edge-runtime smoke test.
apps/supabase/src/tests/deploy.e2e.test.ts	Adds deploy/invoke/delete edge function E2E test (not CI-enabled).
apps/supabase/src/tests/bundle.test.ts	Adds tests asserting edge-function code is bundled into dist output.
apps/supabase/package.json	Introduces publishable @stripe/sync-integration-supabase package manifest.
apps/supabase/build.mjs	Adds esbuild-based bundling pipeline including custom ?raw loader.
apps/service/vitest.integration.config.ts	Adds Vitest integration test config.
apps/service/vitest.config.ts	Adds Vitest unit test config.
apps/service/tsconfig.json	Adds TypeScript build configuration for service app.
apps/service/src/temporal/workflows/pipeline-workflow.ts	Adds generic pipeline workflow implementation (live + reconcile).
apps/service/src/temporal/workflows/index.ts	Exports workflow entrypoints.
apps/service/src/temporal/workflows/google-sheet-pipeline-workflow.ts	Refactors Google Sheets workflow into live/reconcile/write loops.
apps/service/src/temporal/workflows/_shared.ts	Centralizes workflow signals + proxied activities types.
apps/service/src/temporal/worker.ts	Adds worker factory wiring activities and workflows path.
apps/service/src/temporal/activities/write-google-sheets-from-queue.ts	Adds queue consumer → Google Sheets writer activity with row augmentation.
apps/service/src/temporal/activities/update-pipeline-status.ts	Adds activity to persist pipeline status updates.
apps/service/src/temporal/activities/read-into-queue.ts	Adds generic read-into-Kafka activity for queue-backed workflows.
apps/service/src/temporal/activities/pipeline-teardown.ts	Adds pipeline teardown activity wrapper.
apps/service/src/temporal/activities/pipeline-sync.ts	Adds pipeline sync activity wrapper that persists connector config updates.
apps/service/src/temporal/activities/pipeline-setup.ts	Adds pipeline setup activity wrapper that persists connector config updates.
apps/service/src/temporal/activities/index.ts	Registers activities for the Temporal worker.
apps/service/src/temporal/activities/discover-catalog.ts	Adds catalog discovery activity (uses source discover + selection).
apps/service/src/temporal/activities/_shared.ts	Adds shared activity context (remote engine + Kafka queue utilities).
apps/service/src/logger.ts	Adds pino logger with redaction defaults.
apps/service/src/lib/utils.ts	Adds retry policy + constants (+ deepEqual utility).
apps/service/src/lib/stores.ts	Defines PipelineStore interface.
apps/service/src/lib/stores-memory.ts	Adds in-memory PipelineStore implementation for tests.
apps/service/src/lib/stores-fs.ts	Adds filesystem-backed PipelineStore implementation.
apps/service/src/lib/createSchemas.ts	Adds service-level schema factory (desired/workflow statuses + pipeline schemas).
apps/service/src/index.ts	Adds service barrel exports (schemas, app factory, Temporal bindings).
apps/service/src/cli.ts	Adds service CLI commands (serve/worker/webhook + OpenAPI-driven commands).
apps/service/src/cli.test.ts	Adds unit tests for CLI worker flag plumbing.
apps/service/src/api/index.ts	Adds API barrel exports.
apps/service/src/tests/openapi.test.ts	Adds OpenAPI schema validation test for service spec.
apps/service/package.json	Introduces publishable @stripe/sync-service package manifest.
apps/engine/vitest.config.ts	Adds Vitest config for engine app.
apps/engine/tsconfig.json	Adds TypeScript build configuration for engine app.
apps/engine/src/serve-command.ts	Adds engine serve action with connector discovery flags.
apps/engine/src/logger.ts	Adds pino logger with optional pretty transport and redaction.
apps/engine/src/lib/state-store.ts	Adds StateStore interface + readonly implementation.
apps/engine/src/lib/source-test.ts	Adds test source connector for engine tests.
apps/engine/src/lib/source-exec.ts	Adds subprocess-backed source connector wrapper.
apps/engine/src/lib/select-state-store.ts	Adds optional destination-colocated state store resolver.
apps/engine/src/lib/resolver.test.ts	Adds resolver tests (specifier, bin resolution, caching, fallback).
apps/engine/src/lib/remote-engine.ts	Adds HTTP remote-engine implementing Engine interface over OpenAPI+NDJSON.
apps/engine/src/lib/remote-engine.test.ts	Adds remote-engine integration tests against in-process HTTP server.
apps/engine/src/lib/pipeline.ts	Adds pipeline composition helpers (catalog enforcement, limits, logging, pipe).
apps/engine/src/lib/ndjson.ts	Adds NDJSON parsing/streaming utilities.
apps/engine/src/lib/ndjson.test.ts	Adds tests covering NDJSON parsing for strings/chunks/streams.
apps/engine/src/lib/index.ts	Adds engine library barrel exports.
apps/engine/src/lib/exec.test.ts	Adds tests covering exec wrappers using real built connector bins.
apps/engine/src/lib/exec-helpers.ts	Adds subprocess helpers (spawn + NDJSON streaming + splitCmd).
apps/engine/src/lib/destination-test.ts	Adds test destination connector for engine tests.
apps/engine/src/lib/destination-filter.ts	Adds catalog selection filter (fields + primary keys).
apps/engine/src/lib/destination-filter.test.ts	Adds tests for catalog selection filtering.
apps/engine/src/lib/destination-exec.ts	Adds subprocess-backed destination connector wrapper.
apps/engine/src/lib/default-connectors.ts	Adds default bundled connectors (stripe/postgres/google_sheets).
apps/engine/src/lib/createSchemas.ts	Adds connector-driven Zod schema factory + OpenAPI naming helpers.
apps/engine/src/index.ts	Adds engine public entrypoint exports (library + app factory).
apps/engine/src/cli/supabase.ts	Adds CLI commands for Supabase install/uninstall integration.
apps/engine/src/cli/index.ts	Adds CLI bin entrypoint.
apps/engine/src/cli/command.ts	Adds CLI program builder with connector discovery flags + OpenAPI CLI.
apps/engine/src/api/openapi-utils.ts	Adds OpenAPI patch helpers (endpoint table + ControlMessage patching).
apps/engine/src/api/index.ts	Adds standalone “engine API” entrypoint server.
apps/engine/src/tests/sync.test.ts	Adds dockerized Postgres tests for sync checkpoints/resume.
apps/engine/src/tests/stripe-to-postgres.test.ts	Adds stripe-mock → postgres integration tests + selective sync/backfill tests.
apps/engine/src/tests/openapi.test.ts	Adds OpenAPI spec validation and schema sanity tests.
apps/engine/src/tests/docker.test.ts	Adds Docker image build/run tests for engine distribution.
apps/engine/package.json	Introduces publishable @stripe/sync-engine app package manifest.
apps/dashboard/vitest.config.ts	Adds Vitest config for dashboard.
apps/dashboard/vite.config.ts	Adds Vite config including proxying to engine/service APIs.
apps/dashboard/tsconfig.json	Adds TS config with OpenAPI type paths for engine/service.
apps/dashboard/src/pages/PipelineList.tsx	Adds pipeline list UI with delete and status badge.
apps/dashboard/src/pages/PipelineDetail.tsx	Adds pipeline detail UI with pause/resume/delete and stream table list.
apps/dashboard/src/main.tsx	Adds dashboard React entrypoint.
apps/dashboard/src/lib/utils.ts	Adds Tailwind class merge helper.
apps/dashboard/src/lib/stream-groups.ts	Adds stream grouping/filter heuristics for UI and helpers.
apps/dashboard/src/lib/stream-groups.test.ts	Adds tests for stream grouping/filtering.
apps/dashboard/src/lib/api.ts	Adds typed API clients for engine/service via openapi-fetch.
apps/dashboard/src/index.css	Adds Tailwind entrypoint CSS import.
apps/dashboard/src/components/StreamSelector.tsx	Adds grouped stream selector UI with expand/search/select-all.
apps/dashboard/src/components/JsonSchemaForm.tsx	Adds JSON-schema-driven config form renderer.
apps/dashboard/src/App.tsx	Adds minimal SPA routing and page switching.
apps/dashboard/playwright.config.ts	Adds Playwright E2E configuration.
apps/dashboard/package.json	Introduces dashboard app manifest and test scripts.
apps/dashboard/index.html	Adds dashboard HTML entrypoint.
apps/dashboard/e2e/pipeline-create.test.ts	Adds E2E test for full pipeline creation flow.
apps/dashboard/e2e/global-teardown.ts	Adds E2E teardown for engine server.
apps/dashboard/e2e/global-setup.ts	Adds E2E setup to boot engine API during e2e.
apps/dashboard/Dockerfile	Adds dashboard dev Dockerfile.
apps/dashboard/.gitignore	Adds dashboard-specific ignore rules.
README.md	Removes top-level README content (repo docs likely moved elsewhere).
Dockerfile	Replaces single-image Dockerfile with multi-target engine/service images.
AGENTS.md	Replaces agent notes with expanded monorepo architecture + conventions.
.vscode/settings.json	Updates workspace settings (Deno paths + Ruby LSP config).
.vscode/extensions.json	Adds VS Code extension recommendations.
.verdaccio/config.yaml	Adds Verdaccio configuration for local npm registry.
.ruby-version	Pins Ruby version for infra tooling.
.prettierignore	Expands ignore list (docs/out, terraform, generated files).
.npmrc	Adds registry indirection for @stripe/* via environment config.
.github/workflows/release.yml	Reworks release workflow to “promote” packages by SHA.
.github/workflows/docs.yml	Removes docs publish workflow.
.github/workflows/audit.yml	Adds scheduled repo audit workflow gated by secret availability.
.githooks/pre-push	Adds pre-push hook to ensure OpenAPI specs are generated/checked.
.eslintrc.js	Removes root ESLint config (likely migrated elsewhere).
.dockerignore	Adjusts dockerignore entries (no longer ignores dist/).

Files not reviewed (2)

• apps/engine/src/generated/openapi.json: Language not supported
• apps/service/src/generated/openapi.json: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The workflow ignores result.errors coming back from writeGoogleSheetsFromQueue. Since the activity currently returns errors instead of throwing, write failures can be silently dropped and the workflow will continue as if the queue was handled successfully. Recommended fix: either (mandatory) make the activity throw when errors.length > 0, or (also acceptable) handle result.errors here by failing the workflow / updating pipeline status to error and stopping the loops—otherwise you risk silent data loss.

Suggested change

	if (result.errors.length > 0) {
	throw new Error(
	`writeGoogleSheetsFromQueue failed with ${result.errors.length} error(s): ${result.errors
	.map((error) => String(error))
	.join('; ')}`,
	)
	}

[Copilot]

The workflow ignores result.errors coming back from writeGoogleSheetsFromQueue. Since the activity currently returns errors instead of throwing, write failures can be silently dropped and the workflow will continue as if the queue was handled successfully. Recommended fix: either (mandatory) make the activity throw when errors.length > 0, or (also acceptable) handle result.errors here by failing the workflow / updating pipeline status to error and stopping the loops—otherwise you risk silent data loss.

[Copilot]

This activity collects connector error traces into errors but never throws. In Temporal, returning an errors array is easy to ignore (and currently is ignored by the workflow), so failures may not trigger retries and can lead to silent partial sync. Mandatory fix: if errors.length > 0, throw an Error (ideally preserving failure_type / stream in the message) so the activity fails and Temporal retry policy applies.

[Copilot]

This activity collects connector error traces into errors but never throws. In Temporal, returning an errors array is easy to ignore (and currently is ignored by the workflow), so failures may not trigger retries and can lead to silent partial sync. Mandatory fix: if errors.length > 0, throw an Error (ideally preserving failure_type / stream in the message) so the activity fails and Temporal retry policy applies.

Suggested change

	if (errors.length > 0) {
	const details = errors
	.map((error) => {
	const errorRecord = error as Record<string, unknown>
	const failureType =
	typeof errorRecord.failure_type === 'string' ? errorRecord.failure_type : 'unknown_failure'
	const stream = typeof errorRecord.stream === 'string' ? errorRecord.stream : 'unknown_stream'
	return `${failureType}:${stream}`
	})
	.join(', ')
	throw new Error(`Google Sheets write failed with connector errors: ${details}`)
	}

[Copilot]

KafkaJS consumer.run() returns a Promise that can reject (e.g., broker errors). Here it’s not awaited or captured, so failures can surface as unhandled promise rejections and the activity may continue/commit offsets in an undefined state. Recommended fix (mandatory for robustness): capture the Promise from consumer.run(), ensure it’s awaited (or stopped/cancelled deterministically on finish()), and propagate errors so the activity fails and retries cleanly.

[Copilot]

Relying on @ts-ignore for ?raw imports suppresses type-checking broadly and can mask real issues. Recommended fix: add a narrow ambient module declaration for *.ts?raw (or *?raw) so these imports are typed as string without suppressing TypeScript errors.

Suggested change

	// @ts-ignore
	import setupCodeRaw from './edge-functions/stripe-setup.ts?raw'
	// @ts-ignore
	import webhookCodeRaw from './edge-functions/stripe-webhook.ts?raw'
	// @ts-ignore
	import syncCodeRaw from './edge-functions/stripe-sync.ts?raw'
	export const setupFunctionCode = setupCodeRaw as string
	export const webhookFunctionCode = webhookCodeRaw as string
	export const syncFunctionCode = syncCodeRaw as string
	declare module './edge-functions/stripe-setup.ts?raw' {
	const content: string
	export default content
	}
	declare module './edge-functions/stripe-webhook.ts?raw' {
	const content: string
	export default content
	}
	declare module './edge-functions/stripe-sync.ts?raw' {
	const content: string
	export default content
	}
	import setupCodeRaw from './edge-functions/stripe-setup.ts?raw'
	import webhookCodeRaw from './edge-functions/stripe-webhook.ts?raw'
	import syncCodeRaw from './edge-functions/stripe-sync.ts?raw'
	export const setupFunctionCode = setupCodeRaw
	export const webhookFunctionCode = webhookCodeRaw
	export const syncFunctionCode = syncCodeRaw

[Copilot]

iteration is hard-coded to 0, so the iteration UI can never render and the variable is effectively dead code. Recommended fix: either remove this block entirely or wire it to a real value from the service response (if available) so it reflects actual workflow iteration/continue-as-new behavior.

Suggested change

	const iteration = 0
	const iteration =
	typeof (pipeline as Pipeline & { iteration?: number }).iteration === 'number'
	? (pipeline as Pipeline & { iteration?: number }).iteration
	: undefined

[Copilot]

iteration is hard-coded to 0, so the iteration UI can never render and the variable is effectively dead code. Recommended fix: either remove this block entirely or wire it to a real value from the service response (if available) so it reflects actual workflow iteration/continue-as-new behavior.

Suggested change

{iteration > 0 && <p className="text-xs text-gray-400">Iteration {iteration}</p>}

[Copilot]

Catching and swallowing all errors will also hide operational failures (e.g., filesystem permission issues, transient I/O errors), leaving workflow status stale and making incidents harder to detect. Recommended fix: only suppress the specific 'not found' case (if your PipelineStore distinguishes it), and otherwise log or rethrow so failures are observable.

[Copilot]

With dist/ no longer excluded, Docker build context can grow significantly (especially in a monorepo), slowing local and CI builds and reducing layer cache efficiency. Recommended fix: re-add dist/ (and any other build outputs) to .dockerignore, since the Dockerfile already builds and copies artifacts from build stages.