feat: proxy-aware transport (--use-env-proxy), backfill CLI, retry improvements

[tonyxiao]

Summary

IR-readiness improvements across the proxy/transport layer, a new backfill CLI, and observability enhancements.

Proxy-aware transport refactor (Node 24 --use-env-proxy)

• Remove manual undici ProxyAgent dispatchers — Node 24's --use-env-proxy flag makes fetch respect HTTPS_PROXY/HTTP_PROXY natively, replacing the custom fetchWithProxy/withFetchProxy plumbing with a simpler tracedFetch wrapper
• assertUseEnvProxy() guard (packages/ts-cli/src/env-proxy.ts) — throws at startup if proxy env vars are set without --use-env-proxy, preventing silent proxy bypass. Bun is exempt (respects proxy natively)
• Docker entrypoints updated to pass --use-env-proxy to both engine and service containers
• Signal propagation — makeApiFetch(signal) + combineSignals ensure abort signals flow through to OpenAPI spec fetches and retries (withHttpRetry now accepts signal)

mitmweb intercept tooling

• scripts/mitmweb-env.sh — sources into a shell to route Node/Bun/curl through mitmweb; auto-installs mitmproxy if missing, auto-detects upstream proxy from env. Uses a portable _port_listening helper (ss → lsof → nc) so it works on both Linux and macOS
• scripts/mitmweb-env.test.sh — verifies curl, Node fetch, Bun fetch, and ws WebSocket all route through the proxy
• New mitmweb CI job runs the intercept test in parallel

Backfill CLI command

• pipelineSyncUntilComplete (apps/engine/src/lib/backfill.ts) — loops pipeline_sync until eof:complete, carrying state forward across state_limit/time_limit boundaries
• backfill CLI subcommand (apps/engine/src/cli/backfill.ts) — calls a remote sync engine repeatedly with state persistence to a JSON file

Retry & observability improvements

• Retry log labels — withHttpRetry now accepts a label option (e.g. GET /v1/customers) for easier debugging; error messages included in retry output
• Curl-style trace logs — tracedFetch emits reproducible curl commands when DANGEROUSLY_VERBOSE_LOGGING=true
• Fault-tolerant getAccountCreatedTimestamp — catches errors and falls back to STRIPE_LAUNCH_TIMESTAMP instead of halting backfill
• getAccount retry restored — fixed accidental removal of retry wrapper from an earlier commit

Housekeeping

• Test files in ts-cli moved next to source files (co-located test pattern)

Test plan

• All CI checks pass (lint, build, test, E2E Stripe, E2E Docker, E2E CDN, E2E Test Server, mitmweb)
• No merge conflicts with v2 base branch

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces a new apps/engine sync engine implementation (HTTP API + CLI + connector resolution), adds a new apps/service package, and adds a React dashboard app—along with Docker/CI/repo-ops updates. The scope appears substantially larger than the PR title/description suggests.

Changes:

• Add a new streaming-based sync engine library/API/CLI in apps/engine (NDJSON streaming, connector resolver, progress/state utilities, OpenAPI schema generation + validation).
• Add a new apps/dashboard React UI with Playwright e2e coverage and typed API clients via openapi-fetch.
• Update repo operations: new multi-target root Dockerfile, new audit workflow, docs/agent guidance changes, and various tooling configs.

Reviewed changes

Copilot reviewed 91 out of 699 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
apps/service/src/tests/openapi.test.ts	Adds OpenAPI 3.1 schema validation test for the Service spec.
apps/service/package.json	Introduces the @stripe/sync-service package manifest and scripts.
apps/engine/vitest.config.ts	Adds Vitest configuration (exclusions + timeouts).
apps/engine/tsconfig.json	Adds engine TS build config (dist output + excludes tests).
apps/engine/src/serve-command.ts	Adds a serveAction to run the engine server with connector resolution.
apps/engine/src/logger.ts	Adds pino logger config with redaction settings.
apps/engine/src/lib/state-store.ts	Adds state store abstraction + file/readonly implementations.
apps/engine/src/lib/source-test.ts	Adds test source connector implementation for engine tests.
apps/engine/src/lib/source-exec.ts	Adds subprocess-backed Source wrapper (exec-based connector).
apps/engine/src/lib/select-state-store.ts	Adds dynamic destination state-store selection via optional package import.
apps/engine/src/lib/resolver.ts	Adds connector resolution (specifier/bin) + resolver cache + schema extraction.
apps/engine/src/lib/resolver.test.ts	Adds tests for resolver behavior and caching.
apps/engine/src/lib/remote-engine.ts	Adds HTTP client implementing the Engine interface using streaming endpoints.
apps/engine/src/lib/remote-engine.test.ts	Adds integration tests for the remote engine client against an in-memory server.
apps/engine/src/lib/progress.ts	Adds progress tracking/enrichment and record counting for sync output.
apps/engine/src/lib/progress.test.ts	Adds tests for progress enrichment and state merging.
apps/engine/src/lib/pipeline.ts	Adds pipeline stages (catalog enforcement, state persistence, limits, piping).
apps/engine/src/lib/ndjson.ts	Adds NDJSON parsing/serialization helpers (string/chunks/streams).
apps/engine/src/lib/ndjson.test.ts	Adds tests for NDJSON parsing across chunk boundary scenarios.
apps/engine/src/lib/index.ts	Re-exports engine library surface area.
apps/engine/src/lib/exec.test.ts	Adds tests for exec-based connector wrappers using real connector binaries.
apps/engine/src/lib/exec-helpers.ts	Adds spawn helpers for streaming/collecting subprocess NDJSON.
apps/engine/src/lib/engine.ts	Adds the core in-process Engine implementation (streams everywhere).
apps/engine/src/lib/destination-test.ts	Adds test destination connector implementation for engine tests.
apps/engine/src/lib/destination-filter.ts	Adds catalog field selection pruning middleware for destinations.
apps/engine/src/lib/destination-filter.test.ts	Adds tests for destination field-selection pruning.
apps/engine/src/lib/destination-exec.ts	Adds subprocess-backed Destination wrapper (exec-based connector).
apps/engine/src/lib/default-connectors.ts	Defines default in-process connectors bundled with engine.
apps/engine/src/lib/createSchemas.ts	Builds typed connector config/input schemas for OpenAPI generation.
apps/engine/src/lib/backfill.ts	Adds retry-until-complete helper for paging/backfill via repeated sync calls.
apps/engine/src/lib/backfill.test.ts	Adds tests for backfill retry and state carry-forward.
apps/engine/src/index.ts	Exposes engine library + API factory + NDJSON response helper.
apps/engine/src/cli/sync.ts	Adds a sync CLI shortcut for Stripe→Postgres with optional state handling.
apps/engine/src/cli/supabase.ts	Adds Supabase install/uninstall CLI integration.
apps/engine/src/cli/index.ts	Adds CLI entrypoint.
apps/engine/src/cli/command.ts	Adds CLI program builder + shared connector discovery flags.
apps/engine/src/cli/backfill.ts	Adds backfill CLI that retries a remote engine until EOF complete.
apps/engine/src/api/openapi-utils.ts	Adds helper utilities for OpenAPI doc patching/table printing.
apps/engine/src/api/index.ts	Adds a runnable API server entrypoint with Node/Bun branching.
apps/engine/src/api/header-size.test.ts	Adds tests validating support for large X-Pipeline header sizes.
apps/engine/src/tests/sync.test.ts	Adds dockerized Postgres lifecycle test for sync & resume flow.
apps/engine/src/tests/stripe-to-postgres.test.ts	Adds Stripe-mock to Postgres integration tests for selective sync and resume.
apps/engine/src/tests/openapi.test.ts	Adds engine OpenAPI validation and schema/content-key assertions.
apps/engine/src/tests/docker.test.ts	Adds Docker image build/run smoke tests for engine CLI/server.
apps/engine/package.json	Introduces the @stripe/sync-engine package manifest and exports.
apps/dashboard/vitest.config.ts	Adds dashboard Vitest config with @ alias.
apps/dashboard/vite.config.ts	Adds Vite config including proxying to engine/service backends.
apps/dashboard/tsconfig.json	Adds strict TS config + workspace OpenAPI type paths.
apps/dashboard/src/pages/PipelineList.tsx	Adds pipeline list UI with delete action.
apps/dashboard/src/pages/PipelineDetail.tsx	Adds pipeline detail UI including polling and progress display.
apps/dashboard/src/pages/PipelineCreate.tsx	Adds pipeline creation wizard with connector config + stream selection.
apps/dashboard/src/main.tsx	Adds React app bootstrap.
apps/dashboard/src/lib/utils.ts	Adds Tailwind/clsx class merging helper.
apps/dashboard/src/lib/stream-groups.ts	Adds stream grouping/filtering heuristics for UI.
apps/dashboard/src/lib/stream-groups.test.ts	Adds unit tests for stream grouping/filtering heuristics.
apps/dashboard/src/lib/api.ts	Adds typed engine/service API clients and discovery helper.
apps/dashboard/src/index.css	Adds Tailwind import entry.
apps/dashboard/src/components/StreamSelector.tsx	Adds grouped stream selector component with search and bulk select.
apps/dashboard/src/components/JsonSchemaForm.tsx	Adds dynamic form renderer for JSON Schema connector configs.
apps/dashboard/src/App.tsx	Adds minimal client-side routing + top-level app layout.
apps/dashboard/playwright.config.ts	Adds Playwright e2e configuration using system Chrome.
apps/dashboard/package.json	Adds dashboard package manifest and scripts.
apps/dashboard/index.html	Adds dashboard HTML entry.
apps/dashboard/e2e/pipeline-create.test.ts	Adds e2e test for full pipeline creation flow.
apps/dashboard/e2e/global-teardown.ts	Adds e2e teardown that stops the engine server.
apps/dashboard/e2e/global-setup.ts	Adds e2e setup that starts an engine server for tests.
apps/dashboard/Dockerfile	Adds a dev Dockerfile for dashboard (pnpm + Vite).
apps/dashboard/.gitignore	Adds dashboard-specific gitignore.
README.md	Removes the root README content.
Dockerfile	Replaces root Dockerfile with multi-target build (engine + service) using pnpm deploy.
CHANGELOG.md	Adds changelog content including a 2026-04-11 entry.
AGENTS.md	Rewrites agent instructions and repo conventions.
.vscode/settings.json	Updates VS Code workspace settings (Deno path + Ruby LSP + explorer sort).
.vscode/extensions.json	Adds recommended VS Code extension list.
.verdaccio/config.yaml	Adds Verdaccio config for local/private registry workflows.
.ruby-version	Adds pinned Ruby version file.
.releaserc.json	Removes semantic-release config.
.prettierignore	Expands ignore list (generated outputs, terraform, etc.).
.npmrc	Adds configurable @stripe registry via env var.
.github/workflows/release.yml	Changes release workflow to “promote SHA” flow (manual fallback).
.github/workflows/docs.yml	Removes docs publish workflow.
.github/workflows/audit.yml	Adds a scheduled/manual repo audit workflow.
.githooks/pre-push	Adds pre-push hook to validate generated OpenAPI specs are up to date.
.eslintrc.js	Removes root ESLint config file.
.dockerignore	Updates dockerignore (no longer ignores dist).

Files not reviewed (2)

• apps/engine/src/generated/openapi.json: Language not supported
• apps/service/src/generated/openapi.json: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The PR title/description focuses on mitmweb proxy intercept tooling and a new CI job, but the diff is dominated by adding new apps (apps/engine, apps/service, apps/dashboard), Docker/CI/tooling changes, and numerous tests. Either the PR description needs to be updated to match the actual scope, or the unrelated changes should be split into separate PRs to keep reviewable boundaries and reduce merge risk.

[Copilot]

createConnectorResolver and createApp are async in this codebase (used with await elsewhere). Here they are called without await, so resolver/app will be Promises at runtime and serve({ fetch: app.fetch, ... }) will fail. Fix by awaiting both calls (and ensure any subsequent uses handle the resolved values).

Suggested change

	const resolver = createConnectorResolver({
	sources: { stripe: sourceStripe },
	destinations: { postgres: destinationPostgres, google_sheets: destinationGoogleSheets },
	})
	const app = createApp(resolver)
	const resolver = await createConnectorResolver({
	sources: { stripe: sourceStripe },
	destinations: { postgres: destinationPostgres, google_sheets: destinationGoogleSheets },
	})
	const app = await createApp(resolver)

[Copilot]

createEngine is declared as async and returns Promise<Engine>, but this test calls it without await and then immediately calls engine.source_discover(...). This will throw at runtime because engine is a Promise. Update all occurrences in this file to const engine = await createEngine(...).

[Copilot]

Splitting PATH with ':' is not portable (Windows uses ;). Use path.delimiter from node:path (or an equivalent helper) when splitting PATH to keep connector discovery cross-platform.

[Copilot]

The whitespace split does not respect shell quoting/escaping, so commands like node -e \"console.log('a b')\" or paths with spaces will be split incorrectly. Consider either (1) storing command + args as an array throughout the resolver/commandMap path, or (2) using a small shell-quote parser to correctly split strings when you must accept a single command string.

[Copilot]

This test depends on fetching the OpenAPI 3.1 schema from the public internet at runtime. That can make CI flaky or fail in restricted environments. Prefer vendoring/pinning the schema locally (or using a resolver/cache supported by the validator) so tests are deterministic and do not require external network access.

Suggested change

	const output = await validate(
	'https://spec.openapis.org/oas/3.1/schema-base',
	serviceSpec as unknown as Json
	)
	const output = await validate(serviceSpec as unknown as Json)

[Copilot]

The dockerignore no longer excludes dist/, which can significantly inflate the Docker build context and slow down builds (especially in CI). If dist/ is generated locally, it should typically be excluded from the build context unless there's a specific need to include it.

Suggested change

	node_modules
	node_modules
	dist

[Copilot]

Setting maxHeaderSize to 50 MB materially increases memory usage per connection and can amplify DoS risk (large headers are buffered/parsed before app handling). If large payloads are required, consider moving pipeline/state off headers (e.g., request body for state and an ID reference header), or at minimum make this limit configurable via env with a smaller default and document required reverse-proxy limits.