feat(seed): run PostGenerationPipeline.VerificationStep for TS SDK fixtures

[jsklan]

What

Threads verify: true through runLocalGenerationForWorkspace for the TypeScript SDK generator, so seed CI exercises the same verify.sh flow customers hit on fern generate --local --verify and on the Fiddle (remote) path.

Refs FER-10455. TS-only — every other generator continues through the existing seed-script-runner path with zero behavior change. FER-9681 will add the rest as each generator's .fern/verify.sh lands.

Why this shape (the design choice)

Previously, seed had its own bash .fern/verify.sh reimplementation that bypassed generator-cli's VerificationStep — no validator container, no exit-code handling, no parity with what customers run. This PR deletes that parallel path and drives the canonical PostGenerationPipeline.VerificationStep instead, gated by a small allow-list of generators that emit verify.sh today (just fernapi/fern-typescript-sdk + its node-sdk alias).

The intent matches what Niels asked for in this Slack thread: exercise the TS verification plumbing harder before expanding to the other 7 languages.

Testing

• Unit tests: assertVerifyPipelineSucceeded.test.ts (7 cases — happy path, verify.sh absent, validator stderr failure, pipeline-failure-with-no-step-result, errorMessage fallback, generic fallback, precedence)
• pnpm turbo run test --filter @fern-api/local-workspace-runner --filter @fern-api/seed-cli — green locally (79/79 tasks)
• pnpm check (Biome) — green
• Manual: end-to-end seed run against the TS SDK fixture exercises the full validator-container path via CI — seed-test-results (ts-sdk) is SUCCESS
• Full CI: 190 SUCCESS / 12 SKIPPED / 0 failing on b7443258

One note for review

Seed runs the generator at :local (Docker tag of the current checkout), but no :local validator image is built — so seed tests run against published fernapi/fern-typescript-sdk-validator:latest rather than the local checkout. This mirrors what a real fern generate --local --verify user against a published generator sees. Trade-off: validator-image changes in a given PR aren't exercised by seed CI in that same PR. Those are covered separately by generator-cli's verification-step.test.ts integration tests.

Diff map (file-by-file)

• packages/cli/generation/local-generation/local-workspace-runner/src/GenerationRunner.ts — RunArgs gains verify, verifyRunner, verifyValidatorVersion. When verify === true, after executeGenerator writes files, GenerationRunner.run instantiates PostGenerationPipeline with only VerificationStep enabled — same shape as the verifyOnlyPipelineEnabled branch in runLocalGenerationForWorkspace.
• packages/cli/generation/local-generation/local-workspace-runner/src/assertVerifyPipelineSucceeded.ts (new) — inspects the pipeline result and throws an InternalError CliError when pipelineResult.success === false. Keys off pipelineResult.success (not just steps.verify.success) so failures where VerificationStep throws before recording a result (e.g. validator image fails to pull) still surface. Detail-message preference: validator stderr → accumulated pipeline errors → step errorMessage → generic fallback.
• packages/seed/src/utils/generatorEmitsVerifyScript.ts (new) — allow-list helper. Today: fernapi/fern-typescript-sdk + fern-typescript-node-sdk alias.
• packages/seed/src/commands/test/test-runner/LocalTestRunner.ts and ContainerTestRunner.ts — consult the allow-list and pass verify: true only for TS SDK; everything else unchanged.
• packages/seed/src/commands/test/test-runner/TestRunner.ts — skips the seed.yml scripts block for allow-list generators (VerificationStep is the canonical replacement). Block stays in seed.yml as fallback for every other generator.
• Removed: packages/seed/src/utils/resolveScriptPhaseCommands.ts + its unit tests; reverted edits in ContainerScriptRunner.ts, LocalScriptRunner.ts, comment in seed/ts-sdk/seed.yml.
• packages/cli/cli/changes/unreleased/seed-verify-pipeline.yml — CLI changelog entry (type: internal).

Devin session

Link: https://app.devin.ai/sessions/bdd66d2adeec4acc826dce1b86476c9b

Requested by: @jsklan

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[jsklan]

Reviewed FER-10455. The approach is clean — resolveScriptPhaseCommands correctly encapsulates the verify.sh detection logic, the old getCommandsForPhase duplication across both script runners is eliminated, and the fallback semantics (verify.sh absent -> legacy seed.yml commands) are exactly right for the cross-generator story.

The delta table in the PR description (frozen-lockfile / unconditional pnpm build / test / no package.json guard) is the right framing — these gaps should surface as CI failures and get fixed in the generator, not papered over. The YAML comment on the seed.yml fallback block is a nice touch for future FER-9681 cleanup.

Unit test coverage (9 cases, both branches for legacy list and phase-map shapes) is complete. CI 32/32 green. Flipping ready.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

[github-actions]

Docs Generation Benchmark Results

Comparing PR branch against median of 5 nightly run(s) on main (latest: 2026-05-12T05:12:12Z).

Fixture	main	PR	Delta
docs	287.8s (n=5)	208.2s (35 versions)	-79.6s (-27.7%)

Docs generation runs fern generate --docs --preview end-to-end against the benchmark fixture with 35 API versions (each version: markdown processing + OpenAPI-to-IR + FDR upload).
Delta is computed against the nightly baseline on main.
Baseline from nightly run(s) on main (latest: 2026-05-12T05:12:12Z). Trigger benchmark-baseline to refresh.
Last updated: 2026-05-12 23:31 UTC

[github-actions]

SDK Generation Benchmark Results

Comparing PR branch against median of 5 nightly run(s) on main (latest: 2026-05-12T05:12:12Z).

Full benchmark table (click to expand)

Generator	Spec	main (generator)	main (E2E)	PR (generator)	Delta
csharp-sdk	square	68s (n=5)	98s (n=5)	60s	-8s (-11.8%)
go-sdk	square	129s (n=5)	275s (n=5)	126s	-3s (-2.3%)
java-sdk	square	212s (n=5)	271s (n=5)	203s	-9s (-4.2%)
php-sdk	square	56s (n=5)	77s (n=5)	48s	-8s (-14.3%)
python-sdk	square	132s (n=5)	230s (n=5)	127s	-5s (-3.8%)
ruby-sdk-v2	square	81s (n=5)	118s (n=5)	86s	+5s (+6.2%)
rust-sdk	square	153s (n=5)	156s (n=5)	153s	+0s (+0.0%)
swift-sdk	square	53s (n=5)	745s (n=5)	47s	-6s (-11.3%)
ts-sdk	square	88s (n=5)	107s (n=5)	227s	+139s (+158.0%)

main (generator): generator-only time via --skip-scripts (includes Docker image build, container startup, IR parsing, and code generation — this is the same Docker-based flow customers use via fern generate). main (E2E): full customer-observable time including build/test scripts (nightly baseline, informational). Delta is computed against generator-only baseline.
⚠️ = generation exited with a non-zero exit code (timing may not reflect a successful run).
Baseline from nightly runs on main (latest: 2026-05-12T05:12:12Z). Trigger benchmark-baseline to refresh.
Last updated: 2026-05-12 23:33 UTC