feat(cli): add conduit connector new / conduit processor new scaffolding

[devarismeroxa]

Summary

Implements conduit connector new --lang go and conduit processor new --lang go
per docs/design-documents/20260707-connector-processor-scaffolding.md (open in
#2571 — that design doc + the shared 20260707-cli-output-conventions.md spec
this PR follows are not yet merged; this PR's exit codes / error envelope /
preflight wiring were built directly against their reviewed content regardless
of merge order). Roadmap: Phase 1 "Plugin scaffolding" (ROADMAP.md).

Approach (design doc's Option C): orchestrate the canonical upstream
templates (ConduitIO/conduit-connector-template,
ConduitIO/conduit-processor-template) rather than reimplementing their
boilerplate. pkg/scaffold/template vendors a pinned go:embed'd snapshot of
each (SHAs in embed.go) and Go-ports each template's setup.sh rename step
(Rewrite) — no shell/sed/bash dependency. pkg/scaffold/steps installs the
right code-gen tool and runs it, pkg/scaffold/scaffold.go's Generate
orchestrates preflight -> stage -> rewrite -> generate -> verified go build ./... -> git init -> atomic rename into place.

The connector/processor asymmetry (the top risk called out in review)

Verified against both templates' actual setup.sh + Makefile (cloned at the
pinned SHAs) before writing any code:

• Connector setup.sh: rename -> (if conn-sdk-cli missing) make install-tools -> make generate, which runs go generate ./... (triggers
  //go:generate conn-sdk-cli specgen in connector.go) then conn-sdk-cli readmegen -w.
• Processor setup.sh: rename only — it does not call
  install-tools/generate at all. Its Makefile's generate target is a
  single go generate ./..., which triggers a different directive,
  //go:generate paramgen -output=paramgen_proc.go ProcessorConfig, in
  processor.go.

pkg/scaffold/steps does not assume symmetry: InstallTool/Generate
each resolve the tool (conn-sdk-cli vs paramgen) from a
map[template.Kind]string, and the engine deliberately always runs
install+generate for both kinds (unlike upstream's processor setup.sh,
which skips it) — a freshly generated repo should reflect a real, verified
run of its own generator, not just whatever happened to be pre-committed in
the snapshot. --skip-generate is the escape hatch (both templates ship
pre-generated output, so the result still builds either way — verified by
TestGenerate_SkipGenerate_StillBuilds).

Must-fix items from the design review — how each was handled

• Error --json envelope carries suggestion/configPath/fix: every
  validate() failure builds a *conduiterr.ConduitError with Suggestion
  set (see request.go's newErr helper); cecdysis.CommandWithResult's
  shared envelope carries it through to --json's error.suggestion.
• Exit codes via pkg/conduit/exitcode: pkg/scaffold/codes.go registers
  scaffold.* conduiterr.Codes classified so toolchain-preflight failure ->
  3 (environment), bad --module/--lang python/destination-exists-without-
  --force -> 2 (validation), a code-gen or go build failure once files are
  staged -> 1 (runtime). Verified end-to-end via the CLI binary and
  pkg/scaffold's tests (exitcode.ExitCode(err) asserted directly).
• --lang python is an honest gated error, not a half-scaffold: exit 2,
  names the blocker (no Python connector SDK), points at
  docs/design-documents/20260707-python-connector-sdk.md.
• Preflight before any write, no partial directory on failure: Generate
  runs preflight (reusing pkg/conduit/check's BinaryOnPath/DirWritable
  — not a forked doctor, per the shared check-engine ADR) before creating
  anything; on any hard failure past that point the temp staging directory
  (a sibling of the destination, same filesystem) is os.RemoveAll'd before
  returning, so the requested destination path never exists in a half-written
  state. --force only removes a pre-existing destination immediately before
  the final rename, after the staged tree has already passed the build step.
• Toolchain preflight via the shared pkg/conduit/check engine: composed
  from check.BinaryOnPath("go", minGoVersion), check.BinaryOnPath("git", ""), check.DirWritable(gopathBin, ""). preflight() deliberately does
  not delegate to check.Report.ExitCode()'s per-check classification
  (that method's DirWritable default is validation-class, which is right for
  doctor's "point config at a writable dir" case but wrong for scaffold's
  "your Go install is broken" case) — every check in this set is an
  environment question, so the whole preflight collapses to one bucket-3
  CodeToolchainUnavailable error. Documented explicitly in preflight.go.

What's covered now vs. deferred

Covered / tested now:

• connector new --lang go s3 --yes scaffolds ./conduit-connector-s3,
  go build ./... compiles with no edits (TestGenerate_Connector, and
  verified manually via the built CLI binary).
• Setup.sh -> Go rewrite is golden-file tested, byte-stable
  (template/rewrite_test.go against template/testdata/golden/{connector,processor}).
• Preflight with go removed from PATH -> exit 3, before any write, no
  partial directory (TestGenerate_MissingToolchain_NoPartialDirectory).
• --lang python -> gated exit 2 (TestGenerate_UnsupportedLanguage).
• --json envelope conforms to the CLI output conventions' shape
  (TestResultMarshalsToTheDocumentedEnvelope, plus a manual jq-able run).
• Processor path produces a building repo using paramgen, not specgen
  (TestGenerate_Processor asserts paramgen_proc.go's generated header).
• Destination-exists-without---force -> exit 2; --force overwrite works.

Deferred to #2575 (tracked, not
blocking) — matches the design doc's own risk-tiering, which already
declares the Windows-CI-green criterion "an EXTERNAL dependency... tracked
separately, not this repo's own deliverable":

1. The <30-min, CI-measured matrix job (ubuntu/macos/windows ×
   connector/processor, clean runner, new -> make test -> make build -> smoke-run in a pipeline).
2. The nightly job that regenerates the embedded snapshot from upstream
   main + latest SDK and fails loudly on drift (today it's a hand-synced
   pin, recorded in template/embed.go's ConnectorRef/ProcessorRef).
3. Upstream Windows-matrix PRs to the two template repos (their test.yml is
   ubuntu-latest only today) — coordinated PRs to those repos, not an
   in-repo edit (which would diverge the vendored snapshot).

Not implemented, documented rather than silently missing: the design
doc's TTY guided-prompt flow for a missing name/module. --yes/-y is
accepted (per the CLI output conventions' flag vocabulary for mutating
commands) but the command is always non-interactive today — a missing name
or --module is a deterministic CodeInvalidModule error either way, not a
prompt.

A go:embed wrinkle worth flagging explicitly

Both templates have their own go.mod (and a nested tools/go.mod), which
makes go:embed's directory-embedding refuse them outright ("cannot embed
directory: in different module"), and — independently — would make this
repo's own go build ./.../golangci-lint/make generate try to treat
their loose .go files as this module's source if the go.mod boundary were
removed some other way. Fixed by suffixing every go.mod and *.go path in
the vendored snapshot with .tmpl on disk (see embed.go's tmplSuffix
doc); Extract restores the real names on write-out. Confirmed
validate-generated-files-equivalent stays green: make generate at the
repo root produces zero diff under pkg/scaffold/template (nested go.mod
correctly excludes it from ./...).

Verification

• go build ./..., go vet ./... clean.
• go test ./pkg/scaffold/... ./cmd/conduit/... -race -count=1 — all green,
  including full end-to-end scaffold.Generate runs for both connector and
  processor that actually shell out to go install/go generate/go build/git and assert the result builds.
• golangci-lint run ./... — 0 issues in every file this PR touches (the
  36 pre-existing issues golangci-lint reports repo-wide are all in
  pkg/plugin/processor/builtin/internal/diff/** and
  pkg/foundation/cerrors/conduiterr/conduiterr.go, none touched here).
• Manually ran the built CLI binary end-to-end: human output (glyphs, step
  timing, "Next steps"), --json, --force, --skip-generate, --no-git,
  missing-toolchain, bad-module, destination-exists, and both connector new
  / processor new — all matched the design doc's UX mockup and exit-code
  table exactly.

Adversarial self-review

• Connector/processor asymmetry — the top named risk. Handled by
  verifying both templates' real setup.sh+Makefile first (not assumed),
  and encoding the tool choice as an explicit map[template.Kind]string in
  steps/install.go with a doc comment pointing at exactly which
  //go:generate directive and Makefile line justifies each entry.
• Atomic rename / no partial directory — the other named risk. Verified
  with dedicated tests (TestGenerate_MissingToolchain_NoPartialDirectory,
  TestGenerate_ValidationFailure_NoWrite) that assert the destination
  directory doesn't exist, and that no staging directory is left behind in
  its parent either.
• Residual, documented, accepted risk: in finalize, if --force's
  os.RemoveAll of a pre-existing destination succeeds but the subsequent
  os.Rename then fails (disk full, a permissions race), the destination is
  left absent rather than reverted — the same risk an ordinary rm -rf old && mv new old has. Not a silent clobber (the error is returned, nothing is
  reported as success); commented explicitly at the call site in
  scaffold.go.
• gosec (G122/G703) flagged rewrite.go's original
  filepath.WalkDir-callback file I/O as symlink-TOCTOU-prone. Fixed for
  real (not suppressed) using os.Root (Go 1.25) scoped at the staging
  directory for every read/write/remove/rename Rewrite performs.
• Ran golangci-lint run ./... repo-wide, not just on touched packages, to
  confirm zero new findings anywhere (see Verification).

Files

• pkg/scaffold/ — engine (scaffold.go, request.go, codes.go,
  preflight.go, sdkversion.go, template/, steps/) + tests.
• cmd/conduit/internal/scaffoldcmd/ — shared command body.
• cmd/conduit/root/connectors/new.go, cmd/conduit/root/processors/new.go
  — thin per-kind wrappers, registered in each group's SubCommands().
• go.mod — github.com/stretchr/testify promoted from indirect to direct
  (this PR's tests import it directly; it was already in the module graph
  transitively, not a new external dependency).

Tier 2 (Features — connectors, processors, CLI; per the design doc's explicit
declaration). No data-path/protocol changes.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Tapg9dLWXKMZJoL65R24vd