feat(generate): eval harness + benchmark corpus (design-only path)

[devarismeroxa]

Summary

Ships v0.19 Workstream 1's committed deliverable for conduit generate — per
docs/design-documents/20260722-conduit-generate.md's Status section, the command
implementation itself is deferred to v0.20 (the v0.19 build slot went to the Python
connector SDK). No generate command, MCP tool, or LLM provider client ships in
this PR. What ships is the frozen acceptance bar the future command must clear:

• Benchmark corpus (cmd/conduit/internal/generate/testdata/eval_requests.yaml,
  rendered for humans in docs/generate-benchmark.md): 28 canonical natural-language
  pipeline requests (≥ 25 required), each grounded in a real built-in connector (file,
  generator, kafka, log, postgres, s3 — respecting that generator is source-only and
  log is destination-only) with its expected semantic intent (source/destination
  category, required processor capabilities).
• Scoring harness (ScoreRun/ScoreMedian in score.go) reporting two
  independent numbers, never collapsed into one:
  1. Validate-pass rate — runs the exact cmd/conduit/internal/validate engine
     conduit pipelines validate uses. Since validate.Run has no in-memory
     (RunBytes/RunReader) seam yet (confirmed absent from the tree), this uses the
     design doc's documented fallback: a private (0600) temp file, validated, then
     removed (validate_score.go) — the one place in the package that touches disk.
  2. Semantic-intent-match rate — parses the candidate with the same YAML parser
     validate uses and checks the actual source/destination connector category and
     processor set against the request's Expect, even when the candidate validates
     cleanly (semantic_score.go). A missing candidate fails both metrics rather
     than shrinking the denominator; repeated runs reduce by median, never
     best-of (CLAUDE.md's benchi discipline, applied to model-output scoring).
• Fixture tests prove the scorer correct without a live LLM (score_test.go, 19
  tests): known-good candidates score true on both axes; known-bad ones — bad schema,
  wrong connector (still validates), dropped filter, swapped direction, an unknown/
  fabricated plugin, malformed YAML, ambiguous multi-source — each score false on
  exactly the axis they're wrong on.

Adversarial self-review

Re-read the diff hunting specifically for the classes CLAUDE.md flags:

• Resource cleanup: validateCandidate's temp file is removed via defer on
  every return path, including the write/close error paths (verified each early
  return happens after os.CreateTemp and before returning, so the defer is always
  registered first).
• Silent wrong answers (the risk class this whole feature exists to prevent, one
  level down): initially, connectorCategories silently kept the first connector
  found in a role and ignored a second one in the same role — i.e. it would have
  guessed instead of flagging ambiguity. Caught in self-review and fixed: a second
  same-role connector now makes that role's category permanently ambiguous (""),
  which always mismatches a non-empty Expect. Added
  TestScoreSemantic_MultipleSourceConnectors_NeverGuessesACategory to lock this in.
• Error paths: a parse failure that still recovers partial pipelines (the parser's
  documented "return whatever parsed successfully" contract) is judged on what did
  parse, with the error surfaced as an additional issue rather than discarded or
  treated as an unconditional hard failure.
• Concurrency: no shared mutable state; os.CreateTemp's per-call random suffix
  means concurrent scoring (not currently done, but not precluded) wouldn't collide.
• Data-path invariants (1–7): none apply — this package never touches a running
  pipeline, ack/position/checkpoint logic, or any serialized format; it only reads
  YAML text it's given and writes/removes its own private temp file.

Gates run (this PR only — see Process maturity table for what's not live)

• go build ./... — clean.
• golangci-lint run ./cmd/conduit/internal/generate/... — 0 issues.
• go test -race ./cmd/conduit/internal/generate/... — 19/19 pass.
• make generate && git diff --exit-code — clean (no generated-file drift; this PR
  adds no //go:generate directive).
• npx markdownlint-cli2 docs/generate-benchmark.md — 0 errors.
• No --json CLI surface added (no CLI command ships) — Family-A/golden-fixture gate
  N/A.
• No error code added — allcodes.go registration N/A (those are generate.* codes
  owned by the v0.20 command build).
• Full-repo golangci-lint run ./... was attempted but returned findings entirely
  inside a different concurrent worktree's vendored-Go-authors code
  (internal/diff, unrelated to this change) — a sandbox artifact of this
  multi-agent environment, not a finding against this diff. Only the scoped run above
  is claimed as verified.

Tier

Tier 2 (docs/testdata + a self-contained internal package, no CLI/MCP surface, no
data-path code).

Test plan

• go build ./...
• golangci-lint run ./cmd/conduit/internal/generate/...
• go test -race ./cmd/conduit/internal/generate/...
• make generate && git diff --exit-code
• npx markdownlint-cli2 docs/generate-benchmark.md

Open questions for DeVaris

• Corpus size/mix: 28 requests across 6 connectors × ~9 processor capabilities. Worth
  adding adversarial/prompt-injection-shaped fixtures now, or defer to the v0.20 PR
  that actually wires up retries and can exercise them live?
• capability.go's tag→processor map is hand-maintained (not structurally linked to
  builtin.DefaultBuiltinProcessors, to avoid pulling plugin-runtime deps into a
  YAML-only package) — acceptable for a design-only PR, but flagging so it doesn't
  silently drift if builtin processors are renamed before v0.20 lands.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GQFzakPShAYj8CcwajYDD