feat(embed): libconduit B1 Go embedding API (New/Engine/Handle/Import)

[devarismeroxa]

Summary

Implements B1 of the libconduit / embedded v1 workstream (v0.19 Workstream 2, design doc #2651, plan workstreams/embed-libconduit.md): a frozen, semver-committed root package github.com/conduitio/conduit giving a Go application an embeddable Conduit engine — conduit.New(ctx, Options) -> Engine.Run(ctx) -> Handle.Stop(ctx) — with host-injected *slog.Logger and prometheus.Registerer, so nothing writes to a global logger, the default Prometheus registry, or os.Stdout, and nothing calls os.Exit.

B2 (pipelines-in-code builder) is explicitly out of scope — this PR ships Engine.Import(ctx, PipelineConfig) against hand-built PipelineConfig literals only; the NewPipeline(id).With...().Build() builder is a later PR.

What shipped

pkg/conduit seams (additive, inert on the CLI path — AC-5):

• RuntimeOption + WithLogger, WithMetricsRegisterer, WithDialContext on NewRuntime(cfg, ...opts).
• OpenStore now takes a context.Context (bounds the Postgres/SQLite dial instead of context.Background()); its one other caller (conduit doctor's store-reachable check) updated.
• Per-Runtime configureEmbeddedMetrics (fresh registry + grpc stats handler per embed Runtime, registered only into the host's registerer) alongside the unchanged CLI-path configureMetrics() (package sync.Once + promclient.MustRegister).
• TestNewRuntime_CLIDefaultsUnchanged proves the seams are inert with zero options passed.

Root package (conduit.go, doc.go, log_adapter.go):

• Options (Logger, MetricsRegisterer, DB, PipelinesDir, API), Engine, Handle, New.
• Engine.Run/Handle.Stop per AC-4's exact two-resolution-path shape (Ready vs. a failed Run, never blocking).
• Handle.Stop reuses Runtime.Run's existing ctx-cancellation-driven tomb drain unchanged — Invariant 7, no new drain mechanics.
• PipelineConfig — a type alias (not a copy) of provisioning/config.Pipeline; doc comment added to parser.go extending the deprecation policy by name (grep-checked: constrained by.*PipelineConfig).
• Engine.Import enriches + validates the config (provisioning/config.Enrich/Validate) before delegating to provisioning.Service.Import — see self-review finding below.
• Engine.StartPipeline/StopPipeline thin delegates to Orchestrator.Pipelines.
• Internal zerolog→slog adapter (log_adapter.go) so Conduit's existing zerolog-based logging routes through a host *slog.Logger without teaching pkg/foundation/log about slog.

Trimmed per review must-fix: no new conduiterr codes. Run-called-twice, Stop-on-nil-Handle, and Stop-deadline-exceeded all return the existing conduiterr.CodeInvalidArgument. AC-8's "under 15 minutes" is split: TestExampleHelloPipeline_WithinBudget asserts a 60s CI wall-clock ceiling on the example's own execution; the human-copy-paste-timing claim is left as a doc/UX statement, not a test.

Tests: TestNew_ConstructsEngine, TestNew_Nil{Logger,Registerer}_*, TestRun_FailsBeforeReady, TestRun_AlreadyCanceledContext (Engine- and Runtime-level), TestRun_CalledTwice_ReturnsCodedError, TestStop_{NilHandle,ConcurrentIdempotent,DeadlineExceeded}, TestNew_StoreAlreadyOpen_ReturnsCodedError, TestNew_MetricNameCollision_ReturnsCodedError, TestPipelineConfig_IsAliasOfProvisioningConfig, TestImport_RoundTripsThroughRunningEngine, TestTwoEngines_LoggerIsolated (must pass), TestTwoEngines_MetricsCrossTalk_KnownLimitation (documents, doesn't fix), TestObservabilityIsolation_Subprocess (subprocess re-exec proving zero bytes to real stdout/stderr — AC-2c), Example_helloPipeline + its budget test.

⚠️ Tier-1 commit requiring DeVaris sign-off

Commit d5922de — fix(conduit): guard Runtime.Run against an already-canceled context is isolated from the Tier-2 embed work per the design doc's escalation rule (anything touching Runtime.Run's tomb/cleanup mechanics is Tier 1, regardless of diff size).

This is not just Tier-1-caution — it's a real bug, found empirically while writing TestRun_AlreadyCanceledContext: without the one-line ctx.Err() guard at Run's entry, calling Runtime.Run with an already-canceled context panics (tomb.Go called after all goroutines terminated) instead of returning an error. tomb.WithContext's own cancellation-watcher goroutine kills the tomb before initServices's later t.Go calls (e.g. serveGRPC) run, so those calls hit an already-dead tomb.

conduit run never triggers this (a live process's context is only canceled by a signal arriving after Run is already executing), so the CLI path is unaffected — confirmed by removing the guard locally and re-running the full pkg/conduit suite plus the new CLI-parity test, all green except the two tests this fix targets. The guard is a pure early return before the tomb or registerCleanup is constructed; it does not touch drain/cleanup logic itself.

Requesting explicit DeVaris review of commit d5922de before merge, per the standing Tier-1 gate.

Failure-mode analysis (Tier-1 commit)

• What could this break: nothing on the CLI path — the guard is unreachable there (conduit run never calls Run with a pre-canceled context). On the embed path, it changes a panic into a clean error return, strictly safer.
• Which metric/alert would show a regression: N/A — no metric surfaces a panic-vs-error distinction today; a regression would show as a crashed process / non-zero-without-log exit in whatever harness runs conduit run (unaffected) or as an embedder's process crashing on Engine.Run (this fix's whole point).
• Rollback: revert commit d5922de alone; it has no dependents (the Tier-2 commit that touches Engine.Run was written and tested with this guard in place, so reverting would reintroduce the empirically-confirmed panic for the embed path specifically, not affect the CLI).

Adversarial self-review

• Error paths: Engine.Import originally delegated straight to provisioning.Service.Import per the design doc's citation — but a hand-built PipelineConfig{} has a nil DLQ.WindowSize/WindowNackThreshold (zero-value *int), and provisioning.Service's createPipelineAction.Do dereferences those pointers directly, assuming the file-based path's config.Enrich already ran. Caught this via TestImport_RoundTripsThroughRunningEngine, which panicked (nil-pointer deref) before the fix. Import now calls config.Enrich+config.Validate first, mirroring provisionPipeline's own convention exactly.
• Concurrency: Handle.stopOnce + buffered done/select ensures Stop is safe under concurrent callers (TestStop_ConcurrentIdempotent, race-clean). Engine.started uses atomic.Bool.CompareAndSwap so two goroutines racing Run() can't both proceed.
• Resource cleanup: Engine.Run's failure branch (Ready never closes) calls cancel() before returning to release the derived runCtx, even though the tomb has already exited by construction.
• Known, accepted limitation (not fixed here, per design doc's Non-goals): a failed configureEmbeddedMetrics call (e.g., the metric-name-collision case) still leaks a foundation/metrics/prometheus.Registry into pkg/foundation/metrics's process-global bookkeeping, because the collision can only be detected after metrics.Register(reg) has already populated reg's descriptors — there's no Unregister. Harmless (never scraped, since the host's own Register call is what failed) but real; flagging so it doesn't reappear as a surprise later.
• Invariant 7: verified Handle.Stop calls only cancel() (no reimplementation) and reads Runtime.Run's real return value; registerCleanup/V1/V2 are untouched by either commit.
• /metrics HTTP endpoint gap (documented, not fixed): if Options.API.Enabled is true, the HTTP /metrics route still serves promhttp.Handler() (the default gatherer), not the host's injected registerer — out of scope for this PR's ACs (none of AC-1..8 test the HTTP metrics route), noting for a follow-up.

Open questions for DeVaris

1. Confirm the Tier-1 commit's review is what you want before merge (per the standing gate) — I have not merged this PR.
2. Confirm the /metrics HTTP-route gap above is an acceptable known limitation for v1, or should be tracked as a follow-up issue.

Gate results

• make generate && git diff --exit-code: clean, no drift.
• golangci-lint run ./... --new-from-rev=main: 0 issues.
• go build ./...: clean.
• go test -race ./...: full repo green (including this PR's new packages).

Roadmap: v0.19 Workstream 2 (libconduit embed v1, B1). Risk tier: Tier 2 overall, Tier 1 for commit d5922de specifically.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GQFzakPShAYj8CcwajYDD

[devarismeroxa]

Review must-fixes applied (Tier 1 — still needs DeVaris sign-off before merge)

Addressed the fresh-context review findings on the embed lifecycle/DB-leak issue.

1. Added Engine.Close(ctx) error — the resource-release path for the database New opens
eagerly. Covers all three states the review called out:

• New → Close (Run never called)
• New → Run (hits the already-done-context guard) → Close
• New → Run → Stop → Close (safe no-op; the drain already closed it)

Idempotency is enforced at the Runtime level: added Runtime.CloseDB() (a sync.Once around
r.DB.Close()), and routed both registerCleanupV1/V2's cleanup goroutines and
Engine.Close through it, so neither the normal drain path nor the new release path needs to
know whether the other already ran — no double-close race either way.

TestNew_StoreAlreadyOpen_ReturnsCodedError now defers e1.Close(ctx) instead of discarding
e1. Added TestClose_NoRun_ReleasesDB, TestClose_AfterRunHitsAlreadyDoneContextGuard_ReleasesDB
(both proven against a real Badger file lock — a second New at the same path only succeeds if
the first Engine's DB was actually released), TestClose_DoubleCloseIsSafe, and
TestClose_AfterSuccessfulStop_IsSafe.

2. The already-done-context guard now returns a coded error — conduiterr.CodeInvalidArgument
wrapping the original ctx.Err(), instead of a bare context.Canceled/DeadlineExceeded. Reuses
the existing code rather than minting a new one, consistent with how this package already codes
the double-Run and Stop-timeout cases.

3. Documented the lifecycle contract on Engine's doc (and referenced from New, doc.go):
started flips on the Run call, not on success, so a failed Run retires the Engine for
running — and Close is the resource-release path regardless of whether Run was ever called
or succeeded.

4. Filed and cross-referenced two tracked follow-ups (accepted as v1 limitations, not
blocking this PR):

• embed: configureEmbeddedMetrics leaks a registry into pkg/foundation/metrics global bookkeeping on failed Register #2669 — configureEmbeddedMetrics leaks a registry into pkg/foundation/metrics's
  process-global bookkeeping on a failed registerer.Register.
• embed: /metrics HTTP route serves the default gatherer, not the host registerer #2670 — the /metrics HTTP route always serves promclient.DefaultGatherer, not a
  host-supplied MetricsRegisterer.

5. Fixed the doc.go overstatement that Connector/Processor/DLQ are annotated "the same
way" as Pipeline — it's one shared comment block above Pipeline in parser.go covering all
four by name, not per-type docs. Clarified.

Gates run locally

• make generate && git diff --exit-code — clean, no generated-file drift.
• golangci-lint run ./ ./pkg/conduit/... — 0 issues.
• go build ./... — clean.
• go test -race -count=1 ./ ./pkg/conduit/... — all green, including the new TestClose_*
  tests and the existing Run/Stop suite.
• gofmt -l . — only a pre-existing unrelated golden-testdata file, not touched by this diff.

Self-review note

The alternative I considered — an Engine-level sync.Once around Close alone — would not
have prevented a double r.DB.Close() call after a successful Stop (the tomb-driven cleanup
path and Close would each think they were the only closer). Pushing the sync.Once down into
Runtime.CloseDB() and having both paths call it is what actually makes "safe after a
successful Stop" true, not just "safe if you only ever call Close once."

Per CLAUDE.md: this is Tier 1 (embed lifecycle + DB resource release on a frozen public API). Not
merging — needs DeVaris review with fresh context in a separate session.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com