fix(embed): lazy DB open + skip file provisioning on empty PipelinesDir (B1 DX hardening)

[devarismeroxa]

Summary

DX audit fast-follow on the merged libconduit B1 embed API (#2667). Adoption is zero, so now is the right time to fix two frozen-Options-contract issues rather than carry them forward.

Fix 1 — silent provisioning failure on empty Options.PipelinesDir. Leaving PipelinesDir empty fell through to pkg/conduit's CLI-oriented cwd/pipelines default. That directory usually doesn't exist for a pure-embed caller, so provisioning failed as a swallowed ERROR log while Run still returned a healthy Handle — contradicting New's doc promise that every failure is returned as an error.

• Added pkgconduit.Config.Pipelines.Disabled (no long/mapstructure tag — not a CLI flag, zero value false, conduit run never sets it). The embed layer's New sets it whenever Options.PipelinesDir == "", which short-circuits initServices's call to ProvisionService.Init entirely instead of falling back to a scan.
• Added pkgconduit.Runtime.ProvisioningErr — purely additive, set regardless of ExitOnDegraded, never read by the CLI. Engine.Run reads it right after Ready fires; if set, it cancels + drains (Invariant 7) and returns the error instead of a Handle. A genuine failure with a configured PipelinesDir (missing dir, malformed YAML) now surfaces as a returned error, matching conduit run's own log-and-continue default being intentionally NOT reused on the embed path.
• CLI behavior is untouched: Disabled is always false on that path, and ProvisioningErr is never consulted there.

Fix 2 — eager DB open / mandatory Close footgun. New opened the database eagerly, so a constructed-but-never-Run Engine leaked a Badger file lock or Postgres pool, and Close was mandatory even without Run.

• New now only validates Options (including calling cfg.Validate() directly, so a bad config still fails fast). Constructing pkg/conduit's Runtime — which is where the DB actually opens — is deferred to ensureRuntime, called lazily on the first Run or Import (also wired into StartPipeline/StopPipeline so a runtime-never-built Engine can't nil-pointer-panic on those either).
• Close reads whether anything was actually opened and is a safe no-op otherwise; when something was opened it still delegates to Runtime.CloseDB (feat(embed): libconduit B1 Go embedding API (New/Engine/Handle/Import) #2667's idempotent double-release path, unchanged).
• Added a runtime.SetFinalizer safety net on Engine: if an Engine that DID open resources is GC'd without Close, it logs a leak warning through the injected logger (else slog.Default()). Verified working end-to-end with a standalone smoke test (construct → Run → Stop → drop the reference → GC → warning observed in the log). Close clears the finalizer.
• TestNew_StoreAlreadyOpen_ReturnsCodedError is preserved but the contention now has to be forced via Run (both engines), since New alone opens nothing — same for the metrics-collision and metrics-cross-talk tests, which also moved their trigger from New to Run.

Store-already-open timing change

Documented explicitly in Engine's "Lifecycle contract" doc and in the updated test: the coded CodeUnavailable collision now surfaces "when the DB actually opens (on Run/Import)," not at New.

Failure-mode analysis (Tier-1-adjacent: frozen Options semantics + DB lifecycle + provisioning behavior)

• What could this break: any caller relying on New eagerly failing on a bad PipelinesDir/DB config would now see that surface at Run/Import instead — mitigated by keeping cfg.Validate() synchronous in New (a nonexistent custom PipelinesDir still fails fast at New, proven by TestNew_NonexistentCustomPipelinesDir_FailsFast). A caller relying on metrics being registered immediately after New (before Run) would see a behavior change — there is no such use case in the codebase (adoption is zero) and it's the described intent of this fix.
• Which metric/alert would show it: none live yet for this package (it's a library, not a running service); the example (Example_helloPipeline) and its CI wall-clock budget test are the closest thing to a canary and both still pass.
• How to roll back: revert this commit; feat(embed): libconduit B1 Go embedding API (New/Engine/Handle/Import) #2667's eager-open behavior returns immediately, no state/format migration involved (nothing here touches serialized data).
• Concurrency: ensureRuntime/Close share one sync.Mutex (not sync.Once+atomics) specifically so a Close racing the very first Run/Import blocks until that open attempt resolves, rather than observing "nothing opened yet" and returning early while an open completes moments later with nothing left to release it — documented in Close's own doc as the one remaining caller-orchestrated race (calling Close concurrently with the first Run/Import, as opposed to a later one, which just reuses the cached result).
• Uncertainty surfaced, not assumed away: the leak-check finalizer is explicitly documented as best-effort — GC timing is unpredictable and finalizers aren't guaranteed to run before process exit. It's a safety net, not a substitute for Close.

Adversarial self-review

• Re-checked that Config.Pipelines.Disabled has no CLI flag tag and defaults false, so conduit run is provably unaffected (initServices' new branch is unreachable from the CLI).
• Re-checked Runtime.ProvisioningErr doesn't change what's logged or when conduit run itself fails — it's an added field next to unchanged control flow, not a rewrite of it.
• Traced the opened/closed mutex-guarded reads in Close and the finalizer to confirm no data race (ran the full suite under -race).
• Found and fixed a test-writing mistake myself: my first draft of TestRun_BadPipelinesDir_ReturnsError assumed a nonexistent PipelinesDir would fail at Run — it actually fails at New (cfg.Validate()'s existing non-default-path stat check), which is a better outcome than the bug report implied for that specific sub-case. Split it into two tests: TestNew_NonexistentCustomPipelinesDir_FailsFast (New-time) and TestRun_MalformedPipelinesDir_ReturnsError (an existing directory with unparseable YAML — the case that genuinely can only be discovered at Run/Init time).
• StartPipeline/StopPipeline now also call ensureRuntime — not strictly requested, but necessary to avoid a new nil-pointer-panic footgun these lazy semantics would otherwise introduce for a caller that invokes them before any Run/Import.

B2 conflict — sequencing note for DeVaris

This will conflict with the in-flight B2 builder PR #2673 (feat/embed-b2-pipeline-builder, currently open) on conduit.go — confirmed via git diff main origin/feat/embed-b2-pipeline-builder -- conduit.go, which touches the same file (adds ImportPipeline near Import). Recommend merging whichever lands first, then rebasing the other before merge — I have not attempted to resolve that conflict here.

Test plan

• make generate && git diff --exit-code — clean, no drift
• golangci-lint run ./... --new-from-rev=851c406 — 0 issues
• go build ./...
• go test -race ./ ./pkg/conduit/... ./pkg/provisioning/... — all green, including feat(embed): libconduit B1 Go embedding API (New/Engine/Handle/Import) #2667's Close tests (updated for lazy semantics) and new regression tests for both fixes
• Manual smoke test of the GC finalizer (outside the repo, in a throwaway module) confirming the leak warning actually fires

DO NOT MERGE — flagging for DeVaris awareness per frozen-contract-semantic changes and the B2 sequencing conflict above.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_015GQFzakPShAYj8CcwajYDD

[devarismeroxa]

Applied the review-requested fix, rebased onto current `main` (no conflicts — disjoint from #2680/#2674).

1. Use-after-close guard in ensureRuntime: it now checks the existing closed flag (under the same mutex) before opening anything, and returns conduiterr.CodeInvalidArgument — the same code already used for double-Run and nil-Handle.Stop — instead of minting a new one. New → Close → Run/Import (no prior open) now fails fast rather than silently opening a fresh runtime/database.

2. Regression test: TestRun_AfterClose_WithNoPriorOpen_ReturnsCodedError covers New → Close → Run and → Import, both expected to return the coded error, plus a no-leak proof (a second Engine at the identical BadgerDB path opens without contention — it would hit CodeUnavailable if the guarded call had actually opened e1's store). Verified by hand: fails without the fix, passes with it.

3. Known-behavior doc (no code change): ensureRuntime's godoc now states explicitly that only the first caller's ctx bounds the lazy DB dial, so a concurrent Import with a short-lived ctx can cause a concurrent Run's dial to observe that cancellation — accepted for v1, not fixed here.

Gates run locally: make generate && git diff --exit-code (clean), golangci-lint run --new-from-rev=origin/main ./... (no new issues), go build ./..., go test -race -count=1 ./ ./pkg/conduit/... (239 passed, includes the existing embed/Close/lazy-open suite + the new test), TestAllCodesComplete (passed — no new code registered, reused CodeInvalidArgument).

Not merging — Tier-1-adjacent (data-path lifecycle), needs human sign-off per CLAUDE.md.