v0.39.0

[v0.39.0] - 2026-07-14

Highlights

• From resolved library to typed clients: pipelex resolve and pipelex codegen — pipelex resolve distills a bundle's whole closure into the normalized library crate — a flat, fully-qualified, fingerprinted snapshot — and pipelex codegen types projects it into TypeScript (ts-zod) or Python (python-pydantic, python-structures) artifacts. A trust chain of stamped headers and a codegen.lock makes drift detectable offline: pipelex codegen check verifies artifacts by pure hashing — no engine, no network, no API key.
• Validation now fixes what it finds — When an error has a deterministic safe fix, every surface says so: a 💡 suggested-fix line in the CLI output, a structured suggested_fix object in agent and API payloads. The new pipelex fix bundle / pipelex-agent fix bundle commands apply those fixes in place through a shared fix loop, with --diff to preview the changes without writing anything.
• Smart Inputs: the signature decides the shape — Inputs are now interpreted top-down against the pipe's declared input signature instead of bottom-up from their shape alone: a bare string becomes the declared concept (never silently degrading to generic Text), lists shape element-wise, bare URLs/paths resolve for Image/Document inputs, and generated input templates default to the light bare-value form.
• New native concepts: Date, Time, and YesNo — Built-in concepts for calendar dates (preserving source precision — no fabricated times of day), times of day, and boolean judgments, plus a time structure-field type completing the temporal triple. Every native now materializes from its pinned normative definition, so crate fingerprints byte-agree across implementations.

Added

• pipelex resolve and the pipelex codegen family: pipelex resolve assembles a bundle closure and emits the normalized library crate — a flat, fully-qualified, natives-expanded, self-contained, fingerprinted snapshot of a resolved library — as JSON or TOML. On top of the crate:
• pipelex codegen types --target ts-zod|python-pydantic|python-structures projects the crate's concept set into typed artifacts. The ts-zod target emits a pure types.ts (imports only zod, with wire-native snake_case keys so a schema validates a wire payload directly) plus a binder.ts exposing a typed parse/serialize pair per concept; python-pydantic emits self-contained BaseModels; python-structures emits runtime StructuredContent classes.
• pipelex codegen inputs --pipe <ref> renders a runnable inputs template for a pipe (defaulting to the closure's main_pipe).
• Generated files are never edited — each carries an AUTOGENERATED header pointing at the sibling extension-file mechanism.
• Codegen trust chain (stamps · codegen.lock · offline codegen check): Every file pipelex codegen types writes now carries a self-describing stamp header (source crate fingerprint, engine version, projection, options, and a content hash of the body below it), and the artifact set is recorded in a sibling codegen.lock. pipelex codegen check [DIR] verifies generated artifacts are current offline — pure hashing, no engine, network, or API key — reporting drift by category (missing, modified, hand-edited, orphan) with the verdict on the exit code (0 current · 1 drift · 2 no lock). Emission is idempotent (write-if-changed — unchanged output leaves files untouched) and prunes previously-generated files that drop out of the set, so a deleted concept never lingers as a stale generated file. Input templates (codegen inputs) are deliberately not stamped or locked — they are user-editable scaffolds, not tracked generated code.
• Agent-CLI codegen mirror (pipelex-agent codegen types|check): The codegen family is now available to machine consumers through the agent CLI's two-stream envelopes (--format markdown|json success stream, --error-format error stream). codegen types runs the same engine and write-if-changed emission as the bare CLI; codegen check presents the offline drift verdict structurally — drift is a CodegenDriftError envelope enumerating drifts[] by category (exit 1), a missing or unreadable codegen.lock is a no-verdict error (exit 2). With the pipelex runner, mthds-agent codegen … forwards to these commands.
• Host-facing resolve/codegen engine cores (what the HTTP routes ride): pipelex.pipeline.resolve_bundle.resolve_crate_from_contents resolves in-memory MTHDS contents into the normalized crate — same engine, same ValidateBundleError verdict vocabulary, and the same loaded-on-success library contract as validate_bundle — so a host (the pipelex-api /v1/resolve + /v1/codegen routes) serves resolution without touching the filesystem. pipelex.codegen.emission.build_stamped_projection is the new pure stamping core (stamped files + codegen.lock content, no disk writes) that write_stamped_projection now rides, so artifacts served over HTTP are byte-identical to locally written ones and pass the offline codegen check verbatim. The shared bundle-loading error cascade translate_to_validate_bundle_error is now public (renamed from its underscore-private form) since it gained a third entry point.
• Agent bundle fixer: Added pipelex-agent fix bundle, a deterministic in-place fixer over the shared validation fix loop. The command supports --format/--error-format, repeatable --select/--ignore fix-rule filters, --allow-signatures, --max-iterations, and the same bundle-file or directory resolution as pipelex-agent validate bundle. Successful fix results report pending_signatures and is_runnable; by default the command exits non-zero when a fixed bundle remains valid-but-not-runnable.
• Human bundle fixer: Added pipelex fix bundle, the human counterpart of the agent fixer over the same fix loop, with Rich-rendered output naming every change made (fix descriptions, files written, iteration count), the same --select/--ignore/--allow-signatures/--max-iterations options, and the same bundle-file or directory resolution as pipelex validate bundle. A new --diff flag previews the changes as a unified diff without writing anything — the real fix loop runs against a temp-copy sandbox, and exit codes keep the verdict semantics so --diff answers "would it converge?".
• Suggested fixes surfaced in pipelex validate: When a validation error has a deterministic safe fix, the human error output now shows a per-error 💡 Suggested fix: line and ends with an actionable footer naming the exact pipelex fix bundle command (echoing the invocation's -L dirs) in place of the generic tip.
• suggested_fix field in /validate payloads: Each validation error that has a deterministic safe fix now carries an additive structured suggested_fix object — fix_code (the kebab-case rule id), description (the author-facing 💡 text), safety (safe/unsafe), optional source file, and ops (the semantic patch operations) — on every validation_errors[] item, both in the pipelex-agent validate/fix JSON envelopes and in the API /validate response body. This is the machine-actionable counterpart of the 💡 line, letting a software consumer apply the fix directly from the structured payload.
• Smart Inputs (signature-driven input shaping): Inputs are now interpreted top-down against the pipe's declared input signature rather than bottom-up from their shape alone.
• Bare values (strings, numbers, booleans, dicts, lists) are automatically shaped into the declared concept (e.g., a bare string becomes a legal.Question, not a generic native.Text).
• Lists shape element-wise into the declared item concept; single values auto-wrap into lists where required, and empty lists are now legal.
• Bare URLs/paths for Image or Document inputs are resolved relative to the inputs file's directory.
• Declared structured lists (e.g., Person[]) accept a direct path to a .csv file by signature, without requiring an envelope.
• Comprehensive input error handling: A suite of typed errors for input shaping failures (WrongScalarKindError, ListWhereSingularError, MultiplicityCountMismatchError, StructureValidationError, UnknownInputNameError) that provide clear hints and render the expected JSON shape.
• CLI --explicit flag: Added --explicit to pipelex build inputs and pipelex-agent inputs to generate the legacy {"concept", "content"} envelope templates.
• Native YesNo concept: Built-in concept (backed by YesNoContent) for boolean judgments. Requires a strict boolean, renders as yes/no in downstream prompts, and can be read via pipe_output.main_stuff_as_yes_no.yes_no.
• Native Date concept: Built-in concept (backed by DateContent) for calendar dates. Requires a datetime.date and accepts an optional datetime.time. Preserves source precision (never forcing the LLM to fabricate a time of day) and retains UTC offsets verbatim.
• Native Time concept: Built-in concept (backed by TimeContent) for a time of day, optionally with a UTC offset — never attached to an invented date. A top-level TOML time-of-day literal in an inputs file now maps to Time (Breaking: Time joins the reserved native concept codes, and the former bare-time rejection — InputsTimeOnlyNotSupportedError — is gone).
• Structure-field type = "time": The concept structure language gains a time field type (a time of day, optionally with a UTC offset), completing the temporal triple beside date and datetime. Generates a datetime.time field; emitted by every codegen target.
• Pinned normative native definitions: Native concepts now materialize into the normalized crate by lookup into the standard's pinned per-version definitions (pipelex/core/concepts/native/pinned_blueprints.py, mirroring the mthds spec's new native-concepts.md), never by reflection over the runtime content classes — so crate fingerprints over materialized natives byte-agree across implementations. The old reflection is retained solely as a consistency test proving each runtime content class still matches its pinned blueprint.
• Subject-grants registry — the keyword-only convention hardened: A positional first parameter ("subject") in pipelex/ source is now legal only under an explicit grant recorded in subject_grants.toml at the repo root, carrying the subject's param name and an honest, def-specific review rationale — every grant in the registry has been individually reviewed against the grant rubric. pipelex-dev subject-grant is the sole registry writer; the check-keyword-only AST guard enforces the registry symmetrically (an ungranted subject fails, and so does a stale grant whose def was renamed, moved, demoted, or newly carved out) in make check, CI, and the edit-time hook. Subjects typed bool/int/float are banned outright — no grant can cover them. The registry schema is strict: exactly param + rationale, unknown keys fail the check. See docs/contribute/keyword-only-arguments.md.
• Drift contracts (pipelex-dev drift plan|check|ack): Deterministic review obligations between code and docs. A root drift.toml declares contracts ("when these trigger files change, these review targets must be re-examined and the examination recorded"); committed ack files under .drift/acks/ record each fulfilled review with a digest, reviewer, and rationale. A contract is fulfilled iff its stored ack digest equals the digest recomputed from the current tree (staged blob OIDs from the git index — no base ref, no timestamps), so trigger edits, matched-file adds/deletes, and contract-definition edits all reopen it.
• make drift-check gates the make check aggregate and runs as an advisory CI job; make drift-plan prints per-contract Markdown packets with exact per-file changes since the last ack.
• make drift-ack CONTRACT=… RATIONALE="…" runs the contract's verify commands, then records the review and stages the ack file it writes, so the local gate checks the same index the commit is built from. If staging that ack fails, the ack file is removed rather than left unstaged, so a later check reports a missing ack instead of a false green.
• For a contract with verify commands, a trigger file left unstaged-modified or untracked is a hard ack error (the verify run would certify content the digest does not cover) — checked both before and after the verify commands, so a trigger the commands themselves format or generate is caught too. For other contracts it stays a warning.
• Seeded with contracts for the config docs, the CLI docs, and the keyword-only convention spec; workflow documented in docs/contribute/drift-contracts.md.

Changed

• Keyword-only demotions across the public API (Breaking): The subject-grant review demoted every positional first parameter that failed the grant rubric to keyword-only, and external code calling these surfaces positionally (or subclassing them with positional signatures) breaks. Notable demoted families: the PipeAbstract run family (run_pipe, dry_run_pipe, validate_before_run, … — job_metadata/visited_pipes now keyword-only), StuffContent.rendered_for_prompt/rendered_for_template_async (text_format) and the TextFormatRenderable/ImageRenderable protocols with all their implementations, the LibraryManagerAbstract load family (library_id), ensure_pipelex_booted (config_overrides), pipeline_run_setup (execution_config), ConceptLibrary.is_compatible (tested_concept), GraphTracerProtocol.setup (graph_id), the PromptImageFactory/PromptDocumentFactory content factories (uri), the model-manager/backend-library setup+load (secrets_provider), the reporting protocol's set_event_log/clear_event_log (context_key), and assorted tools helpers (pluralize/count_with_noun, LogLevel.from_int, set_dry_run_forced, …).
• pipelex build structures is now an alias of codegen types --target python-structures (Breaking): The legacy always-qualified per-file generator is deleted. The alias resolves the target's whole directory closure and emits one stamped structures.py module plus a codegen.lock (bare-when-unique class names, declared imprecision instead of the old silent TextContent guess for structureless concepts). The --force flag is gone (emission is write-if-changed), a .mthds file target now resolves its parent directory's closure, and concepts-only loading of invalid-pipe bundles is no longer supported (the crate requires a valid library). The concepts-only loading API (load_concepts_only, load_concepts_only_from_directory, and the library-manager methods behind them) is removed with it.
• pipelex build runner emits its structures through the codegen engine (Breaking): The scaffolded structures/ directory now contains the stamped single-module projection (structures.py + codegen.lock) instead of per-concept files, and the generated runner script imports and instantiates the emitted class spellings (from structures.structures import Invoice) — fixing dropped imports for nested custom concepts along the way. A user class backing a structure = "<ClassName>" concept is imported from its real module.
• Human validation error rendering routed through the shared structured items: pipelex validate's error details now render from the same ValidationErrorItems the agent CLI and API emit, which surfaces information the old renderer silently dropped: pipe-factory errors (e.g. an unknown concept) get their own section, a parse-level failure (e.g. a TOML syntax error) now shows its message instead of no detail at all, and pipe validation errors name their source file. The dry-run message remains visible alongside categorized errors.
• Validation error messages now speak MTHDS author syntax: Pipe validation errors state the problem and the fix the way an author writes them — a sequence output-multiplicity mismatch now reads "declares its output as 'StoryIdea', but its last step yields 'StoryIdea[]'. Update the sequence's output to 'StoryIdea[]'", and an input-type mismatch reads "input 'dish' is declared as 'Number' but its step needs 'Text'. Update the input to 'Text'" — instead of leaking Python internals (multiplicity=None, Concept(...)/PresenceMarker reprs). The misleading (required: […], provided: …) suffix is suppressed for multiplicity errors (where both sides are identical by definition) and rendered as joined author-syntax refs — never Python list-repr brackets — elsewhere.
• Agent validate / fix output renders prose, not a JSON dump: pipelex-agent validate bundle and pipelex-agent fix bundle markdown output now renders validation errors as category-grouped prose (a humanized title, identity fields, the message, and a 💡 Suggested fix: line) followed by a fix-aware footer naming the exact pipelex-agent fix bundle command when a safe fix exists — mirroring the human panel. The machine-facing JSON stream (--format json) is unchanged, so software consumers keep branching on the same structured fields.
• Clean validation summary message on every surface: The top-level error message (agent JSON, the API /validate body) is now a clean, author-facing summary derived from the structured errors, instead of the raw pydantic dump (Validation error(s): … Value errors: '<field>': Value error, …). The API's opt-in invalid-verdict rendered_markdown now carries the same per-error 💡 Suggested fix prose as the CLI. A still-invalid fix that stops with no progress now names the re-proposed fix in author terms rather than dumping an internal fingerprint string.
• Input templates default to "light" shape (Breaking): pipelex build inputs and pipelex-agent inputs now generate light templates by default, outputting the bare values expected by the signature rather than the verbose {"concept", "content"} envelopes. TOML templates include the declared concept as a # concept: ... comment.
• Structure-field type = "date" behavior (Breaking): A date field now generates a datetime.date (JSON schema format: date), preventing LLMs from hallucinating times for calendar dates. A new type = "datetime" field handles full timestamps (format: date-time), and default-value validation now rejects a datetime default on a date field.
• Strict input validation (Breaking): Undeclared input names are now a hard error (previously silently ignored), and top-level null values are rejected — absence must be expressed by omitting the key. Known limitation: this also rejects the extra stuffs carried by a pipelex-agent run --with-memory envelope, so piping one run's full working memory into a downstream pipe that declares only a subset of it now fails with UnknownInputNameError. Until envelope-derived inputs get their own leniency, narrow the envelope to the declared inputs before piping, or pass the inputs explicitly with --inputs.
• Explicit envelope validation (Breaking): The {"concept", "content"} escape hatch is now compatibility-checked against the pipe's declared signature; incompatible concepts raise a typed error.
• TOML date/time handling (Breaking): TOML date and datetime literals are natively supported and map directly to the Date concept; a bare time-of-day literal maps to the new Time concept (it never silently becomes a Date — shaping a Time into a Date slot fails the compatibility check).
• ImageContent flattens its pixel dimensions (Breaking): size: ImageSize | None is replaced by paired optional width / height integer fields (both-or-neither, enforced by a validator) — a named nested shape in the crate is a concept, and the size grouping did not earn that citizenship. ImageSize remains as the internal img-gen parameter type. Image wire payloads change shape accordingly.
• native.Date declares real structure (Breaking): Materialized Date is no longer structureless — its pinned definition carries a required date field plus an optional time field (the new time field type), so generated clients get real Date types. DateContent's wire form is unchanged.

Fixed

• Library loaders now bind the target library for the whole load: load_from_blueprints / load_from_crate / load_libraries took an explicit library_id but resolved concepts and the class registry through the ambient current-library ContextVar one hop below — so loading into a library that wasn't current read (and registered generated structure classes into) whatever library the ambient pointer happened to name. Production hosts always set the target current before loading, so this only bit multi-library callers — exactly the isolation the loaders exist to provide. The loaders (and blueprint removal) now wrap their body in scoped_current_library(library_id), restoring the caller's binding on exit, so everything beneath a load resolves against the load's target by construction. A regression test loads into a deliberately non-current library.
• PipeExtract broke crate normalization (500s on every static-core route): PipeExtractBlueprint.validate_output only accepted the authoring spelling Page[], but normalization qualifies concept refs (Page[] → native.Page[]) and then rebuilds the blueprints — so the validator rejected its own normalized output, and any bundle containing a PipeExtract raised a raw pydantic ValidationError out of resolve_crate_from_contents, 500-ing /v1/build/inputs, /v1/build/output, /v1/resolve, and /v1/codegen. The validator now compares the parsed concept ref and multiplicity instead of the raw string, so it holds for both spellings. The output contract is unchanged and still exact — a variable-length list of native Page — since a refined output would be an unverifiable promise: PipeExtract mechanically produces native pages and cannot make them more specific.
• resolve / codegen rejected bundles that pin a model: The resolve and codegen commands booted without model specs, so library validation could not check pipe model pins (model = "gpt-4o-mini") against the deck and rejected any bundle carrying one. They now boot like validate — offline, but with model specs loaded.
• Native Image no longer materializes structureless: Every native now materializes from its pinned normative definition, so generated clients for image-producing pipes expose url, public_url, caption, width, height, etc. (The brief interim dict-with-imprecision mapping for the nested size model is superseded by the ImageContent flatten above.)
• Deterministic input-template placeholders: Mock URL placeholders in generated inputs templates (codegen inputs / build inputs) are now deterministic (https://mock.invalid/<field>), so committed templates no longer churn on every regeneration.
• codegen inputs is write-if-changed: Like codegen types, an already-current inputs template is left untouched (no mtime churn) and the console reports Unchanged instead of claiming Generated.
• Opaque generated classes no longer strip content: A structureless or Python-class-backed concept's generated Python class now carries model_config = ConfigDict(extra="allow"), so model_validate keeps the payload verbatim instead of pydantic's default extra="ignore" silently dropping every field. Opaque is pass-through, never lossy — matching the ts-zod z.unknown() behavior.
• Human CLI ~ expansion: pipelex validate bundle (and the new pipelex fix bundle) now expand ~ in the bundle path and -L/--library-dir values, matching the agent CLI.
• Validate telemetry label: The CLI_COMMAND telemetry tag for pipelex validate subcommands no longer double-suffixes (previously "validate bundle bundle").
• Silent mistyping of inputs: Providing a bare string for a refined text concept (e.g., legal.Question) no longer silently degrades the type to a generic native.Text; the declared concept type is now retained.
• TOML inline table formatting: Light templates now use inline tables for structured values, preventing trailing scalars from being swallowed into [sections].

Documentation

• Smart Inputs guides: Updated provide-inputs.md, executing-pipelines.md, and native-concepts.md for the Smart Inputs paradigm, leading with the bare-value approach and demoting the envelope format to an escape hatch.

What's Changed

• Update README and deferred-step-f-notes with release status and follo… by @lchoquel in #1034
• feat(core): Native YesNo and Date concepts + Smart Inputs by @lchoquel in #1028
• Add graph provenance mode metadata by @lchoquel in #1038
• Autofix by @lchoquel in #1027
• feat(codegen): normalized library crate + deterministic projections (types, inputs, trust chain, resolve/codegen CLI + agent CLI) by @lchoquel in #1039
• Subject grants: every positional subject is an explicit, recorded, reviewed exception by @lchoquel in #1040
• drift ack hardening: auto-stage the ack file; hard-fail dirty triggers on verify contracts by @lchoquel in #1041
• docs: fix code-doc drift across under-the-hood + advanced pages (Drift Hunt S2-3) by @lchoquel in #1042
• docs: fix drift on building-methods pipes/ pages (Drift Hunt S2-1) by @lchoquel in #1043
• Drift Hunt S2-2: fix building-methods concepts + top-level docs drift by @lchoquel in #1044
• Drift Hunt S2-4: features + the tail — Stage-1 drift fixes + cross-cutting sweeps by @lchoquel in #1045
• Drift Hunt: campaign close-out — records, handoff, and tracker by @lchoquel in #1046
• Stabilization fixes and changelog polish by @lchoquel in #1047
• Release v0.39.0 by @lchoquel in #1048

Full Changelog: v0.38.0...v0.39.0