v0.38.0

[v0.38.0] - 2026-07-06

Highlights

• Optionality is a first-class language feature — Concept references in pipe inputs/outputs can be marked ? (optional) or ! (force), so absent values become tracked, first-class data with provenance instead of crashes or silent nulls. Absence flows through a runtime trichotomy — a plain input skips its pipe, a ? input absorbs it and runs, a ! input fails loudly — and a static taint pass proves at author time that every maybe-absent value reaches an explicit sink.
• Storage and secrets are now plugin seams — Both backends are selected at boot from a config-keyed registry, so third parties can ship pipelex-storage-<backend> / pipelex-secrets-<backend> packages via the pipelex.plugins entry point — the same discovery machinery the inference and orchestrator seams already use.

Added

• Optionals & presence markers (?, !): Concept references in pipe inputs/outputs can now be marked ? (optional) or ! (force), e.g. clause = "PenaltyClause?". Absent values are first-class data, tracked in working memory via an AbsenceRecord ledger (variable, producing pipe, kind = declared-absent / skipped / not-provided, reason, upstream chain) with provenance captured at the moment absence is produced:
• A plain input fed an absence causes the pipe to be skipped (implicitly lifted); its own output is recorded absent, chaining back to the origin so the miss short-circuits like Swift's a?.b.c.
• A ? input absorbs the absence and the pipe runs, handling both arms.
• A ! input fed an absence fails loudly with a typed OptionalValueAbsentError naming the variable, the consuming pipe, the producing pipe, and the original reason.
• Callers may omit ?-marked method inputs; the slot starts as a recorded not_provided absence rather than raising a missing-inputs error, and the missing-required-inputs message now names the optional inputs a caller may omit.
• Post-run reads get a tri-state resolved accessor (WorkingMemory.resolve_stuff / resolve_main_stuff: a value or a recorded absence). A slot with neither a value nor a record is still a hard error — never-produced is a bug, not an absence. Markers round-trip through blueprints, builder specs, IO contracts, and bundle representations; grammar misuse (X[]?, ! on an output, markers on concept definitions) is rejected at parse with optional_marker_invalid.
• Static optionality validation (the absence-taint pass): Validation proves the absence-safety theorem at author time — a taint pass over each controller's dataflow computes per-slot presence (guaranteed / maybe-absent) and rejects any maybe-absent slot that escapes without an explicit sink, with typed errors (optional_not_handled, optional_output_required, optional_branch_required_field) that each name the absence origin, the propagation path, and the fixes. The validation report also lists every liftable (skippable) pipe (liftable_pipes) — build-time visibility for "may be skipped when X is absent" — and gains a general warnings array for advisory lints that never flip is_valid (first occupant: optional_force_redundant, a ! whose slot is guaranteed present in every flow).
• Template guard-lint (optional_input_unguarded): Every template reference to a declared-optional (?) input must be guarded — @?var, a {% if var %}…{% endif %} block, or an inline presence conditional — or validation fails with the precise fix. Applies to PipeLLM prompts and system prompts, PipeCompose templates, and PipeCondition expressions. In the same motion, @? finally means what it says: an optional variable is no longer presence-required by the controller miss-gates, so the pipe runs and its guarded templates take the absent arm.
• Execution graph updates: GraphSpec nodes gain the skipped status with a skip_reason — "why did my workflow produce nothing?" is now answerable from the graph — and data edges fed by a declared-optional (?) output carry optional: true. Both flow through the in-process tracer and the distributed event-replay assembler (new pipe_end_skipped trace event).
• Storage provider plugin seam: The storage backend (local / in_memory / s3 / gcp) is selected at boot from a StorageProviderRegistry keyed by the open storage_config.method token, populated by the always-on built-in StoragePlugin. Third parties can ship a pipelex-storage-<backend> package that advertises itself under the pipelex.plugins entry-point group — the same discovery/denylist machinery the inference and orchestrator seams use. storage_config.method is now an open str: an unknown method fails loudly at boot with UnknownStorageMethodError (listing the registered methods), not at config parse. See the new docs/under-the-hood/storage-provider-plugins.md.
• Secrets provider plugin seam: The secrets backend is now a plugin seam selected via the new open secrets_config.method token (default "env") from a SecretsProviderRegistry, populated by the always-on built-in SecretsPlugin. Third parties can ship a pipelex-secrets-<backend> package (Vault, AWS Secrets Manager, …) under the pipelex.plugins entry-point group — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: env stays the default. An unknown method fails loudly at boot with UnknownSecretsMethodError. See the new docs/under-the-hood/secrets-provider-plugins.md.
• TOML pipeline inputs & templates: The --inputs file passed to pipelex run and pipelex-agent run (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — TOML's multi-line strings make text-heavy inputs much easier to author. run bundle <dir> auto-detects inputs.toml alongside inputs.json when --inputs is omitted, erroring if both exist. Complementarily, pipelex build inputs and pipelex-agent inputs accept --format json|toml (default json) to emit the generated inputs template as TOML. Inline JSON ({…}) and agent-CLI stdin inputs stay JSON-only; a bare TOML datetime/date/time literal is rejected with an explicit "quote it as a string" error (native datetime concept support is a separate track).
• Gateway models: Added support for nano-banana-2-lite through Pipelex Gateway.

Changed

• Dropped Python 3.10 support (Breaking): requires-python is now >=3.11,<3.15. The 3.10 compatibility bridges are gone — StrEnum, Self, and Traversable are imported directly from the stdlib at every call site, the pipelex.types re-export module that existed only to paper over 3.10 is deleted, and the backports.strenum conditional dependency (and its type stub) is dropped. CI test/lint matrices, the package-check requires-python floor gate, and the contributor docs now start at 3.11.
• Plugin API version bumped to 3 (Breaking): PLUGIN_API_VERSION is now 3 (was 2) to add the add_storage_provider and add_secrets_provider registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare targets_api = 3. pipelex-mistralai-workflows registers no storage or secrets provider, so the targets_api bump is its only change. pipelex-temporal needs the bump and a code migration: its payload-codec factory imports the now-removed make_storage_provider_from_config (see Removed) and must switch to resolving the provider via the registry (get_storage_provider_registry().get_required(method=...)).
• PipeSignature is not a pipe type (Breaking): A signature is now declared by omitting type — a [pipe.x] section with no type and nothing but the contract (description, output, optional inputs, optional signature_for) is a PipeSignature. Writing type = "PipeSignature" explicitly is rejected with a migration error ("PipeSignature is no longer a pipe type — delete the type line"), and a typeless section declaring any non-contract field is a hard error naming the field. The pipelex-agent pipe authoring command mirrors this. MTHDS JSON Schema shape change: the signature arm no longer carries a type property (an explicit tag now fails schema validation), so the downstream schema copies (mthds, vscode-pipelex, mthds-ui) must be re-synced on the next release via the mthds-schema-sync skill.
• PipeCondition continue now resolves the output as absent (Breaking): The continue special outcome no longer passes the current main stuff through (and no longer errors when there is none). It records a declared-absent AbsenceRecord for the condition's declared output — with the evaluated expression as the reason — and returns success, memory otherwise unchanged. A continue-reachable PipeCondition must therefore declare its output optional (e.g. output = "Constraint?"), or validation rejects it (optional_output_required). Dry-run parity holds. Migration: the previous value stays in working memory under its own name, so downstream pipes consume it explicitly by that name (declared ? when it may be absent); a coalescing operator is the planned ergonomic replacement for pass-through.
• Controllers combine under absence: PipeParallel no longer crashes when a branch result resolves absent (lifted branch or continue): with a Composite output the absent component is omitted (ledger note kept), and with a structured output a non-required field absorbs the absence as its default (None unless the author declared another), while a required field fed an absent branch raises a typed error naming the branch, the field, and the fixes. PipeBatch compacts: absent branch results are dropped from the aggregated list, so batching a "keep or skip" condition over items yields only the kept results.
• Absent output delivery is now a first-class success: A run whose declared ? output resolves absent succeeds everywhere. main_stuff_name on PipelexPipeRunOutput and the /execute response names the declared output slot even when it resolved absent — consumers branch on the absence record in the serialized working memory (absences), never on transport, and the ledger round-trips cross-process (hydrate_working_memory reconstructs it). The main_stuff.json/md/html artifact files become an explicit absence document ({"absent": true} plus the provenance chain) on both the typed and raw delivery paths, pipelex run / --save-main-stuff and the agent CLI's run surface it instead of crashing, and OTel/Langfuse capture serializes it. A working memory with neither a value nor a recorded absence still fails delivery loudly.

Fixed

• Template truth-tests on singular values no longer crash: {% if var %} (and @?var, which expands to it) on a present singular value used to raise TypeError: '…' content does not support len(). — Jinja2's truth test fell through to the artefact's list-only __len__. StuffArtefact now defines __bool__: a present non-list artefact is truthy, a list artefact follows list emptiness. This makes the optionals guard idiom safe on both arms.
• PipeCompose escaped-sigil literals ($$, @@) no longer double-rewrite: PipeCompose alone stored its template already-preprocessed and then re-ran the sigil rewriter at guard-lint and render time. Because the escape collapse ($$name → $name) is not idempotent, the second pass resurrected escaped literals into interpolations — raising a spurious optional_input_unguarded on a $$name that only looks like an optional reference, and silently rendering the value of name instead of the literal $name. PipeCompose now stores authored source and rewrites exactly once, like every other pipe.
• --inputs ~/… now expands on run pipe / run bundle: a quoted or =-form tilde path (--inputs "~/inputs.json", which the shell leaves unexpanded) is now expanduser()-ed before loading, resolving to the home directory instead of failing on a literal ~ — matching the existing run method behavior, on both the human and agent CLIs.
• Invalid storage/secrets method errors stay actionable under STRICT: UnknownStorageMethodError / UnknownSecretsMethodError are marked caller-facing, so their "registered methods: … — check storage_config.method" guidance survives STRICT error disclosure instead of being redacted to a generic internal-error message.
• Non-string JSON pipe type gives an actionable error: a type that is a list/dict/number in a pipelex-agent pipe JSON spec now surfaces as an ArgumentError naming the valid pipe types, instead of a cryptic internal TypeError: unhashable type.
• GitHub Actions: Removed an invalid environment block from the manual-trigger-tests-check.yml workflow.

Documentation

• Optionality guide: New "Understanding Optionality" page (docs/building-methods/pipes/understanding-optionality.md) beside the multiplicity guide — presence markers, the runtime trichotomy (skip / run / fail), absence records and provenance, template guards, controllers under absence, and the static safety net. PipeBatch documents compaction under absence, PipeLLM documents the @? optional block sigil, and the run CLI page documents the absence artifact an absent main output produces.

Removed

• make_storage_provider_from_config (Breaking): The module-level storage-provider factory (pipelex/tools/storage/storage_provider_factory.py) is removed — storage is now resolved through the config-selected StorageProviderRegistry at boot. Downstream code that imported this helper (notably pipelex-temporal's payload-codec factory) must select through the registry instead.

What's Changed

• PipeSignature is not a type — declare signatures by omitting the type by @lchoquel in #1023
• feat: TOML pipeline inputs + --format toml template generation by @lchoquel in #1022
• feat(plugins): storage & secrets provider plugin seams by @lchoquel in #1024
• Drop Python 3.10 support by @lchoquel in #1025
• Optionals phase 1: presence markers ?/!, absence ledger, and static optionality validation by @lchoquel in #1021
• Release v0.38.0 by @lchoquel in #1026

Full Changelog: v0.37.0...v0.38.0