feat(protocol)!: PipelexRunner → PipelexMTHDSProtocol — implement the MTHDS Protocol locally

[thomashebrard]

Summary

Phase B2 of the MTHDS Protocol unification program (workspace docs/plans/mthds-protocol-20260610/03-pipelex.md). The runtime now implements the MTHDS Protocol standard (mthds-ai/mthds#44) through mthds >= 0.4.0 (mthds-ai/mthds-python#25).

Breaking changes

• PipelexRunner → PipelexMTHDSProtocol, implementing MTHDSProtocol: execute_pipeline → execute, start_pipeline → start (still NotImplementedError locally — async execution belongs to the API layer; signature gains the protocol's run_id/callback_urls/method_id).
• Response classes: PipelexPipelineExecuteResponse → PipelexRunResult, PipelexPipelineStartResponse → PipelexStartAck; wire fields run_id/state on the response models ONLY — runtime internals (PipeOutput, tracing, telemetry, Temporal) keep pipeline_run_id per master D1 (deep internal rename is explicitly out of scope).
• mthds pin → >= 0.4.0. Temporarily sourced from the mthds-python refactor/protocol git ref via [tool.uv.sources]; the source override drops when the PyPI release is cut at the Phase B checkpoint.

Added

• Protocol validate / models / version on the runner: validate wraps validate_bundle (blueprints + per-pipe structures → protocol ValidationReport); models wraps the builder's list_models → protocol ModelDeck; version reports protocol_version 0.1.0.

Also

• Agent CLI API path now uses MthdsAPIClient directly (upstream ApiRunner wrapper deleted), context-managed so the HTTP client closes.
• Builder's generated runner code emits the new API.
• Docs swept; CHANGELOG v0.33.0 includes the pipelex-worker lockstep note (Temporal payload renames ride this pin at Phase C deploy).

Verification

• make agent-check clean — ruff + plxt + pyright + mypy over the full tree.
• make agent-test green (full suite, exit 0).
• Live CLI smoke: pipelex run bundle with a real LLM call and pipelex validate bundle both pass through the refactored runner; zero runner/v1/platform/v1 strings remain in the repo.

Coordination

Blocks pipelex-api (05) which pins this release; pipelex-worker bumps the same version in lockstep at Phase C.

🤖 Generated with Claude Code

---

Summary by cubic

Implements the local MTHDS Protocol via PipelexMTHDSProtocol (replacing PipelexRunner), aligning method names, response models, and wire fields; updates docs/CLI and pins mthds >= 0.4.0 via git. Adds clean input URL errors and protocol-compliant webhooks.

• Migration

  • PipelexRunner → PipelexMTHDSProtocol; execute_pipeline → execute, start_pipeline → start (still not implemented locally). No callback_urls/method_id; both methods accept extra and reject non-empty values. pipeline_run_id stays (incl. start kwarg).
  • Responses: PipelexPipelineExecuteResponse/PipelexPipelineStartResponse → PipelexRunResultExecute/PipelexRunResultStart (protocol RunResult* bases). Wire uses state; run identifier remains pipeline_run_id.
  • Webhooks: completion payload sends pipeline_run_id + state (plus transitional status); state is now a reserved webhook key.
  • Imports/CLI: moved to mthds.protocol.* and mthds.runners.api.*; Agent CLI uses MthdsAPIClient; builder emits the new API; docs updated (incl. mthds_contents fix).
  • Dependencies: pin mthds to >= 0.4.0 via a git source in [tool.uv.sources] until the PyPI release.

• New Features

  • validate: returns a protocol ValidationReport; restores the caller’s current library and closes the validation library on success.
  • models: returns a protocol ModelDeck (optional category filter).
  • version: reports protocol_version 0.1.0 and the installed Pipelex version; falls back to "unknown" for source checkouts.
  • Error handling: blank or unreadable input URLs now raise PipelineInputContentError (INPUT domain).

^{Written for commit 8b34cc0. Summary will update on new commits.}

[greptile-apps]

Greptile Summary

This PR moves the local runner onto the MTHDS protocol shape. The main changes are:

• PipelexRunner is replaced by PipelexMTHDSProtocol with execute and start.
• Run responses now use run_id and state protocol fields.
• Local protocol methods for validate, models, and version are added.
• The agent API path now uses MthdsAPIClient directly.
• Generated runner code, docs, tests, and the mthds dependency are updated.

Confidence Score: 3/5

This should wait for fixes before merging.

• Successful protocol validation can leave a temporary library open and current.

• Several updated docs now point users at entrypoints that do not exist or have the wrong call shape.

• The new version handshake can crash in source-checkout runs without installed package metadata.

• pipelex/pipeline/runner.py

• pipelex/pipeline/validate_bundle.py

• The updated docs examples that mention execute or PipelexClient

Important Files Changed

Filename	Overview
pipelex/pipeline/runner.py	Adds the local MTHDS protocol implementation, including validation and version discovery paths that need cleanup.
docs/building-methods/libraries.md	Updates runner examples, but some snippets now call nonexistent or incorrectly shaped APIs.
docs/building-methods/pipes/provide-inputs.md	Updates client execution naming while keeping a removed PipelexClient import.
docs/building-methods/pipes/pipe-output.md	Updates execution examples but omits the required runner instance and response wrapper.

Comments Outside Diff (1)

1. docs/building-methods/pipes/provide-inputs.md, line 380-386 (link)

   Remove stale client This updated example imports PipelexClient from pipelex.client, but that client is no longer present in this package. A reader following the snippet gets ModuleNotFoundError before reaching the new client.execute(...) call. Please update the example to the actual local runner or the current mthds API client.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: docs/building-methods/pipes/provide-inputs.md
   Line: 380-386
   
   Comment:
   **Remove stale client** This updated example imports `PipelexClient` from `pipelex.client`, but that client is no longer present in this package. A reader following the snippet gets `ModuleNotFoundError` before reaching the new `client.execute(...)` call. Please update the example to the actual local runner or the current `mthds` API client.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 5 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 5
docs/building-methods/libraries.md:53-56
**Fix runner example** This snippet now calls a bare `execute()` function and passes `library_dirs` into it, but the changed API only exposes `execute` as an instance method on `PipelexMTHDSProtocol`, with `library_dirs` configured on the constructor. Copying this example fails with `NameError`, and adapting it to `runner.execute(...)` still fails because `execute` does not accept `library_dirs`.

### Issue 2 of 5
docs/building-methods/pipes/pipe-output.md:27-30
**Use real entrypoint** The example now tells readers to call `execute(...)` directly, but this PR does not add a module-level `execute` helper. The only runtime entrypoint is `PipelexMTHDSProtocol().execute(...)`, which returns a response wrapper, so this snippet fails as written and also skips `response.pipe_output`.

### Issue 3 of 5
docs/building-methods/pipes/provide-inputs.md:380-386
**Remove stale client** This updated example imports `PipelexClient` from `pipelex.client`, but that client is no longer present in this package. A reader following the snippet gets `ModuleNotFoundError` before reaching the new `client.execute(...)` call. Please update the example to the actual local runner or the current `mthds` API client.

### Issue 4 of 5
pipelex/pipeline/runner.py:307-312
**Clean validation library** `validate_bundle` opens a fresh library and only tears it down on failure, so successful calls through this new protocol method leave that validation library open and current. A caller that runs `await runner.validate(...)` and then performs another operation in the same task can inherit the validation-only current library, and repeated validations keep accumulating libraries. The protocol wrapper should ensure the validation library is restored and torn down on the success path too, either by fixing `validate_bundle` or by using a validation helper with success cleanup.

### Issue 5 of 5
pipelex/pipeline/runner.py:359
**Handle source checkouts** `metadata.version("pipelex")` raises `PackageNotFoundError` when Pipelex is run from a source checkout that is on `PYTHONPATH` but is not installed as a distribution. That makes the new public protocol `version()` method fail even though the local runtime can otherwise execute methods. Please catch the metadata error and return a stable fallback version.

```suggestion
        try:
            pipelex_version = metadata.version("pipelex")
        except metadata.PackageNotFoundError:
            pipelex_version = "0.33.0"
```

_{Reviews (1): Last reviewed commit: "feat(protocol)!: PipelexRunner → Pipelex..." | Re-trigger Greptile}

[cubic-dev-ai]

1 issue found across 54 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[thomashebrard]

Review triage (greptile + cubic) — all five findings accepted and fixed in the latest commit:

• ✅ Clean validation library (greptile P1): validate() now restores the caller's current-library and tears down the validation library on success (on failure validate_bundle already does both — the wrapper's cleanup is a no-op there).
• ✅ Handle source checkouts (greptile P2): version() catches PackageNotFoundError → "unknown" (a hardcoded fallback version would go stale silently).
• ✅ Fix runner example + Use real entrypoint (greptile P2 ×2) and mthds_content singular (cubic P0): these doc snippets were stale against an API that disappeared before this PR (bare execute() helper, per-call library_id/library_dirs, singular mthds_content) — the rename sweep exposed them. All examples in libraries.md / pipe-output.md / executing-pipelines.md rewritten onto the real PipelexMTHDSProtocol API.

Also in the same push (cross-repo follow-up from the pipelex-api phase): the webhook completion payload now carries run_id/state alongside the legacy pipeline_run_id/status for one transitional release (master D1/D7; the run-completion Lambda accepts both spellings during the deploy skew), and both new keys join the reserved webhook payload set.

make agent-check clean; delivery + CLI test modules green.

[thomashebrard]

Heads-up: D1 was revised by user ruling during C2 — the run identifier stays pipeline_run_id everywhere on the wire (only pipeline_state → state renames). The latest commit reverts the id half of the rename: PipelexRunResult.from_pipe_output(pipeline_run_id=…), webhook payloads carry pipeline_run_id + state (+ transitional status), reserved keys updated. Full suite green.

[thomashebrard]

Dev-debug fix: a blank url on a Document/Image input crashed the hosted runner with a sanitized 500 (Path('') → '.' → IsADirectoryError, only FileNotFoundError was caught). New PipelineInputContentError (INPUT domain → 422): blank urls rejected before resolution, and the whole OSError surface around the local-file read mapped to it. TDD tests in tests/unit/pipelex/pipeline/test_input_normalizer.py; full suite green. Pairs with a security hardening in pipelex-api-deploy (local-path upload disabled on hosted — LFI).