Skip to content

Release v0.5.0#11

Merged
lchoquel merged 12 commits into
mainfrom
release/v0.5.0
Jul 22, 2026
Merged

Release v0.5.0#11
lchoquel merged 12 commits into
mainfrom
release/v0.5.0

Conversation

@lchoquel

@lchoquel lchoquel commented Jul 22, 2026

Copy link
Copy Markdown
Member

Release v0.5.0

Bumps version from 0.4.0 to 0.5.0.

Changelog

Added

  • Input preparation: upload_file and prepare_inputs (hosted upload capability). The Python counterpart of @pipelex/sdk's uploadFile / prepareInputs, in parity. client.upload_file(source, *, filename=None, content_type=None) uploads one local asset — a filesystem path (str/Path) or raw bytes — and returns an UploadRecord (uri, content_type, size, filename) assembled client-side. client.prepare_inputs(*, files, pipe_ref=None, inputs) resolves the target pipe's declared signature from the explicit inputs template, interprets the caller's compact inputs top-down against it (the file signal is the canonical Image/Document content shape — a {"url": …} dict — mirroring the runtime's input_normalizer), uploads the file-bearing values, and returns PreparedInputs: a copy-on-write rewrite of inputs with each asset reference replaced by canonical content carrying pipelex-storage:// in url, plus one UploadRecord per prepared asset. HTTP(S) URLs and existing pipelex-storage:// URIs pass through unchanged; data URLs and local/byte sources are uploaded; the same source referenced twice uploads once (dedup by source identity); all failures are raised before any run is created. Failures are typed per category: InvalidLocalSourceError, RejectedAssetError, UnsupportedUploadCapabilityError, UploadAuthenticationError, UploadTransportError (all extend InputPreparationError). prepare_inputs takes the method closure as inline files; catalog method_id resolution and opt-in http(s) ingest are deferred and additive. See docs/input-preparation.md.

  • build_inputs route (POST /v1/build/inputs). Closes the /v1/build/* parity gap the Python SDK had — client.build_inputs(BuildInputsRequest) projects a pipe's declared inputs as a fill-in template, returning a 200 verdict discriminated on is_valid (BuildInputsValidReport | CrateInvalidReport); a no-verdict condition throws ApiResponseError. It is the signature source prepare_inputs reads (with explicit=True). Models live in pipelex_sdk/build_models.py.

  • Typed run usage: RunResults.tokens_usages + RunResults.usage_assembly_error. The per-call usage records a run produces — token counts by category, the server-computed cost in USD, model name and id, the pipe that made the call, job-kind fields and timing, for LLM and img-gen/extract/search calls alike — are now first-class typed fields instead of riding model_extra. Records validate into a new TokensUsageRecord model (pipelex_sdk/runs.py) mirroring the wire contract specified in the MTHDS protocol spec. Both paths populate the pair: the hosted durable path reads it off GET /v1/runs/{id}/results (which unpacks the runner's tokens_usages.json artifact), and the blocking fallback lifts the same pair out of the execute response's extension-open pipe_output — so result.tokens_usages reads the same regardless of which path ran.

    Note that the rate table (unit_costs) no longer crosses the wire: a record now carries the computed cost for the call instead, which is None when the model has no rate table at all (own-GPU, mock, dry run) and 0 when a rate table priced it at zero. There is no run-level aggregate — sum the records.

    tokens_usages is None whenever usage assembly produced no list (it was off, it broke, or the run was delivered before the artifact existed) and [] when assembly ran and no inference happened; usage_assembly_error is the only field separating a broken assembly from an off one. TokensUsageRecord keeps every field optional and is extension-open, so durable artifacts written before the contract shipped — relayed verbatim, never migrated — still parse: cost and pipe_code come back None, and the legacy job_metadata / unit_costs survive in model_extra. Enum-ish fields (model_type, job_category, unit_job_id) are open sets typed as plain str, so runtime enum churn stays non-breaking.

Changed

  • RunResults.pipe_output is now DictPipeOutputAbstract | None, was dict[str, Any] | None (breaking). The blocking path already parsed the protocol model and then threw the types away with a .model_dump() round-trip; it now carries the parsed model straight through. Read the working memory as attributes — result.pipe_output.working_memory.root["out"].content — rather than nested dict keys. The durable path still leaves it None.

🤖 Generated with Claude Code


Summary by cubic

Release v0.5.0 adds hosted input preparation (upload_file, prepare_inputs), introduces build_inputs for signature templates, and exposes typed run-usage records on results. It also changes RunResults.pipe_output to a typed model. Adds a WIP doc with deferred review findings (no code changes).

  • New Features

    • Input preparation: client.upload_file(source, filename?, content_type?) and client.prepare_inputs(files, pipe_ref?, inputs) rewrite file inputs to pipelex-storage:// URLs, pass through http(s)/existing URIs, dedup repeated sources, and raise clear typed errors. Mirrors @pipelex/sdk.
    • build_inputs support: client.build_inputs(BuildInputsRequest) returns the explicit inputs template with an is_valid verdict; used by prepare_inputs.
    • Typed run usage: RunResults.tokens_usages and usage_assembly_error validate into TokensUsageRecord and are available on both hosted and blocking paths; each record includes per-call token counts and computed USD cost.
  • Migration

    • RunResults.pipe_output is now DictPipeOutputAbstract | None (was dict). Access working memory via attributes, e.g. result.pipe_output.working_memory.root["out"].content.

Written for commit 4ef1d1e. Summary will update on new commits.

Review in cubic

lchoquel and others added 11 commits July 18, 2026 17:28
Both paths populate them: the hosted durable path reads the pair
GET /v1/runs/{id}/results now returns (unpacked from the runner's
tokens_usages.json artifact — pipelex-platform PR #82), and the blocking
fallback unpacks the same pair from the execute response's
extension-open pipe_output, so result.tokens_usages reads the same
regardless of which path ran. None against older platforms or
pre-artifact runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`RunResults.tokens_usages` was a `list[dict[str, Any]]` bag. It now validates
into `TokensUsageRecord`, a client-side mirror of the wire contract specified in
docs/specs/pipelex-mthds-protocol.md — every field optional and `extra="allow"`,
so pre-contract durable artifacts (relayed verbatim, never migrated) still parse
with `cost`/`pipe_code` None and their legacy `job_metadata`/`unit_costs` kept in
`model_extra`. Enum-ish fields stay plain `str` so runtime enum churn is
non-breaking.

Both paths feed the same typed records: the durable path off the results body,
the blocking path lifted out of the execute response's extension-open
`pipe_output`.

Breaking: `RunResults.pipe_output` is now `DictPipeOutputAbstract | None` rather
than `dict[str, Any] | None`. The blocking path already parsed the protocol
model and then discarded the types via `.model_dump()`; it now carries the parsed
model through, so consumers read the working memory as attributes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the TokensUsageRecord wire contract on RunResults
…mple

Review follow-ups raised on the TypeScript mirror of this page; the same three
defects were present here, so both SDKs stay in sync.

- Drop a dead link to the protocol spec. It pointed at a path that does not
  exist in the public runtime repo, and the spec's real home is an internal
  repo, so there is no public target to redirect to — reference the spec and
  its section by name instead. While there, correct the brand: the record is a
  Pipelex runtime concept, not part of the MTHDS standard, which says nothing
  about usage reporting.
- Make the example meaningful. `mthds_contents=[...]` parses in Python but
  passes an `Ellipsis` where a bundle source is expected; a registered
  `pipe_code` with `inputs` is both valid and the more idiomatic hosted call.
- Scope two guarantees that the page stated absolutely while documenting their
  exception further down. The spec qualifies both with "on a record emitted
  under this contract": pre-contract artifacts are relayed verbatim, so they
  do carry a raw `unit_costs` rate table and `job_metadata`. Name the
  exemption at both claims rather than only at the compatibility section.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scope the contract-only usage claims and fix the run-usage example
Phase 0 of the hosted input-upload & SDK-preparation project: promote the
approved contract into this repo's own docs before the code lands. New
docs/input-preparation.md is the Python-flavored mirror of the JS SDK contract —
upload_file/prepare_inputs, str/Path/bytes sources, signature-driven asset
identification, upload-record guarantees, error/capability outcomes, inherited
storage policy, and stability across the future endpoint move. Notes the
build_inputs parity gap prepare_inputs closes. Pointer added from
architecture.md's storage bullet. Docs-only; no code touched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9RPphjd62DywHtok2fX1q
…@pipelex/sdk)

Add the hosted upload-capability surface, the Python counterpart of @pipelex/sdk's
uploadFile/prepareInputs, plus the build_inputs route the signature step needs:

- client.upload_file(source, *, filename=None, content_type=None) -> UploadRecord —
  accepts str/Path paths and raw bytes; record (uri, content_type, size, filename)
  assembled client-side (mimetypes for MIME).
- client.prepare_inputs(*, files, pipe_ref=None, inputs) -> PreparedInputs — resolves
  the pipe's declared signature via build_inputs(explicit=True), template-guided walk
  (file signal = a canonical {url} content dict, mirroring the runtime's
  input_normalizer), uploads file-bearing values, returns a copy-on-write rewrite
  carrying pipelex-storage:// in `url` plus one UploadRecord per prepared asset.
  http(s)/pipelex-storage:// pass through; data URLs and local/byte sources upload;
  dedup by source identity; all failures before any run. An unrecognized value at a
  file position fails as a typed InputPreparationError.
- client.build_inputs(BuildInputsRequest) -> BuildInputsResponse — closes the
  /v1/build/* parity gap the Python SDK had. The response is an is_valid discriminated
  union parsed through BuildInputsResponseAdapter (a TypeAdapter), mirroring the repo's
  PipelexValidationResultAdapter precedent, so a malformed 200 raises a clean
  ValidationError rather than being read as a valid verdict.
- Typed exception family mirrors the JS SDK: InputPreparationError base +
  InvalidLocalSource / RejectedAsset / UnsupportedUploadCapability /
  UploadAuthentication / UploadTransport.

Scope: prepare_inputs takes the closure as inline files; catalog method_id and opt-in
http(s) ingest deferred (additive). Matrix-derived parity tests + real-client wiring
tests. make agent-check + make agent-test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9RPphjd62DywHtok2fX1q
…feedback)

Addresses the bot-review findings on PR #10 that are genuine correctness or
doc-accuracy issues; deliberately defers the rest.

Fixed:
- Malformed base64 data URL no longer escapes the typed-error contract. Decode
  with `validate=True` (junk chars are rejected, not silently discarded into
  corrupted bytes) and map `binascii.Error` to `InputPreparationError`, so all
  preparation failures stay within the documented exception family. (Codex,
  Greptile, cubic all flagged this.)
- Percent-encoded binary data URLs keep their exact bytes: decode with
  `urllib.parse.unquote_to_bytes` instead of `unquote(...).encode("utf-8")`,
  which corrupted any byte >= 0x80.
- docs/input-preparation.md: the URL / storage-URI pass-through is a
  `prepare_inputs`-level behavior, not a feature of `upload_file` (which treats
  every string as a filesystem path); removed the `checksum` row that listed a
  field `UploadRecord` does not have.
- docs/architecture.md: upload_file/prepare_inputs are now implemented, not a
  "planned addition".

Deferred (not bugs): offloading the file read/base64 off the event loop (perf
nicety on an inherently I/O-bound op), a template-required-per-format validator
on BuildInputsValidReport (prepare_inputs already raises a clean typed error),
and splitting a P3 two-scenario transport-error test.

Regression tests: malformed base64 (bad padding + non-alphabet) raises a typed
error and uploads nothing; percent-encoded binary round-trips its exact bytes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
… shape

Second pass on PR #10 review-bot feedback — the two remaining substantive
items, both promoted from "defer" to "fix now" after independent verification
showed each closes a documented JS-parity gap in new code at near-zero cost.

- upload_file: offload the synchronous whole-file read via asyncio.to_thread so
  a large read no longer blocks the event loop before the first await. Matches
  the JS SDK (which reads off-loop via node:fs/promises) and the repo's
  async-only invariant. base64 stays inline on purpose — CPython's binascii
  holds the GIL, so threading it would not free the loop (JS also encodes
  inline); `_to_asset_bytes`/`_read_path` stay sync. (Greptile P2 + cubic P2.)
- BuildInputsValidReport: add a model_validator enforcing the template-present-
  per-format invariant (inputs for json, inputs_toml for toml, not the other).
  This makes the adapter's documented malformed-200 guarantee true for the
  template shape too — an `is_valid: true` body missing its template now raises
  a clean ValidationError at parse time instead of surfacing one layer down in
  prepare_inputs with a misleading "got json" message. Mirrors the JS sibling's
  per-format-required modeling. (cubic P2.)
- Tests: assert the read is offloaded (to_thread awaited with _to_asset_bytes,
  real behavior preserved via wraps); assert the validator rejects the three
  malformed template shapes and accepts a well-formed toml report. Parametrized
  the two-scenario transport-error test for per-case attribution (cubic P3).

make agent-check + make agent-test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
feat: upload_file + prepare_inputs + build_inputs route (parity with @pipelex/sdk)
@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown

Greptile Summary

This PR releases v0.5.0 with hosted input preparation and typed run usage. The main changes are:

  • Adds file upload and signature-driven input preparation.
  • Adds the /v1/build/inputs client route and models.
  • Adds typed per-call usage records to run results.
  • Preserves the protocol model for blocking-run pipe output.
  • Updates documentation, tests, and package versions.

Confidence Score: 4/5

The upload rejection mapping needs a fix before merging; empty list templates can also bypass file preparation.

Some valid asset rejections are exposed as transport failures. Empty list templates can leave raw file values in prepared inputs. The main build-input and run-usage paths otherwise preserve their documented shapes.

pipelex_sdk/upload.py and pipelex_sdk/prepare_inputs.py

T-Rex T-Rex Logs

What T-Rex did

  • The focused pytest run reproduced asset rejections surfacing as transport errors when uploading assets through the public upload_file path, with HTTP 400 and 422 cases triggering UploadTransportError rather than RejectedAssetError, and it surfaced the originating ApiResponseError with full traces.
  • After the implementation revisions (9ca1147 and then bba5074), the test harness shows the prepare_inputs path is available, captures the expected requests and a successful rewritten upload, and the resulting upload record includes URI, filename, application/octet-stream content type, and a 16-byte size.
  • Artifacts from both proofs include the repro harness, the stack traces, and related logs that enable inspection of the observed upload behavior and the rewritten upload result.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
pipelex_sdk/prepare_inputs.py Adds template-guided asset discovery, upload deduplication, data URL decoding, and copy-on-write input rewriting.
pipelex_sdk/upload.py Adds path and byte upload normalization plus semantic error mapping.
pipelex_sdk/build_models.py Adds discriminated request and response models for explicit input templates.
pipelex_sdk/client.py Exposes the new build, upload, and preparation APIs and maps blocking usage data into run results.
pipelex_sdk/runs.py Adds extension-open usage records and types blocking pipe output with the protocol model.
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
pipelex_sdk/upload.py:101-103
**Asset Rejections Become Transport Errors**

When the upload endpoint rejects an asset with a client-error status other than 401, 403, 404, or 413, this fallback raises `UploadTransportError`. A validation rejection such as 400 or 422 is therefore reported as a network or server failure instead of the documented `RejectedAssetError`, so callers cannot handle rejected assets by category.

### Issue 2 of 2
pipelex_sdk/prepare_inputs.py:155-160
**Empty List Template Skips Uploads**

If an explicit template represents a multiple file input with an empty list, the truthiness check skips the element walk. Caller values such as a list of raw bytes then remain unprepared, and the later run receives bytes instead of canonical `{"url": ...}` content.

Reviews (1): Last reviewed commit: "Release v0.5.0" | Re-trigger Greptile

Comment thread pipelex_sdk/upload.py
Comment thread pipelex_sdk/prepare_inputs.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bba5074a29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pipelex_sdk/prepare_inputs.py
Triaged the unresolved SWE-bot comments on the v0.5.0 release PR against
the JS SDK (@pipelex/sdk) and the pipelex runtime. Records the one
confirmed-but-deferred bug (nested asset under a structured url field is
not uploaded — parity-bound, needs an upstream template-contract change)
plus the server-side 422-vs-413 upload seam, and dismisses the two false
positives. No source changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
@lchoquel
lchoquel merged commit d05fa87 into main Jul 22, 2026
20 of 21 checks passed
@lchoquel
lchoquel deleted the release/v0.5.0 branch July 22, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant