Skip to content

Releases

v0.6.0

Choose a tag to compare

@jlevy jlevy released this 26 Oct 22:50
4289836

Metaproc v0.6.0

A minor release, because a gcp run job’s console moved off Cloud Logging and a mapped
composite now runs as concurrently as its run was told to.

Most of what changed is about a long cloud run staying observable without becoming
expensive to observe.
A gcp run job appends its console to a file on the job’s run-storage mount and sends
Cloud Logging only a start line, a heartbeat a minute, and the exit line; a bounded live
view in the dispatch bucket lets metaproc gcp logs and metaproc gcp status read the
job without mounting the run-storage share.
Validation reports now carry each layer’s actual execution rather than leaving it
unknown.

The one change that alters how an existing process runs is the mapped-composite scope
ceiling: a run whose ceiling is 80 or 160 used to open at most 32 item scopes anyway,
and now opens as many as the ceiling admits.
Read Compatibility before upgrading.

No CLI command or flag was removed, no adapter default or accepted model identifier
changed, and no plugin entry point moved.

What’s Changed

Features

  • A gcp run job keeps its console in a file. With a run-storage mount and the
    shipped wheel, the command runs inside metaproc.cloud.gcp.gcp_run_console, which
    appends its output to <mount>/logs/gcp-run/<job>/console.log. Cloud Logging keeps
    the wrapper’s start line, a heartbeat a minute, and the exit line, plus the bootstrap
    output from before the wrapper starts — so a job that fails during bootstrap is still
    diagnosed there, while the console file is the complete record of the command itself.
    A whole run’s console no longer has to be a Cloud Logging stream to be readable.

    • --console file requests the file explicitly, --console cloud-logging keeps the
      whole stream in Cloud Logging, and the default --console auto chooses the file
      when the job has a run-storage mount and ships the wheel.
      Cloud Logging therefore remains the default for --no-wheel jobs.
      --console file without a mount is refused rather than silently downgraded.
    • A bounded atomic live view in the dispatch bucket pairs the console tail with its
      cursor on each heartbeat, so a reader always sees a consistent tail.
    • --watch-run <container path> names run directories, repeatably, whose
      metaproc status a background worker caches every five minutes.
      It needs the file console.
    • metaproc gcp logs <job> gains --follow, --errors, and --cloud-logging, and
      metaproc gcp status <job> --run-status reports a watched run’s status.
      Both read the job through the live view, without mounting the run-storage share.
    • Live following stops at terminal Batch state when a final upload is missing, instead
      of waiting for a tail that is never coming.
    • The dispatcher sets METAPROC_GCP_RUN_CONSOLE_PATH, METAPROC_GCP_RUN_LIVE_GCS,
      and METAPROC_GCP_RUN_WATCH inside the job; none is set by hand.
  • Validation reports state what actually ran. Each layer carries its own execution
    — not_run, completed, or errored — so a report distinguishes a check that passed
    from one that never ran.
    Built-in model-only checks include the document-enforcement-via-model-only advisory.
    A report retained from an earlier release has no such evidence, and its execution
    stays unknown rather than being guessed.

Fixes

  • A mapped composite opens as many item scopes as the run’s ceiling admits. It used
    to open at most 32 at once whatever --max-concurrency said, so a run with a ceiling
    of 80 or 160 never had more than 32 items’ leaves competing for the pool.
    Scopes hold no admission capacity of their own, so they now follow the run’s ceiling,
    with 32 as the floor.
    The pool’s memory and provider admission still bound the work in flight.
    See Compatibility: this changes how an existing process runs.

  • Frontmatter repair no longer rewrites valid frontmatter. Its self-check used
    ruamel’s raw safe loader, whose C scanner rejects flow collections that are valid YAML
    — URL values among them — so on a machine with ruamel’s C extension installed the
    repair could fire on a document that was never broken.
    The check now uses the same guarded reader as downstream validation, which falls back
    to the pure parser, and such documents are left unchanged.
    The narrow malformed-scalar repair is unaffected.

  • Resuming a cached task no longer rewrites its state bytes. A terminal task status
    took its completion time from the clock at the moment it was written, so a resume that
    reused a cached task produced a different completed_at on every tick.
    It now comes from the accepted durable attempt’s end time.

  • Pool files are written the same with or without ruamel’s C extension. With
    _ruamel_yaml installed, the safe YAML handle behind pool status, scale-state,
    scale-override, and kill-sentinel files switched to libyaml’s emitter, which wraps
    long values at different points, so the same state serialized to different bytes
    depending on what was installed.
    They now dump through metaproc.io.new_dump_yaml, which keeps the pure-Python
    emitter.

Documentation

  • Installation describes the optional YAML C extension and what
    installing it does and does not change.
  • The shipped file-I/O architecture topic (metaproc help arch-file-io) describes the
    guarded frontmatter repair check and the reader-compatibility boundary.

Compatibility

  • Two dependency floors rise. Metaproc now requires softschema>=0.9.1,<0.10 and
    frontmatter-format>=0.5.1,<0.6. Both are resolved for you on install; a pinned
    environment holding either below its floor cannot install this release, and that
    includes softschema==0.9.0 and frontmatter-format==0.5.0 — the floors are the
    patch releases, not the minor series.
    The frontmatter-format readers use guarded C-parser paths when available and fall back
    to the pure parser, and its writers retain deterministic pure-Python emitters.

  • A mapped composite may now run more work at once. This is the one change that can
    alter an existing run’s behavior without any spec or flag change.
    A process whose run ceiling is above 32 — --max-concurrency 80 or 160, say —
    previously had at most 32 item scopes open; it will now open as many as the ceiling
    admits, so more leaves compete for the pool and peak memory and provider pressure rise
    accordingly. The pool’s memory and provider admission still bound what actually
    executes, and 32 remains the floor.
    If a run was tuned against the old implicit cap, set --max-concurrency to what you
    actually want rather than relying on the 32.

  • Validation execution is additive. A report written by this release carries each
    layer’s execution; one retained from an earlier release does not, and readers treat
    that as unknown. No existing report token changed.

  • Three new environment variables, all set by the dispatcher inside a gcp run job:
    METAPROC_GCP_RUN_CONSOLE_PATH, METAPROC_GCP_RUN_LIVE_GCS, and
    METAPROC_GCP_RUN_WATCH. Nothing else in the environment contract changed.

  • Nothing public was removed. No CLI command or flag, no adapter default (opus,
    gpt-5.5, gemini-3.1-pro-preview-customtools, sonnet), no accepted model
    identifier, no process-file field, no runtime artifact shape, and no plugin entry
    point was removed or renamed.

  • Still open from v0.5.0: the GCP image interpreter. Metaproc has required Python
    3.14 since v0.5.0, and container bootstrap refuses the shipped wheel on an image whose
    /opt/venv is older.
    That image build lives outside this repository and is still unverified (mp-mo74).
    Until it is confirmed or rebuilt on 3.14 and re-pinned, dispatch with --no-wheel or
    from 0.4.x; metaproc help cloud-dispatch says the same.

  • Still open from v0.4.0: Gemini and ignored files. Gemini steps ship with
    context.fileFiltering.respectGitIgnore: False, so a Gemini step’s file tools can
    read ignored files anywhere in the workspace, .env included.
    Whether the default should become opt-in is still open as mp-x1qh; treat a workspace
    a Gemini step runs in as readable by that step.

Full commit history:
v0.5.0 … v0.6.0

v0.5.0

Choose a tag to compare

@jlevy jlevy released this 21 Oct 20:05
0714762

Metaproc v0.5.0

A minor release, because Metaproc now requires Python 3.14.

The floor moved so that Zstandard support could come from the standard library’s
compression.zstd, and most of what else changed is about a run accounting for itself.
A run can now cap what it spends and stop a mapped step whose items keep failing.
Every run writes an operations summary of where its time went.
A resume records the launch-config changes it makes instead of refusing them, re-runs a
consumer whose collected inputs changed, keeps its fingerprints when the same revision
is planned from another checkout, and can re-enter one nested step or mapped item
through its parent. Every executable leaf of a local run, scalar or mapped, agent or
code, now draws on one adaptive controller, which backs off when a provider refuses on a
per-minute quota. Gemini steps run in a private Gemini home per launch, and failures of
every kind keep a bounded, credential-redacted cause.

No CLI command or flag was removed, no adapter default or accepted model identifier
changed, and no plugin entry point moved.
Some Python APIs were renamed, and three changes can stop something that worked on
v0.4.1: the 3.14 floor, the GCP image interpreter, and mapped agent steps on profiles
with different pool resources.
Read Compatibility before upgrading.

What’s Changed

Features

  • A run can cap its own spend. run-process --max-spend-usd X sets one cap on the
    run’s measured list cost, shared by every scope it enters and kept for the run’s whole
    life: it is recorded as max_spend_usd in run-config.yaml, a resume without the
    flag keeps it, and .state/spend-ledger.yaml (metaproc:SpendLedger/0.1) carries the
    measured spend, so a resume on a host without the earlier logs still counts them.
    Three mechanisms read the cap:

    • A mapped step may declare spend: {per_item_usd, source, max_attempts}. After reuse
      has chosen the actionable items and before any starts, the step refuses when the
      spend already measured plus items × attempts × price exceeds the cap, and names the
      cap that would admit it.
      Only actionable items count, so a resume is priced for the work it will run.
      max_attempts defaults to one plus the largest resolved retry.max_retries among
      the agent steps an item runs.
    • Mapped composite and code steps stop starting items once measured spend reaches the
      cap. Running items finish and the rest stay pending for a resume.
    • for_each.breaker: {max_failure_fraction, min_finished} stops a mapped composite or
      code step’s dispatch once more than that fraction of the items it finished in this
      invocation failed.

    A stopped step keeps state: failed, which every reader already treats as resumable,
    and gains a stopped record in process-status.yaml with the reason, phase, and
    numbers. metaproc status labels the run BUDGET-STOPPED or BREAKER-STOPPED with a
    Stopped: line per step.
    The cap is cumulative, so a budget-stopped run resumed with the same cap refuses
    again; a higher one continues it.
    A process may declare spend_cap: {required: true} to refuse an uncapped launch, or
    spend_cap: {default_usd: X}. Neither a price nor a breaker enters a step
    fingerprint. Measured spend is list cost and a lower bound where a model is unpriced,
    not provider billing.

  • Every run writes an operations summary. Run finalization writes
    operations-summary.md (metaproc.operations:AgentOperationsSummary/v1) beside
    resource-usage-summary.md: real elapsed time, setup, per-stage shares, per-item
    chain running time and barrier wait, step durations, pool concurrency, attempt
    dispositions, agent transcripts, and machine resources.
    A figure the run cannot establish is null with a stated reason, and a failure while
    summarizing is logged without changing the run’s outcome.
    metaproc operations summary RUN_DIR builds the document for a finished run (and
    --write writes it); metaproc operations rollup RUN_DIR... sets runs side by side
    against a per-item chain-time target (--target-min-minutes, --target-max-minutes,
    10 and 15 by default) and reports a run whose summary does not read as unreadable. A
    run that executed as a child scope of a larger run is summarized with its nested
    scopes and transcripts.
    A run that never reached finalization gets its summary from the next metaproc status
    on it. A list cost that leaves out unpriced models reads at least.

  • A resume records launch-config changes instead of refusing them. v0.4.1 refused a
    resume whose resolved variables differed from the ones run-config.yaml recorded.
    A resume may now change any resolved variable (including through an edited
    default:), the --step-variant set, --variant, --artifact-namespace, the
    execution profiles the plan resolves, --backend, the spend cap, and the checkout’s
    git_sha. Each change prints once as Resume changes <field>: <old> -> <new>, all of
    one resume’s changes are appended with old and new values as one
    launch_config_change event in .logs/dispatch-config-changes.jsonl, and
    run-config.yaml is rewritten to the values the resume ran with, so status --steps
    and the operations summary describe the run as it now executes.
    A resume still refuses, before recording anything, on a corrupt run-config.yaml, a
    different process name, or a different run directory (the two canonical Filestore
    mount roots excepted), and names the recorded value and the two ways out.
    What re-runs is still decided by step fingerprints: a value substituted into a
    resolved field such as env: re-runs that step and its downstream, while a step that
    binds the value through with: or reads it at runtime is reused until
    --from <step> --force re-does it.

  • A process input can bind itself to one value for the life of a run.
    on_change: new_run on an inputs: entry records the value in the scope’s
    .state/input-bindings.yaml (metaproc:InputBindings/0.1) when the scope is first
    entered, and every later entry compares against it before writing anything: a
    run-process resume, a composite child scope after with: resolution, run-step,
    and run-parallel. A launch that resolves another value is refused, naming the input,
    both values, and the two ways out: the recorded value or a new RUN_ID. A renamed
    param alias is not a change.
    on_change: record, the default, is the behavior above: record and continue.
    new_run needs a param-backed input.

  • One nested step or mapped item can be re-run through its parent.
    --only <root-step> --only-scope-step <path> enters a composite child by its exact
    scope path, each composite step and mapped item key in turn (such as
    depth/ACME/fetch), verifies that omitted ancestors completed, and re-runs the
    selected step and its descendants; other items, pending ones included, stay untouched.
    Repeat the flag for several item keys under one mapped composite, run as one bounded
    batch under --max-concurrency. A selector that names a mapped item itself runs its
    whole child and can complete that item.
    --scope-through-target runs the target and its prerequisites in their scope and
    stops before anything downstream, which prepares a stage without starting its paid
    successor. A missing, duplicate, terminal, or differently shaped selection fails rather
    than widening the run.
    process-status.yaml records selected_scope_step (and selected_scope_steps for a
    batch) with selected_scope_mode; metaproc status shows SCOPED and a
    Selected scope: line, and completion checks do not certify the full run until an
    unrestricted evaluation clears the marker.
    The selectors cannot be combined with --from, --force, --skip, --cloud, or
    --dry-run.

  • Every executable leaf of a local run shares one adaptive controller. Mapped agent
    steps in a local run-process now submit their items to the run-owned RunPool that
    scalar agent leaves already used, instead of opening a pool per step, and code leaves
    (scalar and mapped mode: code) take permits from the same controller.
    Memory and provider pressure therefore throttle all of them, and concurrent mapped
    steps share one --max-concurrency ceiling, with for_each.max_concurrency still
    bounding each step. Pool state and events are written once per run, to
    .state/runpool-status.yaml and .logs/runpool/, and pool status, rollup,
    health, and events report code leaves (Code leaves:, code_active=). A
    run-owned pool serves several execution profiles as lanes when each resolves the same
    pool ceiling, host-slot limit, estimated_process_rss_bytes, and
    initial_memory_budget_fraction; pool status lists every lane, and a profile that
    differs fails before launch with an error naming each differing value.
    v0.4.1 refused any second profile on the run-owned pool; see Compatibility for what
    extending the pool to mapped steps means for mixed-profile runs.

  • Zstandard artifacts, read everywhere and written on request. Every reader of
    .jsonl.gz also reads .jsonl.zst: metaproc tail, status, stats, trace, the
    operations summary, and the metaproc.io helpers.
    metaproc gzip-text --codec zstd writes Zstandard and records the decoded size.
    Gzip stays the write default, and a run directory holding both is ordinary.
    Metabrowser cannot read .zst yet; see Compatibility.

  • Gemini steps run in a private Gemini home. v0.4.1 handed native settings to
    gemini-cli through GEMINI_CLI_SYSTEM_SETTINGS_PATH, and gemini-cli 0.60.0 and later
    skip a system settings file that is not root-owned and closed to group and others,
    with only a Security Warning transcript line, so a session on such a CLI ran without
    the session-retention guard, dynamic model configuration, the step’s tool
    restrictions, or the subagent overrides.
    A new `n...

Read more

v0.4.1

Choose a tag to compare

@jlevy jlevy released this 10 Sep 19:51
36ad4fa

Metaproc v0.4.1

A patch release about being told the truth by your own run.

Two kinds of evidence were being thrown away.
A step that named a model it wanted could be answered by a different one, because an
unrecognized name warned and ran the adapter’s default instead.
And a run pool that emitted its controller ceilings, health samples, and quota pauses
had a typed reader that dropped them, so the record on disk was richer than anything a
consumer could read back.

Both are fixed here, along with cleanup and admission accounting that could be skipped
when bookkeeping failed.
No public CLI surface, process-file field, plugin entry point, or adapter default
changed, and no previously accepted model identifier was removed.

One behavior change needs a look before upgrading: an unknown model name is now an error
rather than a silent substitution.
See Compatibility.

What’s Changed

Fixes

  • An explicit model selection is preserved, or refused — never quietly replaced. In
    v0.4.0 every adapter validated the configured model against its own allowlist and, on
    a miss, logged a warning and passed its default instead.
    A step pinned to a model ran on a different one, and the run’s cost, latency, and
    quality were attributed to a name that never served it.
    A warning in a log is not a signal an unattended fan-out reads.
    All four adapters — claude-code-cli, codex-cli, gemini-cli, and pi-cli — now
    route selection through one reviewed catalog.
    An omitted selection still takes the adapter default; an explicit one is either passed
    through exactly as written or rejected with an error naming the adapter and the
    identifier. The default is never substituted for something a spec actually asked for.
    metaproc auth check reports such a rejection as a failed check line rather than
    aborting the probe, and its --assert-model help no longer describes a
    warn-and-default path that no longer exists.

  • RunPool controller, health, and quota evidence survives the typed reader. The pool
    event log already recorded the concurrency controller’s memory, provider, and operator
    ceilings, its effective target and current bottleneck, active RSS and log-byte totals,
    periodic health_sample records, and the quota_pause_started / quota_pause_tick /
    quota_pause_resumed sequence.
    The typed event models carried none of it: the extra fields were narrowed away on
    read, and the health and quota-pause records had no model at all.
    A consumer reading the typed stream saw a thinner run than the one on disk, and could
    not tell a pool waiting on a provider quota from a pool that had simply gone quiet.
    The models now retain every field the logger emits, and the three quota-pause records
    and health_sample parse as themselves.

  • Cleanup and admission accounting survive a bookkeeping failure. Three ways a
    failure could compound itself are closed.
    An exception while recording a started child no longer skips the cleanup obligations
    that a poll failure already honored, so admission stays held until the child is reaped
    rather than being released under a process still running.
    A failure writing the host_slot_acquired event now releases the lease the caller
    never received, instead of leaking a slot for the rest of the run.
    And when shutdown fails while the pool is already unwinding a caller’s exception, the
    original error is raised with the shutdown failure attached as a note rather than
    being replaced by it — the second error is the one you can see, and the first is the
    one that explains the run.
    Quota-pause timers are cancelled and drained during shutdown, and the event and health
    loggers are closed even if the shutdown record fails to write.

Added

  • A reviewed model catalog with dated evidence. metaproc.config.model_catalog
    centralizes accepted identifiers, per-adapter defaults, the source URLs each entry was
    reviewed against, a review date and interval, and lifecycle notes recording retirement
    dates and account-dependent routes.
    Nothing was removed from any adapter’s accepted set and all four defaults are
    unchanged (opus, gpt-5.5, gemini-3.1-pro-preview-customtools, and sonnet).
    Newly accepted:

    • claude-code-cli gains claude-fable-5-1, claude-fable-5, claude-opus-5,
      claude-sonnet-5, claude-opus-4-8, and the fable alias.
    • codex-cli gains gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna,
      gpt-5.6, and gpt-5.3-codex-spark.
    • gemini-cli gains gemini-3.5-flash-lite.
    • pi-cli gains claude-opus-5, claude-sonnet-5, claude-fable-5,
      claude-opus-4-8, gpt-6-astra, the three GPT-5.6 variants, and Gemini 3.8, 3.7,
      3.6 Flash and 3.5 Flash-Lite.

    claude-fable-5-1 is accepted by claude-code-cli only: it is absent from the pinned
    Pi 0.84.2 native catalog, and Claude Code needs a newer client than this release’s
    2.1.234 pin. Acceptance means Metaproc preserves an identifier, not that an account,
    client, or API can serve it.

  • Pi model acceptance derives from the packaged provider catalog. pi-cli accepted
    identifiers now come from pi-models.default.json itself, plus a small retained set
    of native selections, so adding a provider entry no longer requires a second,
    separately maintained allowlist edit that could silently disagree with it.
    Vertex MaaS entries resolve under both their publisher-qualified and short IDs, as
    Pi’s own lookup does.

  • Four more Codex reasoning-effort values. none, xhigh, max, and ultra are
    the remaining named values in the pinned codex-cli 0.147.0 parser and are now accepted
    alongside minimal, low, medium, and high. Accepting the syntax does not mean
    every model supports every effort.

  • Pressure checks carry their hold hysteresis. Every pressure_check record now
    includes consecutive_normal and consecutive_elevated, so a reader can tell how
    close the controller is to changing its mind rather than inferring it from a run of
    samples. See Compatibility.

  • Refreshed provider data. pi-models.default.json adds the Gemini Vertex 3.8, 3.7,
    3.6 Flash and 3.5 Flash-Lite entries and the GPT-6 Astra and GPT-5.6 Sol, Terra, and
    Luna entries, the latter using the per-model openai-responses override the pinned Pi
    client supports; existing entries keep the provider’s Chat Completions default.
    Vertex Gemini entries that accept image input are marked as such, and the
    gemini-3.1-pro-preview context window is corrected to 1,048,576 tokens.
    Entries whose exact identity this review could not establish in primary documentation
    — the Moonshot Kimi ID among them — say so where a reader will find it rather than
    implying a verification that did not happen.
    pricing.md is refreshed against the same sources.

Guidelines and content

  • The Agent Skill’s debugging guidance is reversed where it was wrong. v0.4.0’s
    skill told an agent never to inspect a run with ad hoc shell commands over the run
    directory, .state/, or .logs/, and to treat any such need as a missing Metaproc
    command. That is right for run state and lifecycle and wrong for debugging: some
    captures merge stdout and stderr or filter native events, so a Metaproc rollup or an
    extracted trace can omit evidence that only the captured log holds.
    The skill now keeps the strong rule for state and lifecycle — use status, wait,
    tail, pool, trace, stats, auth usage, gcp, and do not wrap them in a
    private orchestrator — while directing an agent to read the relevant attempt’s
    captured output and native transcript directly when diagnosing a failure, to report
    the cause with its evidence location, and to say what is missing when the logs do not
    contain it. Filing a missing diagnostic as an issue is now something you do while
    continuing the investigation, not instead of it.
    The committed .agents/ and .claude/ skill copies are regenerated to match.

Documentation

All shipped documents are metaproc help topics, so these change the installed package.

  • A new operator section for each question the framework kept answering privately.
    metaproc help operator gains Direct Agent Debugging (which captured paths hold
    what, and when a rollup is not enough); Two Concurrent Runs on One Host (they
    share a disk-backed slot gate, each brings its own limit, admission fails open, and
    METAPROC_HOST_MAX_LOCAL_AGENTS composes by minimum, so it can only lower a cap,
    never raise one); Reading Pool Health (available memory is the budget, swap growth
    is the degradation signal, elevated with work pending is back-pressure working as
    intended, and the swap term lags on macOS because the compressor goes unread); The
    Subprocess Count Is a Display Estimate
    , which is not the number the governor acts
    on; and A Green Run Is Not a Reviewed Run, since status --check completed passes
    on a run that produced nothing.

  • The cause-preservation invariant is stated where it binds. metaproc help design
    gains §13 “Cause Preservation at Aggregation Boundaries”: no aggregation boundary may
    discard a cause it received.
    A step summarizing items, a run summarizing steps, a serializer omitting a field, and
    a typed reader ignoring an undeclared key are all aggregation boundaries, each free to
    summarize but not to summarize a cause down to nothing.
    Three current gaps are named so they are not rediscovered as novel: the untyped step
    entry in process-status.yaml that records state: failed with no error for every
    step that is not a scalar code step; admission, which logs a slot taken but never one
    waited for or denied; and the runpool reader gap this release closes.

  • The fan-in manifest’s emitted field set is documented. §11 described four per-item
    fields; the projection of each OutputFailure actually carries at most output,
    `...

Read more

v0.4.0

Choose a tag to compare

@jlevy jlevy released this 09 Sep 07:29
2a0aade

A mode: composite step can now declare for_each, so a process runs a whole child
process per item rather than a single step per item.
Underneath that, local execution collapsed onto one shared thing: one execution context
owning one leaf ceiling, one executor, one adaptive RunPool, and one credential-pool
lifecycle, whether the work is a scalar step, a fan-out item, or a composite descendant
several scopes down.

Around that, this release makes local Gemini fan-out safe to run, gives the browser and
status views the run’s actual runtime facts instead of authored ones, hardens GCP
Batch dispatch, and ships Metaproc’s full documentation set inside the wheel.

It also removes public CLI surface.
Read the Compatibility section before upgrading.

What’s Changed

Features

  • Mapped composite scopes: a mode: composite step may declare for_each and run
    one child process scope per item, in-process, under <run>/<step>/<item-key>/. Each
    parent item records durable status, attempt history, validated outputs, and a result,
    and each scope binds one canonical identity to its run directory, template variables,
    task state, and child event stream, so a fixed child output path stays isolated per
    item. Scope evaluation defaults to a bounded concurrency of 32 and waits for siblings
    to finish after an ordinary item exception; cancellation terminally records the
    affected parent attempt.
    A scope is a DAG rather than a step, so one step failing blocks only that failure’s
    transitive dependents and independent branches in the same item still run.
    The first implementation is single-host: gcp-worker partitioning and whole-scope
    for_each.retry are rejected before any DAG step or cloud dispatch starts, while
    retries remain available on child leaves.

  • One execution context across recursive scopes: local run-process execution
    shares one executable-leaf ceiling across fan-out pools, scalar steps, code work, and
    composite descendants, and lazily owns one adaptive RunPool that every scalar agent
    leaf submits to, so pressure response, process-tree supervision, status, and events
    are the same wherever a leaf runs.
    Synchronous handlers and code commands run on the run-owned executor rather than
    blocking the event loop, command-backed code steps at the same DAG level may now run
    concurrently, and --force reaches composite descendants while root step selectors
    stay root-scoped. Because commands share the process directory, authored steps that
    mutate shared files, repositories, or lockfiles must declare per-item paths or provide
    their own synchronization.

  • One credential-pool lifecycle for scalar and fan-out agents: non-fan-out agent
    steps now lease the configured pool label, apply the same credential scope and scrub
    rules, classify failures, walk fallback labels on retry, and emit the same
    auth_lease_acquired and auth_outcome evidence as RunPool items.
    Nested leaves bind slots and event join keys to their path-relative child scope, so
    credential material stays inside the logical run tree even when a run directory is
    symlinked to another volume.
    An adapter mismatch emits an explicit warning, log record, and auth_skipped event
    before falling back to ambient authentication, including on worker entrypoints.

  • Views built on runtime facts: VizModel now carries public process outputs, the
    complete process-input and default declarations, and the resource, failure,
    execution-profile, namespace, and fan-out fields a resolved plan carries, and the
    Metabrowser process and step panels render them.
    Given a run directory, the same model includes a rebuildable view of scalar, mapped,
    and nested task records: a result is consumable only when it names the latest
    successful attempt, matches the current step fingerprint and declared output port, and
    resolves to an available artifact of the declared kind.
    Historical unbound, stale, missing, undeclared, and external outputs remain visible as
    diagnostics, and hydrated runs rebase portable paths to the local run root without
    writing an additional ledger for the projection itself.
    Every run scope does now carry a run-plan.yaml; see “New and changed runtime
    artifacts” in Compatibility.

  • A refused launch reports every class of problem at once: run-process,
    run-step, plan, and deps collect unresolved template placeholders, unset
    operator parameters, missing input files, and unusable execution profiles together and
    report them as one grouped message.
    Process-input errors are tagged as a parameter or a file, because “pass a --var” and
    “produce a file” are different fixes.
    Previously each launch surfaced exactly one class, so a cohort with four problems took
    four failed launches, and --dry-run kept passing an invocation that was wrong in
    three other ways.

  • GCP Batch dispatch: metaproc gcp run stages immutable source artifacts: it
    validates artifact identity before packaging and creates wheel and workspace objects
    only when their GCS names are unused, so a later dispatch cannot replace bytes an
    existing job references.
    Re-running a dispatch that failed after uploading still works: an object with an
    identical digest is accepted as already uploaded, while one with different content
    fails by name with the instruction to pick a new --job-name. Batch job
    specifications now carry Secret Manager references rather than plaintext-expanded
    secret_variables, require an explicit runtime service account once a job binds any
    secret reference, reject registered credentials passed through --env or plugin
    bootstrap variables, and retry transient startup failures.
    Storage posture is explicit: the default Filestore placement is refused when
    METAPROC_GCP_FILESTORE_SERVER is unset, before anything is uploaded or dispatched.

  • Gemini 3.7 and 3.8 Flash: gemini-3.7-flash and gemini-3.8-flash join the
    validated model set, with a gemini-flash-38 execution profile beside the existing
    3.6 one. The 3.6 profile still pins gemini-3.6-flash, so a name that says 36 keeps
    meaning 3.6, but its lane sizing changed along with every other gemini-cli profile:
    see “Gemini lanes are sized from measured memory” in Compatibility.
    Agent CLI contracts advance to Claude Code 2.1.234, Codex 0.147.0, Gemini CLI 0.55.1,
    and Pi 0.84.2.

Fixes

  • Gemini no longer scans its whole project history at startup. Gemini CLI runs
    session-retention cleanup as an un-awaited background task during startup, and before
    it knows which sessions are expired it enumerates the project’s saved conversations
    and parses every one through a single unbounded Promise.all. On an accumulated
    project bucket that turned a roughly 0.4 GB process tree into 5 GB or more for the
    same short prompt, enough to destabilize a host running several agents at once.
    The adapter now ships general.sessionRetention.enabled: false, which makes cleanup
    return before it enumerates anything.
    This is a Gemini-specific guard, and the evidence for it is Gemini-specific too.
    Matched probes of Claude Code, Codex CLI, and Pi under the same conditions found no
    comparable startup cost, and stored history did not predict one: Claude Code carried
    roughly 1.5 GB of project history and Codex roughly 6.4 GB active plus 10 GB archived,
    yet neither read its full history at startup.
    No equivalent setting ships for the other adapters, because no equivalent problem has
    been measured in them.
    See Compatibility: this also disables Gemini’s own pruning of chats/ and
    tool-outputs/.

  • A Gemini step is answered by the model it asked for. gemini-cli rewrites any model
    id ending in flash that it does not recognize to its own default before the request
    leaves the process, and under Vertex authentication that default is
    gemini-3.5-flash, so a step pinned to gemini-3.6-flash was answered by 3.5 with no
    error, no warning, and a stats.models naming the substitute.
    The adapter now asserts gemini-cli’s dynamic model-resolution setting beneath its
    native-settings defaults, which passes an unrecognized id through untouched, and a
    successful result whose terminal stats.models omits the requested model is rejected
    for both scalar and fan-out agent steps, before valid-output rescue.
    This is upstream google-gemini/gemini-cli#28859, still unfixed at 0.55.1 and later.

  • A Gemini step can read files your ignore rules exclude. The adapter now ships
    context.fileFiltering.respectGitIgnore: false in its Gemini native settings, so the
    agent’s own file tools read ignored files anywhere in the workspace rather than only
    declared runtime inputs.
    This makes a step see the working tree an operator sees, which is what the surrounding
    fixes assume, but it widens what a Gemini step can reach: treat the workspace,
    including files such as .env, as readable by a Gemini step
    , and keep material you
    do not want an agent to read out of the process directory rather than relying on an
    ignore rule. This applies to gemini-cli steps only.
    An operator who wants the previous behavior can set the key back through
    native_settings, at the cost of the ignored-file reads the adapter now expects.

  • A Gemini step runs where its configuration says. working_directory is now an
    accepted Gemini config key and is returned to the launcher instead of None, so
    per-run isolation is real rather than an inherited repository root.
    Gemini prompts are also streamed through stdin while the durable audit prompt is
    retained, so a workspace ignore rule can no longer block or distort prompt delivery.

  • An agent’s own verdict outranks its exit code. Some adapters write every declared
    output, emit a terminal success record, and then exit nonzero while shutting down.
    A scalar agent step whose agent reported success and whose declared outputs ...

Read more

v0.3.0

Choose a tag to compare

@jlevy jlevy released this 25 Aug 02:51
4d518d8

Metaproc now records what each task attempt actually did, as a durable fact written
before the work starts and finalized once when it ends.
Replay reads that history instead of inferring it, and a crashed process reconciles
against it on restart.
Around that, this release adds item-aligned chains and fan-in to process specs, gives
agent steps their validation failures back as retry feedback, and hardens GCP Batch
dispatch.

What’s Changed

Features

  • Durable per-attempt task history: Work managed by run-process, run-parallel,
    or a waited run-step writes a typed metaproc:TaskAttemptRecord/0.1 before
    execution and finalizes it once with its disposition and failure class.
    Replay consumes that exact history when present, and startup reconciliation closes
    attempts orphaned by a crash and rebuilds a missing terminal projection without
    disturbing work owned by a live step-scoped pool.
    Attempt success waits for every attempt-owned validator, including the fan-out
    write-boundary check, and a resume refuses status or history addressed to another run,
    step, or item.
  • Item-aligned chains, fan-in, and declared retry: Process specs can chain steps
    against the same fan-out item, collect fan-out results into a typed fan-in outcome,
    and declare retry policy in the spec.
    A resume enters a chain even when its head is already complete, rerunning incomplete
    tasks and reusing completed ones.
  • Retries that say what was wrong: An agent step that produced invalid output gets
    the structured validation failures appended to its next prompt — output, failure kind,
    path, contract, invariant, location, and message — instead of being asked again with
    no new information. Transport failures never create or replace that feedback.
  • Schema conform for agent-authored YAML: A frontmatter scalar that YAML would
    resolve to the wrong type is requoted against the contract about to judge it, so a
    value genuinely named 1850 survives a type: string field.
    Only pydantic string_type errors are acted on and the document’s own serializer
    rewrites it, so 1.10, 007, 1e3, and 0x1F keep their written form.
    A real type disagreement still fails.
  • Host admission for single-step runs: A run-process invocation with no for_each
    is admitted through the same host gate as fan-out work, so several orchestrators on
    one machine account for each other instead of launching blind.
    Admission is best-effort by design: an unreachable gate or a wait timeout lets the
    launch proceed rather than failing a step that worked before admission existed.
  • GCP Batch dispatch: metaproc gcp run accepts repeatable
    --workspace-package PATH to install current-branch consumer packages, prints Batch
    state transitions while provisioning and executing, and emits an exact resource
    identity that gcp status, gcp logs, and gcp cancel reattach to.
    Default workspace archives exclude top-level and vendored Metaproc source layouts, and
    safe in-repository symlinks are materialized as archive content while external links
    and directory-link cycles are rejected.

Fixes

  • Retry classification no longer depends on a filename: Whether to retry a missing
    output or give up on a structural mismatch was recovered by substring-matching a
    rendered error sentence that contained the artifact’s name, so two outputs missing for
    the same transient reason could get opposite verdicts because one was named for a
    schema manifest. The decision now reads the structured failure kind.
  • Representation-only validation failures: date, datetime, time, Decimal,
    and UUID values normalize to their serialized form before the structural pass, so a
    quoted and an unquoted YAML date stop disagreeing.
    Only types with an unambiguous serialized form convert.
  • Cloud log tailing: Cloud Logging datetimes are normalized to RFC3339 before use as
    tail watermarks, fixing a tail that could fail mid-run, and a blocking generic run
    reuses and closes one logging client instead of leaking one per poll.
  • Wheel overrides preserve the image dependency closure: A verified Metaproc wheel
    override installs with --no-deps, keeping the image’s audited dependencies and
    per-package release-cutoff exceptions, and nested uv commands stay on the baked
    environment after package installation.

Compatibility

  • Code-step outputs are no longer YAML-repaired. run-parallel’s mode: code
    fan-out ran the frontmatter auto-repair pass over each item’s declared outputs before
    validating them; run-process’s code path never did.
    Both now leave the document alone.
    A process whose code handler relied on that repair will start reporting
    invalid_outputs — fix the handler’s serializer rather than the artifact.
    Repair and conform stay scoped to agent-authored output.
  • A Gemini CLI below the supported minimum is now refused, not warned about. The
    adapter passes --skip-trust, introduced in gemini-cli 0.40, so an older CLI failed
    every agent step partway through a run with an unexplained “Unknown arguments”.
    The refusal reports the version found, the path it resolved, and the remedy.
    Drift at or above the minimum stays a warning.
    If a stale CLI is shadowing the pinned binary on PATH, runs that previously started
    and then failed will now stop immediately.
  • cryptography moves to 50.0.0, retiring the audited advisory waiver for
    GHSA-g6cj-pr64-35w5 / CVE-2026-69247. No advisory waiver is active in this
    release.
  • Replay parity has one known gap. A task blocked purely on credential-pool capacity
    is recorded as running with a persisted attempt before admission is granted, and
    replay counts those lost records against max_attempts. A run that merely waited on
    capacity several times can therefore replay as failed while production considered it
    healthy. Live execution is unaffected — the pool-exhausted path does not consume the
    production retry budget — so this is a post-hoc analysis limitation, not a run-outcome
    one. Tracked as mp-ux0f and mp-f5m5.
  • Legacy run trees remain readable. Readers fall back to status-based compatibility
    when durable task facts are absent.
    The full mixed-authority boundary is still being proven out under mp-g315; if you
    depend on reading pre-0.3.0 run trees, verify against your own corpus.

Full commit history:
v0.2.1 … v0.3.0

v0.2.1

Choose a tag to compare

@jlevy jlevy released this 09 Aug 19:31
3aa4c80

Metaproc v0.2.1

What’s Changed

Features

  • Operational resource reporting: Runtime and agent evidence now feeds one
    deterministic resource ledger.
    CLI and Metabrowser views distinguish measured, estimated, and unmeasured values;
    report provider meters and coverage gaps; evaluate reporting-only budgets; and recover
    missing reports for inactive runs without calling provider APIs.
  • Code-step telemetry: run-process, run-step, and run-parallel sample CPU and
    memory for handlers and child processes.
    Step lifecycle events add wall time to the owning resource nodes, including nested and
    fan-out execution.
  • Typed identities: New runs use compact, time-ordered run-... identities.
    GCP Batch jobs retain a readable label and a collision-resistant exact identity key,
    so status, logs, cancellation, and inventory keep distinct runs separate.
  • Version discovery: metaproc --version prints the installed distribution version.

Fixes

  • Resource correctness: Terminal recovery preserves successful outcomes, historical
    reports regenerate their complete artifact set, and current resource documents use a
    registered SoftSchema contract while strict V1 and V2 readers remain available.
  • Latency attribution: Paired Claude and Gemini tool spans now carry measured
    duration. Missing or invalid timestamps stay unmeasured, and reversed timestamps do not
    produce negative latency.
  • Cloud source preflight: Consumer repositories with Metaproc in a submodule or
    vendored path are detected before cloud dispatch, preventing branch changes from
    silently falling back to image-baked code.
  • Cloud artifact guidance: Wheel and workspace URIs are documented with their
    required SHA-256 digests, and the architecture now matches the verified /opt/venv
    installation and workspace-extraction paths.

Guidelines and Content

  • Portable Agent Skill: The generated .agents and .claude skill copies use
    portable tool language, stay byte-for-byte aligned with the packaged skill, and route
    operators to the packaged manuals.
  • Documentation structure: The README and developer guide now provide
    audience-oriented navigation, architecture and file-I/O references match the public
    implementation, and completed project plans live under docs/project/specs/done/.

Compatibility

  • New default run IDs no longer embed process and title slugs.
    Set RUN_ID_TEMPLATE explicitly when an integration still requires the historical
    format. Existing underscore-form IDs and fully legacy GCP jobs remain readable.
  • New resources.json files use metaproc:ResourcesDocument/0.1. Metaproc continues to
    read strict metaproc.resources/v1 and metaproc.resources/v2 files.
  • SoftSchema is now constrained to >=0.6.0,<0.7. Integrations that call SoftSchema
    directly must use its document terminology and 0.6 API.
  • Resource budgets remain observational.
    They report outcomes but do not block dispatch, change scheduling, or terminate a run.

Full commit history:
v0.2.0 … v0.2.1

v0.2.0

Choose a tag to compare

@jlevy jlevy released this 01 Aug 04:44
c85c4e0

Metaproc v0.2.0

What’s Changed

Features

  • Standalone workflow framework: Install and run dependency-aware Markdown process
    specs with local handlers, agent CLI adapters, or optional GCP Batch execution.
  • Resumable execution: Plan, run, resume, validate, inspect, and override workflows
    with structured state, traces, resource reports, usage records, and RunPool controls.
  • Extension surfaces: Register contracts and handlers through plugins, install the
    packaged Agent Skill, and inspect artifacts with the packaged Metabrowser plugin.
  • Credential operations: Manage labeled credential pools, probes, rotation, and
    adapter selection without embedding downstream workflow policy in Metaproc.

Fixes

  • Portable frontmatter output: Mapping-based YAML and frontmatter writes are
    deterministic and alias-free.
    Repeated values are expanded, and cyclic values fail without replacing an existing
    artifact.
  • Validation outcomes: metaproc softschema validate reports SoftSchema’s valid,
    invalid, or input_error outcome alongside the existing ok field.

Compatibility

  • metaproc softschema compile requires --contract CONTRACT_ID.
  • Contract IDs must use [namespace:]Name[/version]. The structure-report contract is
    metaproc:StructureReport/v1; regenerate reports created with the former ID.
  • YAML validation accepts bounded JSON-compatible values and rejects aliases, anchors,
    merge keys, explicit tags, duplicate or non-string keys, unsafe numeric values, deep
    or oversized inputs, and implicit remote schema references.
    Date- and timestamp-shaped YAML scalars remain strings.

Guidelines and Content

  • Portable Agent Skill: metaproc skill metaproc --install installs the packaged
    Metaproc workflow guidance for supported coding agents.
  • Packaged operator guidance: The installed distribution includes command,
    environment, format, and workflow documentation used by the CLI and Agent Skill.

Full commit history:
v0.2.0 commits