v5.16.0

Changed

• #917: a rebuild that re-runs a top-level effect now says so, and 6.0.0 will
  refuse it. Resuming a stored run rebuilds its program by re-running its top
  level (a workflow resume, nodus workflow sweep, a durable coroutine). Since
  #909 that skips main() and the scheduling builtins. Any other top-level call
  that reaches outside the program happened again on every resume, silently.
  Now each one raises a StagedFlipWarning naming the call and its line, and is
  recorded for nodus check --staged when NODUS_STAGED_FLIP_REPORT is set.
  The behaviour is unchanged until 6.0.0, when it is refused (staged flip
  rebuild-effect).

  • What counts as an effect is decided by capability: writes, network
    (GET included), subprocess, env_set/env_unset, agent, tool and syscall
    calls, and memory writes. Reads (read_file, env_get, memory reads) do
    not count.
  • Host functions count unless registered with
    register_function(..., rebuild_safe=True).
  • Override for a run whose stored source predates the fix:
    runtime.sweep(rebuild_effects="allow") or nodus workflow sweep --rebuild-effects allow. "refuse" opts into the 6.0.0 behaviour early.

  Guide: docs/migration/v6.0-staged-flips.md §6.

  nodus check --staged finds them from source too (#922). It looks in any
  file that runs a workflow, a goal or a durable coroutine: at its top-level
  statements outside main, and in the same-file functions those call. It
  reports effectful builtins (by the runtime's own rule), stdlib wrappers
  (classified by reading the stdlib's source), and extern-declared host
  functions, each with its line. A run's report is merged into the same flip,
  which adds host functions called without an extern. Building it found that
  the runtime warning pointed into std:fs's own source for a top-level
  fs.write. It now names the user's line, through the same user_facing_loc
  runtime errors use, and a test holds the static and runtime positions equal.

• Durable coroutines: durable_spawn(key, fn, args), and checkpoint inside
  a function. durable_spawn runs fn(args...) as a coroutine backed by a run
  record in the workflow store. Each checkpoint "label" it reaches is written
  to the store before the coroutine goes on. If the process dies, calling
  durable_spawn again with the same key (typically by running the program
  again) resumes the run from its last checkpoint: the work before it does not
  run again. A finished key does nothing and returns the run id. The contract
  is at-least-once between checkpoints: work after the last checkpoint is
  re-run after a crash.

  fn worker(job) {
      let rows = fetch(job)
      checkpoint "fetched"
      upload(rows)
      return len(rows)
  }
  durable_spawn("ingest-42", worker, ["job-42"])
  run_loop()
  

  Requirements and refusals:

  • Needs a store whose claims are exclusive across processes:
    NODUS_WORKFLOW_STORE_BACKEND=sqlite today; the default from 6.0.0.
  • Checked at the durable_spawn call: the entry must be a named top-level
    function with no captured variables, and the arguments must be plain data.
    They are copied, and the coroutine sees the stored copy.
  • A checkpoint inside a for loop over a list works (#911): the list
    and the loop's position are saved with it, and a resume continues with the
    next element. A for over a user-defined iterator (a record with
    __next__) keeps its position in its own code, so a checkpoint there raises
    a catchable error that says so. The continuation format is now
    nodus-continuation/2, and /1 data is still read.
  • The same key with different arguments is refused: a key names one run.
  • Cancelling one from outside (#912): nodus workflow cancel --key <key>
    (or a run id; runner.cancel_run(run_id_for(key)) from Python). The run
    becomes cancelled, and neither a re-run nor a sweep resumes it. A live
    holder notices at its next checkpoint or claim renewal and unwinds the way
    cancel(task) does: finally runs, catch cannot swallow it, and a
    checkpoint inside finally does nothing. Every durable write now goes
    through the store's new update_claimed_run, which checks the claim and the
    status and writes in one transaction, and renew_claim refuses a terminal
    run. Together they close a race in which a cancel, or a takeover, landing
    between a holder's read and its save was overwritten and the run came back
    running.
  • A run held by a live process is left alone, and the caller is told so on
    stderr, with how long the claim has left. A process that loses its claim
    to another stops its copy at once and never writes to the record again.

  checkpoint changed where it is accepted. Inside any function body it now
  compiles, because whether a function runs durably is decided at run time.
  Outside a step and outside a durable coroutine it fails at run time with a
  message naming durable_spawn. At module level it is still a parse error,
  with a new message. Inside a step it is unchanged. The word is still not
  reserved.

  Deadlines survive the restart. The scheduler's clock counts from process
  start, so a deadline held in a local (runtime.time_ms(), a sleep_until
  target, std:loop) would mean something else in the process that resumes
  it. Each checkpoint now records where the coroutine's clock stood, and the
  resumed coroutine's clock continues from there plus the wall time that
  passed. The offset is per coroutine: two runs checkpointed at different
  times keep their own timelines, and ordinary coroutines are unaffected. A
  host clock that went backwards is clamped, so time never runs backwards for
  the coroutine.

  A long-lived host resumes them. nodus workflow sweep, and the server's
  background sweeper, now resume unfinished durable coroutines (report bucket
  resumed_coroutines), under the same claim as durable_spawn. So a sweep
  and a restarted program can never both resume one run. The program is
  rebuilt from the source stored with the run, and its top-level
  durable_spawn is suppressed during that rebuild. Host functions must be
  re-supplied by the sweeping host, as for a rebuilt workflow. nodus workflow resume <id> on a durable coroutine now says where it is resumed instead of
  failing as if it were a broken workflow.

  Errors point at the checkpoint. A checkpoint that fails, inside or
  outside a durable coroutine, reports the line and column of the checkpoint
  word. Before this, one pointed at its label and the for-loop refusal carried
  no position at all. Both were found writing the guide.

  This is stages 4b–4d of durable coroutines (#180). Experimental. Guide:
  docs/guide/durable-coroutines.md. OPERATOR_OR_EMBEDDER_RUNBOOK.md §6.2,
  which said only a workflow survives a crash, is rewritten. Pinned by
  tests/test_durable_spawn.py, tests/test_durable_clock.py and
  tests/test_durable_sweep.py, each including a real process kill. Known
  limit: a rebuild re-runs a top-level host call or effectful builtin outside
  main. It is reported now and refused at 6.0.0 (see the rebuild-effect entry
  below).

• checkpoint "label" where it can record nothing now fails with an error that
  names the rule. It used to fail at run time with Undefined variable: checkpoint. That told you neither where checkpoint works nor what to do
  instead. At module level it is now a compile-time error, reported by nodus check too. Inside a function it compiles (see the durable-coroutines entry
  above: whether a function runs durably is decided at run time), and outside a
  step and outside a durable coroutine it fails at run time. Both messages point
  at durable_spawn and at running the work as a workflow step.

  This was stage 1 of #180.

  checkpoint is still not a reserved word. Only the identifier
  checkpoint followed immediately by a string literal, at the start of a
  statement, is treated as the statement. That shape can never mean anything
  else: it is a variable reference followed by a discarded string. let checkpoint = ...,
  checkpoint("x"), a record field or parameter named checkpoint, and the name
  and a string on separate lines all still work.

  One program that ran before is now refused: let checkpoint = 1i; checkpoint "x" on one line. It evaluated a variable and threw away a string, so nothing
  it did is lost. Pinned by tests/test_checkpoint_outside_step.py, which also
  checks that every allowed use of the name still runs.

• A coroutine's continuation can be checked for durability, with reasons.
  nodus.runtime.durability.assess_coroutine(co) reads a live coroutine and
  lists everything that would stop it being written down and resumed in another
  process, each with where it is: frame 0 (`holds`), local `ch`: a channel, whose other end does not survive the process. Nothing is persisted and the
  coroutine is not changed. This was stage 2 of durable coroutines (#180); it
  is the reasoning a durable checkpoint refuses with, and it is provisional
  Python API, not yet a promise to embedders.

  It found something the design had wrong. Inside a for body the operand
  stack is not empty: the loop's iterator sits there for the whole body and
  keeps its position in host state. The design assumed an empty stack at every
  statement boundary, and its premises test never measured a loop. That is now
  corrected in docs/design/v5/08-durable-coroutines.md §3.2. Pinned by
  tests/test_durability_predicate.py.

• A suspended coroutine can be written down as data and resumed elsewhere.
  nodus.runtime.continuation.capture_continuation(co) turns a suspended
  coroutine into JSON-safe data, and restore_continuation(vm, data) rebuilds it
  against a VM running the same program, in this process or another. The rebuilt
  coroutine continues from where it stopped, and the work before that point does
  not run again. Shared and cyclic containers keep their shape. Capture refuses
  anything the durability check above reports, and restore refuses a different
  program rather than jumping into unrelated code. This was stage 3 of durable
  coroutines (#180); durable_spawn above is built on it. Provisional Python API; the
  data format is versioned and not yet a stable contract. Pinned by
  tests/test_continuation_roundtrip.py, including a round trip between two
  processes.

  Only restore data your host wrote. A continuation's saved positions are
  jump targets in the program. Restore refuses a different program, a function
  that has moved, and any frame naming a workflow step body (the #394 guard
  applies to rebuilt frames as it does to calls). It cannot tell a tampered
  position from a genuine one.

• A workflow store now declares whether its claims are exclusive across
  processes. WorkflowStore.EXCLUSIVE_CLAIMS_ACROSS_PROCESSES is True for
  SQLiteWorkflowStore and False for LocalWorkflowStore, whose claim is
  guarded by an in-process lock only. require_exclusive_claims(store, feature=...) refuses a store without the property and names the fix. Durable
  coroutines require it, so that a restarted program and a resuming host
  cannot both resume one run. A third-party store says False unless it opts
  in. Each built-in store's declaration is checked against an interleaved race
  in tests/test_store_claim_exclusivity.py. This was stage 4a of durable
  coroutines (#180).

  Both stores write each record atomically, so this is not a crash-safety
  distinction, despite how the JSON store has been described.

  Run kinds (workflow, goal, and coroutine) are now named once,
  in nodus.orchestration.workflow_state, and no other site may compare against
  a literal kind. The resume path used to treat any kind that is not goal as a
  workflow. A run of another kind is now refused by name, and the rehydrate sweep
  skips it rather than writing a rebuild error onto it.

Fixes

• #921: an error raised by a call points at the call. A call is compiled after
  its arguments, and the position it was reported at was the last argument's:
  len( 42i ) said column 18, where len is at column 9. A frame's call
  site is the same position, so every "called from" line in a stack trace was off
  the same way. A no-argument call reported its ( instead. Now each call points
  at the start of what is called: len in len(...), fs in fs.read(...),
  xs in xs[0](...). A print statement points at print. Columns in
  error messages and stack traces change wherever an error was raised by a
  call, so anything that parses them will see new numbers. Lines are unchanged.
  The documented outputs that quoted the old columns were re-run and updated.

• #920: nodus workflow resume / replay and goal-resume exit 1 when they
  fail. These commands print the workflow runner's answer, and a run that does
  not exist, could not be rebuilt, or is not a workflow comes back as
  {"ok": false, ...}. The CLI exited by whether the command executed, so it
  printed the failure and returned 0, and a script or cron job checking the exit
  code was told it worked. The JSON is still printed either way. One helper now
  serves every command of that shape, including workflow replay, which the
  issue did not name.

• #909: a rebuild no longer re-runs main() or restarts spawned work. A
  workflow resume, a sweep, or a durable coroutine's resume rebuilds the program
  by re-running its stored source, only to re-bind definitions. Two more things
  are now skipped during that re-run:

  • the scheduling builtins (spawn, spawn_after, run_loop, wait).
    spawn returns the coroutine unscheduled, so the rest of the top level can
    still pass it around;
  • a top-level call to main(). The loader already did not auto-run main
    during a rebuild, but a program that called it itself ran its work again.
    It is skipped at run time, not compile time, so a durable coroutine's
    program fingerprint still matches.

  A top-level host-function call or effectful builtin outside main still runs
  again, because it may compute a value a definition needs. Put a program's work
  in fn main(). Refusing those calls instead is proposed in #917.

• #908: an embedder can sweep on VMs configured like its own runtime.
  NodusRuntime.sweep(min_idle_ms=..., now_ms=...) runs the workflow sweep
  (expired waits, due retries, released schedules, orphaned workflow runs,
  durable coroutines) on VMs from the new NodusRuntime.new_vm(). Each one is
  built the way run_source builds its VM, through one shared builder: the
  runtime's registered host functions, its confinement flags, capability policy
  and approval channel, agents, memory store and limits. Resumed programs'
  output is collected into the report's stdout/stderr.

  The documented alternative was a bare VM([], {}). That missed the host's
  functions, so resumed code failed with Undefined function. Worse, it is
  permissive by default, so a program an embedded runtime would refuse (reading
  the environment, a subprocess, the network) ran unconfined once it was
  resumed. Both are pinned by tests/test_runtime_sweep_vm.py, which keeps the
  bare-VM escape visible as a contrast. The unused _install_host_functions is
  gone.

• #907: nodus run shows a program's output as it is written. It used to
  collect stdout until the program ended (the buffer that enforces
  --output-limit). A long-running program showed no progress, a killed run lost
  everything it had printed, and stdout landed after stderr, so an error appeared
  before lines printed earlier. Output now streams through a writer that counts
  against the same limit (RuntimeLimitExceeded past it, with what fit already
  shown) and flushes at every newline, so the order also survives a pipe. Only
  nodus run streams. An embedded NodusRuntime and nodus serve still return
  stdout as a value, and --dump-bytecode keeps its disassembly first.

Performance

• #927: the VM's dispatch loop is ~1.6x faster on CPython for compiler-like
  code, ~2x on a tight loop. Measured head to head against main, alternating,
  best of four, each run a fresh process:

  workload	before	after
  examples/expr_compiler.nd, 400 expressions	719K instr/s	1,135K instr/s	1.58x
  hot arithmetic loop	967K instr/s	1,909K instr/s	1.97x

  PyPy is unchanged to slightly faster (1.04x and 1.22x). Before each
  instruction did any work, the loop asked whether a debugger, profiler or trace
  was attached, then called record_instruction, which checked the scheduler's
  slice, the deadline, max_steps, the memory bound and the batch event. Now:

  • a separate loop handles a debugger, profiler or trace;
  • otherwise the loop counts the instruction and runs the full check only when
    the next one could fire.

  Every check still fires on the same instruction it did before. A test runs one
  program through both loops and requires every check at the same count, and
  another compares the check-point arithmetic against a brute-force model.
  Moving any one check point a single instruction late turns a test red.

  Two smaller costs went too:

  • a call no longer computes its function's display name unless an event sink
    or the profiler reads it;
  • pop no longer tests the stack's length before popping.

  Found by profiling for LIMITS-001 (#173), whose native-loop direction this
  defers.

Tooling

• tools/benchmark_runtime.py measures the compiler workload too. Besides the
  hot loop and startup, it now drives examples/expr_compiler.nd over 25 to 1,600
  expressions and checks the answer, so the throughput question LIMITS-001 (#173)
  asks can be re-derived with one command. A tight loop is a JIT's best case, and
  the two workloads had been confused on that issue more than once.

  Re-measured with it on 2026-11-11, the throughput figures in the README,
  LANGUAGE_VISION.md, llms.txt, llms-full.txt and the ecosystem readiness
  assessment were stale: CPython is on the order of 1M instr/sec (they said 200K
  or 400K), and PyPy is ~16x on the compiler workload, not ~4-5x -- that figure
  was taken during the PyPy-only regression fixed in #702. They now give the
  order of magnitude and point at the tool. The same passages listed closure
  mutation and module privacy as missing; both work (#671, export { ... }),
  and only string slicing remains.