Skip to content

plimsoll v0.18.0

Choose a tag to compare

@carrollco carrollco released this 04 Oct 12:22
· 13 commits to main since this release

Docker guests now run as a uid no host account uses, E2B and Docker Cloud runs can draw on a daily allowance of microVM seconds, a signed bundle of run records now proves it is the whole file its harness wrote, and the efficiency advisor hands a caller only a batch route the operator has declared. Docker and OpenShell images must be rebuilt, and a bundle written by an earlier release cannot be verified or continued by this one.

Breaking changes

  • Docker guests run as uid 61000 by default, and the daemon refuses a uid the host already has. Before, every guest process ran as uid 1000. Under runc (Docker's default runtime) without user-namespace remapping, a container's uid is the host's, and 1000 is the first account most Linux machines create, often the operator's, so a container escape acted as that account. SANDBOX_GUEST_UID now sets the guest's uid and gid (default 61000), and the session identity check runs as that uid plus one (61001 by default; it was 2000). 61000 is below 65,536, so it still starts under rootless Docker and userns-remap, which map 65,536 uids, and it lies in 60706 to 61183, a range systemd leaves unallocated. Preflight (the check the daemon runs at startup and on every /readyz) refuses a guest uid, or that uid plus one, that the host's /etc/passwd or /etc/group contains; accounts from a directory service such as LDAP are not in those files, so choose the uid with them in mind. The setting accepts 1 to 65532 (the uid and the uid plus one stay below nobody, 65534) outside the bands systemd gives to accounts it never writes to /etc/passwd, and setting it for any provider other than docker fails startup. The startup smoke test checks in /proc/self/status that every uid, gid and supplementary group of a guest process is the guest uid. Inside the guest the uid has no account, so HOME is / unless the image adds a passwd line for it, a file only node can read is unreadable to a run, and os.userInfo() (Python's getpass.getuser()) fails: give package files to everyone (chmod -R a+rX), or add a passwd line if a tool needs a user name. docs/guest-dependencies.md, SECURITY.md.
  • Docker and OpenShell images must be rebuilt. The project runner (/runner.mjs, the program inside the image that writes a project's files and runs its steps) now receives the image's environment from plimsoll's plan and hands it to the steps, and its report carries version 3 of its marker. An image with an older runner would drop that environment, so it fails the startup smoke test and the daemon does not serve. The runner also kills a step that outlives its budget with SIGKILL (see Docker). Run make docker-images, and rebuild an OpenShell image on the new base image; an OpenShell image must now also carry /usr/bin/env, which plimsoll's own programs start under.
  • Docker ignores an image's ENTRYPOINT, and the image's environment reaches only guest processes. Every container command is now plimsoll's own. plimsoll's programs in the container (the runner, the smoke probes, a session's main process, the sweep, the process lister, the identity check, the relays and the interpreter launcher) start under /usr/bin/env -i with a fixed PATH; a snippet's node, a project's steps and a cell's interpreter get the environment of the verified image's configuration, handed on explicitly. Before, every process started with the image's ENV, and a list of refused variables was all that kept a variable such as NODE_OPTIONS from loading guest-written code into plimsoll's sweep. That list is gone: variables it refused, such as NODE_OPTIONS, NODE_DEBUG, or a PYTHONPATH entry inside a writable directory, are accepted now and reach guest processes only, where they can load only the guest's own code. Preflight still refuses an image that sets a variable the C library or dynamic loader reads before any program runs: any LD_ name, GLIBC_TUNABLES, LOCPATH, GCONV_PATH, and now NLSPATH, which v0.17.6 accepted. An entry that is not NAME=value is refused too.
  • A docker image tag re-pointed after startup is refused until the daemon restarts. The startup smoke test records the image IDs it proved, and a run launches only those. A tag moved to new content later (for example by make docker-images on the same host) fails Preflight, so /readyz, and runs, session opens and pool refills on it are refused before any code runs (reason environment) until a restart proves the new image. Before, the new image was launched within seconds with its lockdown, runner guard and languages unproven. After rebuilding images on a host that serves, restart plimsolld.
  • A bundle written by v0.17.6 or earlier cannot be verified or continued. A bundle (the JSON Lines file of signed run records that plimsoll-attest and the attest package keep) now carries a signed link on every line and must end with a checkpoint (see Run records and attestation). A file without them fails verify, and run refuses to append to it: start a new bundle, and keep the release that wrote an old one to verify it. Version 1 run records are refused, since no bundle with links can hold one. A Go program that uses attest.Harness directly must call Harness.Checkpoint when it finishes writing, or its bundle fails verification as incomplete.
  • Advice reaches a caller only for a batch route the operator declared. With advice: caller, v0.17.6 returned a finding to the calling agent whenever the profile granted the per-item route's collection (GET /items for GET /items/*), on the strength of the path alone. Now the profile must declare the relation in batch_of ({"GET /items": ["GET /items/*"]}, which plimsoll-specgen writes from an x-plimsoll-batch-of marker in the OpenAPI document) and grant the batch route; a path guess stays on the operator's side as a candidate. A profile that relied on the guess returns no caller advice until it declares batch_of. The audit line's grant_route now means a declared batch route the profile does not grant. docs/efficiency-advisor.md.
  • Hardened mode with E2B or Docker Cloud requires a daily allowance on every caller. With PLIMSOLL_HARDENED=1 and a provider billed by the second, the daemon refuses to start, before its smoke test creates a billed microVM, unless every caller in PLIMSOLL_CLIENTS_FILE has a paid_seconds_per_day (see Billing and spend). Outside hardened mode no allowance is set unless you set one. docs/hardened-mode.md.
  • Go embedders that implement a session provider or their own admitter have API changes. sandbox.Session gains Environments(), the environment the opened session actually runs, which the daemon checks the caller's software rule against. sandbox.RefuseCell takes the call's context. A session's Done now closes only once its sandbox is deleted, and its Err is set as it ends. An admitter other than the daemon's limiter or sandbox.WithAdmission should wrap its release with sandbox.WithCapacity, so a provider whose sandbox outlives the call (OpenShell) can hold the slot through the delete with sandbox.HoldCapacity. sandbox.SessionEnd and sandbox.SessionEndedError keep their names and behaviour.

Billing and spend

  • E2B and Docker Cloud runs can draw on a daily allowance in seconds. Each run on these providers creates a microVM (a small virtual machine) that the operator pays for by the second, and a rate limit bounds runs per minute, not seconds per day. A caller's paid_seconds_per_day in PLIMSOLL_CLIENTS_FILE and the daemon-wide SANDBOX_PAID_SECONDS_PER_DAY are allowances of microVM wall time per UTC day. Before admission, a run reserves its timeout (the provider's ceiling when the call sets none) plus the provider's teardown bound, 33 seconds on both (three delete attempts of up to 10 seconds each, with pauses of 1 and 2 seconds between them); a run that would go over either allowance is refused before any code runs (reason capacity, ResourceExhausted over RPC), and the refusal for the daemon-wide allowance states no numbers, which would reveal other callers' spend. At the end the run is charged the wall time of the provider call and the rest is given back. A run that crosses midnight UTC counts in the new day for the part it ran after midnight. Counts are per daemon and in memory: a restart forgets the day's spend, and since a spent allowance is a capacity refusal that placement retries elsewhere, a caller with the same allowance on K daemons can spend K times it. plimsoll-clients gains a limit command and -paid-seconds-per-day on create and import, and list shows allowances. Refusals are counted in plimsoll_shed_total{reason="paid_caller"} and {reason="paid_daemon"}, and the audit line states paid_seconds. Setting SANDBOX_PAID_SECONDS_PER_DAY for a provider that does not bill fails startup. docs/callers.md, docs/placement.md.
  • A run whose microVM may still exist with nothing able to delete it is charged its whole reservation. Such a microVM bills until the lifetime plimsoll requested at create runs out (the run's deadline plus 10 seconds on E2B, plus 30 on Docker Cloud), which the reservation covers, so charging only the run's wall time would undercount. This applies when the provider's delete gave up after its retries, and when a create's outcome is unknown. On E2B a create is unknown unless its error proves the request never left (a DNS or connection failure, or a context already ended before sending), and any answer other than 200, 201 or a 4xx refusal (a 5xx from a proxy, a 202) counts as unknown; a 2xx answer with no usable sandbox ID too. On Docker Cloud a create whose answer is not a refusal (an unknown, internal, unavailable, deadline, aborted, cancelled, data-loss or already-exists answer, an unreadable answer, a lost connection), or whose operation is still running when the run gives up, is charged in full unless the cleanup delete removed the sandbox; a create that was never sent is neither deleted nor charged. The audit line marks such a run teardown_gave_up.

Run records and attestation

  • A verified bundle is the whole file the harness wrote. In v0.17.6 a whole session or a single run deleted from a bundle still verified. Every line now carries a link, a second envelope the harness signs (predicate https://plimsollmark.github.io/plimsoll/bundle-link/v1) stating the line's position, the SHA-256 of the line before it and of its own entry, and the last line must be a checkpoint, a link alone stating how many lines precede it. A line removed, moved, added, duplicated or changed anywhere, or a cut end, fails. Every line must be exactly the harness's encoding of its entry (no whitespace, repeated or case-variant keys, unneeded escapes or trailing data), and payloads and signatures exactly the base64 of what they decode to, so a reader can find nothing in a line the verifier did not check. A harness continues a bundle only if it verifies under the signer's own key. The documented record-version rule is corrected: a session's chain holds version 2 records, plus version 3 for a call that may have run but got no answer. docs/run-records.md.
  • The caller's own list of request digests proves what the file cannot. plimsoll-attest run appends one run per invocation, each ending with a checkpoint, so a bundle cut back to an earlier checkpoint still verifies, and a whole file says nothing about runs the harness was never handed. plimsoll-attest verify -expect FILE holds the bundle to the caller's own list of request digests (request_sha256 in each run record, which the client has already checked against the request it sent), one per line, with anything after the digest treated as a label; the bundle must hold exactly those requests, each as many times as listed, so a call missing, added, or carrying a request the caller did not send fails. -expect-count N checks the number of calls alone. run prints each run's request digest for that list. In Go: attest.VerifyExpected and attest.CountCalls.
  • Bundles are owner-only. A bundle holds every run's request and response in full, submitted code and output included, and v0.17.6 created it readable by every user under the usual umask. On Unix, run now creates bundles 0600 (owner read and write only), and refuses, before anything runs, to add to an existing bundle its group or others can read or write unless -shared-bundle says it is shared on purpose; it never changes an existing file's mode. On Windows, where access is a matter of ACLs, no mode is checked, so keep bundles in a directory only you can read. A record the bundle could not take goes to a new owner-only side file (<bundle>.unsealed-<UTC time>-<random>.jsonl), never into an existing one.
  • An append that fails part way leaves the bundle as it was. run checks that the bundle can be continued before it runs anything, keeps the exchange in memory during the run, and appends under a lock on the file, so concurrent runs take turns (no lock on Windows, Solaris, illumos or AIX: run one at a time there). An append that fails part way, such as on a full disk, is cut back to the last checkpoint and the cut is synced to disk, and a successful append is synced, so the next run continues the bundle instead of refusing a partial line. If the cut or its sync fails too, the error says so and asks you to verify the bundle before the next run.
  • A session call refused before it ran no longer breaks the record chain. A session call refused at admission (the caller's rate or no free slot), by the provider (a sandbox it could not read back) or for its grant was chained by the daemon as an unanswered call that may have run, while the client, reading the not-dispatched mark, did not chain it, so the client's next record check failed. The daemon now reads the mark it sent and chains neither.

Docker

  • A call's outcome is decided by plimsoll's start marker, never by docker's wording. The project runner now gets the same per-call start marker on stderr that a snippet gets, written before node starts. A call whose stderr holds the marker ran, and its exit is the guest's; one without it is docker's failure whatever docker exited with or printed. Two misclassifications are fixed: a docker run that failed with exit 1 came back as the guest's exit 1 with docker's words as its stderr, and a runner docker never started came back as a protocol_error outcome instead of an error, in single runs and session calls. A docker CLI killed by a signal is an infrastructure error, not an exit code of -1. Such errors stay unmarked (may have run), because a broken attach stream can lose the marker of a call that did start.
  • A deadline that passes before the start marker is an error, not a timeout. What ended a run is now read when docker returns, before the container is removed. Before, the deadline was read after the removal, which has a budget of its own, so a docker run refused in milliseconds came back as a timed-out run (exit 124, docker's words as output) whenever removing its container crossed the deadline. A run or session call whose deadline passed before the marker appeared ran none of the guest's code, so it is now an unmarked error wrapping the deadline; timed_out means the guest's code ran out of time.
  • Docker's own refusals before dispatch are marked not dispatched. An image reference docker cannot use, a failed Preflight and an execution state the smoke test did not prove are refused with the not-dispatched mark, reason environment (capacity when the caller gave up), for runs and session opens; so is the wasm provider's configuration error. Before they were unmarked, which tells a caller the code may have run, so placement could not send the call to another daemon when a docker daemon was down.
  • Every container declares its lifetime, and orphaned run containers are removed. Each container carries a lifetime label (a run's is its deadline), and ReconcileOrphans removes any container past its lifetime plus 5 minutes that no open session or pool holds; before, it found only session containers, so run and smoke-test containers a crash left behind stayed. A run whose docker run does not exit 0 has its container removed at once, so a run whose CLI is killed mid-run leaves none.
  • A step that outlives its budget is killed with SIGKILL. The runner sent Node's default SIGTERM, so a step that ignored it ran past its budget, and when the run's outer deadline won, docker killed the container and every step's report with it. The run now returns every step's report, the timed-out step marked TimedOut. Needs the rebuilt images.
  • Every answer plimsoll reads from the docker CLI is bounded, and only standard output is parsed. Six reads bypassed the bounded reader (the context resolve, docker info, the image inspect, the container removal and the smoke probe's two buffers); none does now, and a test fails the build if one is added. A warning docker prints on stderr can no longer land inside a parsed answer.

Sessions

  • A session's capacity comes back only once its sandbox is gone. A session's concurrency slot and memory reservation were released when it ended, while its sandbox, and any process it left, lived on until the delete. They are now held until the sandbox is deleted (docker after the container is removed, OpenShell after its delete), and the call that ended a session says so in its own answer at once. A pool container being removed (waiting too long, made under an older image, or moved to another language) counts toward the pool's size until docker has removed it, so its replacement never runs beside it outside the pool's memory charge. docs/sessions.md.
  • The software rule is checked against what the opened session runs. The daemon checked a caller's software rule against the identity it read before the open, which could name an image docker replaced during it. It now checks the opened session's own environment (on docker, the image its container was created from) after the open, closes a session that fails the rule (not dispatched, reason environment), and states the opened session's identity in the open answer and in every call's record.
  • Calls that should not run in a session are refused. A call that reached a session which ended while the call was being prepared, or one abandoned before any caller held it (a failed open, a pool container), is refused with the session's end instead of running. A caller whose context had already ended no longer takes a free turn and resumes a suspended sandbox. A malformed call waiting behind a busy session is refused as a bad request before it takes the session's turn; before, it was refused as a deadline or as busy, which invites a retry of a request that can never run.
  • A cell cut off by the session's end reports the end. A cell (code run in an interpreter the session keeps alive) whose interpreter ended because the session ended could come back as an ordinary result with the cell marked as raised; it now returns the session's end. A caller who cancels while a cell's interpreter is starting is told it gave up, not that the image cannot run the language.
  • An idle session whose suspend failed is suspended later. A failed idle suspend that left the session alive never re-armed the idle timer, so the session was never suspended again while idle; the timer is re-armed now.
  • A provider panic during a session open no longer leaks capacity. The session reservation and the limiter slot stayed taken until restart; the panic is now recovered into a failed open.
  • A session pool's memory charge cannot overflow. For an absurd pool size given by an embedder, the pool's size times the per-run memory overflowed and the pool was admitted; the charge is now bounded by division first. plimsolld's own cap on the pool size never reached the overflow.

OpenShell

  • One caller's short timeout no longer changes the tier every caller is checked against. The driver check made before every run and session call ran under the caller's request context, so a request with a 1 ms timeout set the provider's tier to unknown and the daemon refused every request that stated an isolation floor. The check now runs apart from the request with a 10-second bound, the same bound Preflight's check uses, and a failed check refuses only that call before any of its code runs (reason environment, or capacity when the caller left). docs/openshell.md.
  • Drain leaves no sandbox to come. An open or a run still in its driver check had not yet named its sandbox, so Drain (the shutdown wait) found nothing to wait for and returned, and the operation then created a sandbox. A create that starts after Drain has begun is now refused before any code runs (reason capacity), Drain waits for creates in flight, and an open that finishes after Drain began is refused and its sandbox deleted instead of being left open.
  • A read-back the gateway does not answer refuses the call and keeps the session. Before each session call plimsoll reads the sandbox back from the gateway. When the gateway did not answer, the session ended, or the error was unmarked (may have run); the call is now refused before any of it runs (reason environment) and the session goes on, as on docker.
  • A run keeps its slot until its sandbox is deleted. An OpenShell run deletes its sandbox after returning its result; the run's concurrency slot and its share of SANDBOX_TOTAL_MEMORY_MB were given back at once, while the sandbox, and any setsid process the run left, lived on for the few seconds the delete takes. Both are now held until the delete is through.
  • Smaller corrections. A session's lifetime now runs from the open, as on docker, not from after the create. A grant refusal marked not dispatched (a token mint that outlasted the deadline) stays marked instead of coming back as an unmarked deadline. The process lister, the sweep, the interpreters' relays and a grant's relay start under /usr/bin/env -i with a fixed PATH; a v0.1.2 gateway passes none of the image's variables to an exec, so this changes nothing today and keeps a variable such as NODE_OPTIONS out of those programs on a gateway that starts passing them.

E2B and Docker Cloud

  • The Docker Cloud provider speaks an API Docker no longer documents. It was written against github.com/docker/sandboxes-api v0.36.0, a protobuf API (protobuf is Google's schema format for messages) that the provider speaks as Connect JSON (Connect is an RPC protocol that carries JSON or protobuf over ordinary HTTP), and its live suite passed against the real service on 2026-09-24. Earlier docs called that API "Docker's published API contract". Docker's API reference now documents a different, experimental REST API that this provider does not speak, and whether the Connect endpoint still answers has not been checked since 2026-09-24; no live suite ran for this release. Before relying on the provider, run make audit DOCKERCLOUD=1 (it creates billable microVMs); a failure there means the provider needs porting. docs/dockercloud.md.
  • E2B orphans are found by lease key, at any age. Every E2B create now stamps a lease key in the sandbox's metadata, tracked from before the request until the kill has finished, and ReconcileOrphans reaps a stamped sandbox whose key is not tracked, whatever its age. Before, a fixed 60-second grace window spared young orphans and could kill a create that took longer than 60 seconds. A malformed or lost create answer is reconciled at once. E2B, Docker Cloud and OpenShell now share one lease rule. The paid E2B live suite checks that E2B's listing returns the lease key; it was not run for this release, so that behaviour of E2B's API is not yet verified. A sandbox listed without its key is left to the lifetime plimsoll requested at create.
  • A missing credential is refused before anything runs. A missing E2B_API_KEY or DOCKER_SBX_TOKEN, and an E2B resource cap the provider cannot enforce, returned errors with no mark, which tells a caller the code may have run; they are now not dispatched, reason environment, so a caller can send the call elsewhere.
  • A failed run still returns its host-API call trace. Calls brokered for a run's grant before a later failure (a broken exec stream, a failed artifact read) now reach the daemon's metrics and audit line, which count them from the result.
  • Vendor and guest error text is scrubbed on the stream paths too. A Docker Cloud end-of-stream error and an E2B envd trailer (envd is the agent inside the E2B microVM, so its text is the guest's) went into errors without credential scrubbing or a size bound; both are now scrubbed and cut like every other vendor error.

Advisor

  • Findings fall into four classes, and only the first reaches a caller. A finding about a per-item fan-out is: a batch route the profile declares and grants (returned to the caller as suggested_route); a declared route the profile does not grant (grant_route on the audit line: the one line the operator adds to the allow list); a candidate the path suggests, the collection route granted or listed in the profile's catalog with nothing declared (candidate_route on the audit line, advice_candidate_routes in the totals, class candidate in the HTML report), for the operator to check and then declare, because a path cannot say whether the route pages, returns fewer fields or covers another scope; or no known route, for which a paste-ready prompt asks for the smallest change or a plain statement that none is warranted. plimsoll cannot check a declaration: the trace holds no bodies. docs/efficiency-advisor.md.
  • batch_of is checked when it is loaded. A profile's batch_of is refused if either side is a write, a per-item route has no *, a route names itself, or two batch routes claim one per-item route. plimsoll-specgen resolves x-plimsoll-batch-of operationIds, refuses one that is unknown, ambiguous or not expressible as a grant, runs its output through the same check, and writes the relation with -emit batch, in the JSON bundle, and as batch_of.json under -outdir. The specgen worked example now declares one relation and leaves it ungranted, to show the operator's case. docs/examples/specgen/README.md.
  • Repeated-read findings count only reads that sent no body. host.call("GET", path, body) sends a body, and the trace keeps its size, not its bytes, so same-size GETs with different bodies to one fixed route (a search with different queries) were reported as one request repeated and told to cache. Fan-out findings still count every successful call. Repeated-read findings stay operator-only, and the docs now say why: one fixed route read again and again is often a loop waiting for a change, where caching would break the loop, and the trace cannot tell the two apart.

Placement

  • Placement keeps the newest description of each backend. An answer is now stamped with when its Describe was asked and stored only if no answer asked later is held, so a slow Pool.Refresh that finishes after an on-demand Describe no longer puts its older answer back.

Docs

  • SECURITY.md states what a deployment still has to handle. Five risks no setting removes: every grant is a way out for data, since the broker checks a call's method and route, never what it carries, and a body can ride on any method, GET included (grant routes only on an API that may receive that data, prefer routes without *, and give writes a profile of their own); sandbox output is untrusted input to the agent that reads it (prompt injection: instructions hidden in data so that the reader follows them), and plimsoll does not inspect it; whoever controls the docker provider's daemon access controls its host; under runc a container's uid is the host's, so rootless Docker or the daemon's userns-remap protect more than the guest uid alone, and since every run shares the guest uid, a runc escape still reaches other runs' processes; and runs on one host share its hardware, with no claim against timing or cache side channels at any tier. SECURITY.md.
  • The required Docker suite runs under the race detector. Both CI jobs that run the Docker suite (under runc and under runsc) now use Go's race detector, and make audit DOCKER=1 runs each docker test once. CONTRIBUTING.md states what each green check proves.
  • sandboxtest.Wasm compiles QuickJS at its first call in a process. It runs one snippet then, so the compiled module is in the process-wide cache before any test runs, and a test that sets its own short timeout no longer pays the compile (seconds under the race detector) inside it.
  • The sessions and capsule example pages are regenerated with a bundle that carries links and a checkpoint, and the sessions example's five calls carry trace IDs.

Clients

  • @plimsollmark/client and the Python client are at 0.18.0, unchanged apart from the version. The protocol number is still 2, so v0.17.6 clients and a v0.18.0 daemon work together.