Repository navigation
Releases
v0.5.4
Otter v0.5.4 — Per-job desired state, and a delete that sticks
Status: release notes. Date: 2026-10-08. Previous release: v0.5.3.
v0.5.3 added otter delete --cloud and it did not work. The delete removed the
release on the runtime, the control plane still desired it, and the agent
converged the runtime back onto it within four seconds — so the job reappeared,
while the command reported success and a note claiming every release was removed.
This release fixes the cause: a runtime's desired state is now a set of jobs, one
entry per job, instead of a single release.
Read this first:
- Deleting a job now removes it from Cloud's desired state, which is what
stops the runtime being reconciled back onto it. Until this release, deletion
and convergence were fighting and convergence always won, because it retries. - One job's deploy no longer disturbs another's. The desired state was a
single digest per runtime, so deploying job B overwrote what job A was relying
on. Each job is now addressed by its stable job id. - Cloud only reports a delete as done once the runtime confirms it. A
removal that has not been confirmed ispending, notdeleted. - The agent protocol gained a capability — see below. It is additive and
gated, so nothing breaks during a rollout.
What changed
The desired-state answer from the control plane is now a snapshot:
The agent converges the set: it installs and activates what the snapshot
names, and removes managed jobs the snapshot has dropped. Removals are not gated,
so deleting one job never pauses its siblings. It only ever removes jobs the
runtime reports as managed, so a job whose source lives in the workspace is out
of reach by construction rather than by a rule someone has to remember.
The runtime-level generation fence is unchanged. Jobs are addressed by their
stable id.
The capability: desired.snapshot.v1
Cloud and agents deploy independently — Cloud on a push, an agent only when a
tenant image is rebuilt — so the shape is dual-served, chosen by an explicit
capability:
- the agent declares
desired.snapshot.v1and reports the releases it holds; - Cloud serves the per-job shape only when the agent declares the capability
and the runtime says it can identify and remove managed jobs; - otherwise the previous single-release shape is served, unchanged, and an agent
that receives nosnapshotmarker removes nothing.
Adoption of per-job state is seeded from what the agent reports it already
runs, never from an empty table — an empty snapshot means "delete every managed
job", so seeding from nothing would have ordered the release store emptied.
Three ways this could have failed silently
Recorded because each was caught late and each would have looked like success:
- A missing
snapshot: truemarker. The agent keys on it, so a per-job answer
without it decodes as the legacy shape: nothing deployed, nothing removed. - Confirmed by silence. An empty release list in a report could wipe the
stored observation and read as confirmation that a removal had happened. The
stored set is now only replaced when the field was present, it is nullable so
"never reported" is not "reported empty", and a removal requires an observed set. - A stale single-release projection. After a snapshot delete, the legacy
projection still named the deleted digest, so one poll from a not-yet-upgraded
agent could re-install it. It is cleared before the delete.
Installing
brew upgrade otterLinux packages are on the release page, per architecture. Upgrading the client
before the runtime matters: a runtime that becomes snapshot-capable expects the
deploy to name a job id, which older clients do not send.
Compatibility
This is the first agent-protocol change since the compatibility policy in
v0.4.0. It is additive: a new optional capability, dual-served, with old agents
receiving exactly today's shape and new agents falling back safely. No manifest,
SDK, CLI-JSON or HTTP shape that the policy froze is changed.
v0.5.3
Otter v0.5.3 — Delete a job, from the command line
Status: release notes. Date: 2026-10-08. Previous release: v0.5.2.
v0.5.3 is the client half of job deletion. v0.5.2 shipped the runtime side —
a job known only from its releases can be deleted, and the agent accepts a delete
command — but nothing in the CLI could ask for one. This release adds the verb, a
confirmation in front of it, and the rule that it names a job rather than a
directory.
Read this first:
- Deletion is irreversible and the command now asks. It prints what it will
destroy, names the job, and waits fory/yes. Anything else —n, blank, end
of input — aborts withaborted; nothing was deletedand exit 1. The default is
No, so a stray Enter does nothing. - A scripted delete needs
--yes. Non-interactive input without it is
refused rather than assumed: the prompt never reads a piped answer as consent,
and it never hangs waiting for one. - The argument is a job name or id, never a directory.
otter delete .and
otter delete jobs/counterare refused with a usage error (exit 2), before the
prompt and before any request. The confirmation can only name something an
operator recognises, and a path is not that.
Deleting a job
otter delete counter # the local runtime
otter delete --cloud counter # a pooled runtime, through Otter Cloud
otter delete --cloud counter --runtime rt-pool-2--cloud resolves the credential from otter login (or OTTER_CLOUD_TOKEN),
finds the job by case-insensitive name or by id, and requires a deploy-scoped
token: a read token is refused. Deleting a job with a run in flight is refused
with the run named; cancel it first.
The runtime reports what it could not remove, and that report is printed rather
than swallowed. The case that matters: a job known only from its releases has no
tombstone, so releasing the same job id again brings it back.
What a delete removes
Every run of the job and the logs those runs produced, every captured request and
response payload, the job's stored state, every schedule it owns, its
configuration, and every release including the active one. Source files in the
jobs directory are left in place.
Limitations
- A dedicated (non-pooled) runtime's job cannot be deleted from Cloud. The
runtime's own delete is admin-only, and Cloud deliberately holds no admin
credential, so the request is refused rather than half-performed. - The note is the only warning you get that a release-only job can reappear.
It is carried from the runtime through the control plane to the terminal, which
cost three hops to get right and is worth knowing about if it ever goes quiet.
Installing
brew upgrade otterThe .deb for Linux is on the release page as usual, per architecture.
v0.5.2
Otter v0.5.2 — Deploy what you released
Status: release notes. Date: 2026-10-07. Previous release: v0.5.1.
v0.5.2 makes otter deploy --cloud promote a release you already made, rather
than packaging the working directory as it goes, and adds job deletion. Deploy
stops being a build; release becomes the build.
Read this first:
otter deploy --cloudchanged meaning. It promotes the latest local
release for the job. Editing a job and deploying no longer ships the edit —
runotter releasefirst.--buildrestores the previous behaviour, packaging
the workspace as part of the deploy.- Name a release when you don't want the latest.
--release <digest>takes
bare hex orsha256:<hex>;--release latestis the default spelled out.
otter release --listshows the digests with their creation times, and "latest"
is the newest by creation time. - The prepared environment is frozen at release time. A job using
python.mode: managedprepares its interpreter and locked dependencies when you
release, and those ship inside the release. Release on the platform the runtime
runs on — a release built on macOS carries a macOS environment. - Deleting a job is new, and it is destructive. It removes the job's state,
runs and logs, captured payloads, schedules, configuration and every release,
including the active one. It refuses while a run is in flight.
Deploying
cd <job directory>
otter release # build the artifact; prints its digest
otter deploy --cloud # promote the latest release
otter deploy --cloud --dry-run # say what would go, send nothingotter deploy --cloud --release <digest> promotes a specific release, which is
also how a rollback is expressed: promote the digest you were on before. A release
Cloud does not already hold is uploaded first — a copy, not a build, so no
preparation is involved and the digest cannot change on the way.
otter deploy --host <user@host> is unchanged: it is a converge that ships the
source tree for every job, and it neither requires nor produces a release.
Deleting a job
otter delete <job> # locallyThis is also available against a pooled runtime from Otter Cloud
(DELETE /api/jobs/{id}, a deploy-scoped operation). What it removes: state,
runs and logs, captured payloads, the webhook token, pause, schedules, job
configuration, and every staged release including the active link.
Two things worth knowing before using it:
- A job with a run in flight is refused (409, naming the run) and nothing is
removed. Cancel the run first. The check runs before any store write, so a
refusal is never a partial delete. - A job that exists only as a release has no tombstone. A workspace job keeps
its source and is remembered as deleted; a job known only from its releases is
removed outright, so staging and activating a new release for the same job id
makes it reappear. The response says so rather than leaving you to discover it.
Deleting a dedicated (non-pooled) runtime's job is refused: the runtime's own
delete is admin-only, and Cloud holds no admin credential by design.
Fixes
- Job detail pages in Otter Cloud reported every job as unavailable. The
transport asked the agent for ajob.getread, which is not one of the reads
the agent is allowed to perform; a single job is meant to be narrowed out of
jobs.list. The job list worked throughout, which is why only the detail page
was empty. Fixed in Cloud; no CLI change. - The README's install section and the changelog index had fallen behind the
releases: the section offered only Homebrew, with no Linux path and no mention
thatapt install otterby name does not exist yet.
Known limitations
otter deploy --cloudreportsupload: neededeven for a release Cloud
already holds. The presence probe (GET /api/releases/<digest>) is not served
by the control plane yet, so the client cannot confirm presence and uploads
idempotently. The upload is harmless; the report is imprecise.
v0.5.1
Otter v0.5.1 — Ubuntu packages
Status: release notes. Date: 2026-10-07. Previous release: v0.5.0.
v0.5.1 adds a Debian package to the release artifacts. v0.5.0 shipped
tarballs only, so a Linux install meant downloading an archive and placing two
binaries by hand: nothing could upgrade them and nothing could remove them.
Read this first:
- The
.debinstallsotterandotterdinto/usr/bin. If you already
put them on yourPATHfrom a tarball —~/.local/bin, most likely — your
shell may keep resolving that copy, because it comes first inPATH. Remove
it, or leave its directory later inPATH, orotter versionwill disagree
withdpkg -l otter. - macOS is unchanged. The cask in the tap still installs the tarballs, and
this release updates it to0.5.1in step.
Installing on Ubuntu
curl -LO https://github.com/tkoizumi/otter/releases/download/v0.5.1/otter_0.5.1_linux_amd64.deb
sudo apt install ./otter_0.5.1_linux_amd64.debapt records the package, so apt remove otter works, and the two binaries are
versioned with the rest of the system rather than living in a home directory.
Nothing else changes in this release: no runtime, protocol or manifest surface
is touched, and no migration runs.
What this is not yet
A hosted APT repository. There is no otter package in an archive you can
apt install otter from by name, and no signing key to trust — so upgrades do
not flow through apt upgrade yet, and each release is still a download from the
release page. That repository is the next step; this release is the artifact it
will carry.
v0.5.0
Otter v0.5.0 — Self-serve deploys to Otter Cloud
Status: release notes. Date: 2026-10-07. Previous release: v0.4.0.
v0.5.0 makes a job deployable by the person who wrote it: otter login against
Otter Cloud, otter deploy --cloud --job <name> to package and promote one job
to a runtime the organization owns, and a pooled tenant that runs it. The runtime
half of the pipeline already existed; this release is the client half and the
hosting that makes it real.
Read this first:
- The tenant container changed shape. A pooled tenant is now one
container running two processes:otterdas uid 10001 and the agent as uid
10002, over loopback. The separate sidecar container is gone, and an existing
tenant must be re-provisioned rather than upgraded in place. otter loginneeds a token minted in Otter Cloud; there is no browser or
device flow. It is read from--token,OTTER_CLOUD_TOKEN, or a terminal
prompt, and stored per machine under~/.otter/cloud.json(0600).- Assigning a runtime is an API call.
POST /api/runtimescreates the
organization's assignment and returns the per-runtime bootstrap secret once;
it is session-authenticated and has no dashboard page yet.
Deploying a job
otter login --cloud https://app.runotter.dev --token <token>
cd my-workspace
otter deploy --cloud --job <name>deploy --cloud packages one job with the same release machinery otter release uses, so the digest it promotes is the canonical release identity, and
asks the control plane to move the assigned runtime onto it. --dry-run and
--runtime are honoured; otter login --status reports the stored identity
without deploying.
A deploy installs and activates a release. It does not run anything: the run is
a separate act, and on a pooled runtime it is triggered from the job's page in
Otter Cloud.
One container per tenant, two uids
The agent must reach otterd over loopback, and gVisor cannot give it that
across containers: it does not honour --network=container:<other> (a joining
container comes up with only lo), and a container's listener is not reachable
across a gVisor bridge either. So both processes live in one container and
share a network namespace by construction. otterd is loopback-only and needs
no API token, and nothing is exposed on any bridge.
The isolation the separate sidecar existed for is kept by uid separation, and it
is why the container starts as root:
| Process | uid | Access |
|---|---|---|
otterd |
10001 | owns the tenant volume, mode 0700 |
| agent | 10002 | its own tmpfs, cannot read or write the tenant volume |
Root is needed only to drop to those two uids; the container is granted exactly
SETUID, SETGID and CHOWN, and runs with no-new-privileges. After the drop
no process holds a capability. The agent's Cloud credential is stripped from
otterd's environment, so the tenant process cannot read it.
What the agent does differently
- Released jobs are discovered. A runtime whose workspace is empty still
resolves the job a release was installed for, by name and by id, so a deploy
to a pooled tenant is runnable without any source in/workspace. - Applying is idempotent. A control plane publishes desired state and keeps
returning the same generation; the agent now compares the content it observes
against what is desired and does nothing when they match. Before this, a stable
desired state re-downloaded and re-activated the same release every few
seconds and leaked a staging directory on each pass. - Polling is paced, and its decisions are logged to stderr — which in a
container is the container log. The agent used to be silent, which made a
stalled deploy indistinguishable from a working one.
Fixes worth naming
- Installing a release that was already in the store left its staging directory
behind; cleanup is now unconditional. - The pool resolver now binds and checks every tenant gateway. It previously
reloaded a unit that declares noExecReload, so a newly added tenant's
gateway was never bound, and the health check probed only the first gateway —
reporting healthy while the new tenant's agent could not resolve anything. - The agent data directory is
chmoded before it ischowned; the other order
needsCAP_FOWNER, which this container deliberately does not have.
Installing
Homebrew (macOS):
brew install tkoizumi/tap/otterThe cask installs the two prebuilt binaries, otter and otterd; no Go
toolchain is needed.
v0.4.0
Otter v0.4.0 — The interface freeze
Status: release notes. Date: 2026-10-03. Previous release: v0.3.0.
v0.4.0 completes the runtime half of Track R: a scoped control credential,
first-class dynamic schedules, a reachable-but-scoped remote API, pinned job
configuration, and the published compatibility policy that freezes the manifest
schema, the Python SDK, CLI JSON and the HTTP API.
Read this first:
- It ships three schema migrations (
0013_api_tokens,
0014_dynamic_schedules,0015_job_configs). An upgrade runs them on first
start, and a rollback is a restore, not a binary swap: a newer schema is
refused by an older binary by design. - Two shapes break. A cron expression now means UTC rather than the host's
local time, and every list response is an object rather than a bare array.
Both changed deliberately, before the compatibility promise takes effect, and
both are recorded in compatibility.md. - The runtime contract is version 2 and reached from
GET /v1/versionorotter --json version.
What breaks, and what does not
| Surface | Before | v0.4.0 |
|---|---|---|
trigger.cron time zone |
the host's local time | UTC; use a schedule's timezone for a local wall-clock meaning |
GET /v1/jobs, /v1/runs, /v1/runs/{id}/logs, /v1/tokens, capture requests |
a bare array, or an object with no version | {"schema_version": 1, "jobs": [...]} and the matching key per endpoint |
otter jobs --json, otter runs --json |
a bare array | the same versioned envelope as the HTTP API |
PUT/DELETE /v1/jobs/{id}/schedule, otter schedule set|clear |
the supported cadence surface | deprecated; still work, removed in the next major |
A manifest-declared trigger.cron |
the API could overwrite it | it is manifest-owned; PATCH/DELETE/legacy PUT refuse it with 409 |
A job that reads ctx.trigger, ctx.state or its environment needs no change.
A script that parses a list response does: add .jobs / .runs / .logs /
.tokens / .requests / .schedules. The old line-oriented CLI output is
unchanged.
Scoped control credential (CL-21)
Named API tokens with read, control and capture scopes, minted by
otter token create, stored as a SHA-256, revocable with immediate effect (the
row is read on every request). Full authority stays the static OTTER_API_TOKEN;
a named admin token is deliberately not offered. capture is control plus the
two payload routes, so a control plane holds one credential that can both
re-run a job and read what it sent; it is deliberately not implied by control,
so a credential minted before the scope existed does not silently gain access to
the client's traffic. The route-by-route boundary is in
api-reference.md.
Dynamic schedules (CL-22)
A schedule is now a first-class, id-keyed runtime object, not one cadence per
job buried in a manifest.
GET/POST /v1/jobs/{id}/schedules,GET/PATCH/DELETE /v1/schedules/{id}, andpause/resume;otter schedule list|add|update|remove|pause|resume.- A per-occurrence
payloadbecomesctx.trigger.body, so "this job, for
dataset 42" is expressible without a release. Idempotency-Keyon create: a retry returns the existing schedule.- Exactly one run per occurrence, across restarts and duplicate wake-ups: the
occurrence ledger (schedule_fires) is written in the same transaction as the
run. - A manifest's
trigger.cronreconciles as amanifest-owned row that the API
cannot change; an API-created schedule is never touched by a reload.
Known limit: the missed-occurrence policy accepts only skip.
coalesce and catch_up are reserved names and are refused with 400 rather
than accepted and ignored (OT-007).
Reachable, scoped remote surface (CL-23)
otter deploy accepts a wildcard or public --listen behind an explicit
--allow-remote-bind and prints a warning; a private bind warns without the
opt-in. assert-host-permissions.sh asserts the configured posture and
requires OTTER_API_TOKEN for any non-loopback API. TLS remains a documented
reverse-proxy pattern (security.md);
the daemon still serves plain HTTP behind it.
Job configuration
A job's configuration is the layer between the release and the run: a store
name, a dataset id, a backfill date. It is versioned and immutable, and every
run pins the version it resolved at submission, so a queued, retrying or
backlogged attempt keeps its values.
GET/PUT /v1/jobs/{id}/config; admin-only to write inv0.4.0.- The child reads
ctx.config(a dict;{}when unconfigured), passed as
OTTER_CONFIG. - Not a secret store: values are returned to any scoped reader and are not
projected into named environment variables in this release.
Interface freeze (R4)
compatibility.md publishes what makes a change breaking
for each frozen interface, how a deprecation is retired (one minor release
minimum, removal only in a major), and the current deprecation table.
GET /v1/version and otter --json version report schema_version,
contract_version, the manifest schema, the SDK version and the supported
platforms in one document. Contract tests pin the version document, the platform
list and the frozen JSON keys of every machine document.
Also in this release
OT-010: release retention and a submission binding a release now serialize
on the database write lock, so a prune cannot strand a queued run.OT-024: the CloudWatch metric contract is one file,
monitoring-contract.json, asserted on both the runtime and the platform
side.OT-029: a retired job whose source is still in the repository no longer
fails a full deploy; the release step skips and reports the suppressed path.- The runtime contract gains §7 "Honest limits": uncoordinated concurrent
updates, retry effect duplication, no downgrade, plain-HTTP transport, and
configuration not being a secret store.
v0.3.0
Otter v0.3.0 — Operable unattended
Status: release notes. Date: 2026-10-02. Previous release: v0.2.0.
v0.3.0 closes the first half of Phase 2. It is a minor release: it adds
capabilities and documented semantics and corrects documents. Two facts a reader
must know before upgrading:
- It ships two schema migrations (
0011_job_terminology,0012_job_schedules).
An upgrade therefore runs migrations on first start, and a rollback is a
restore, not a binary swap. - The
integration→jobrename is public. The manifest file and every
manifest field are unchanged, but the names a client or a child process reads
are not.
Read this first: the rename is on the wire
| Surface | v0.2.0 |
v0.3.0 |
|---|---|---|
| HTTP and CLI JSON | integration_id |
job_id |
| Child environment | OTTER_INTEGRATION_ID, OTTER_INTEGRATION_NAME, OTTER_INTEGRATION_DIR |
OTTER_JOB_ID, OTTER_JOB_NAME, OTTER_JOB_DIR |
| Daemon configuration | OTTER_INTEGRATIONS_DIR |
OTTER_JOBS_DIR |
| Manifest file and fields | otter.yaml, name/entrypoint/… |
unchanged |
What actually breaks, and what does not:
- JSON keys changed. A dashboard or script that parses
integration_idout
ofotter runs --jsonor the HTTP API must readjob_id. Migration0011
renames the affected database columns; the rows are preserved. - The child environment did not lose the old names. The executor exports
OTTER_JOB_ID/_NAME/_DIRandOTTER_INTEGRATION_ID/_NAME/_DIR, the
latter deliberately, so a job inside an immutable release written before the
rename keeps working. The old names are no longer the documented ones, but a
job that reads them needs no change. - The daemon configuration names did change.
OTTER_INTEGRATIONS_DIRand the
--integrationsflag are no longer read; useOTTER_JOBS_DIR/--jobs.
The embedded Python SDK reads the new names, so a job that uses otter.Context
needs no change either way.
You can watch an upgrade
The operations guide told an operator to watch for migration_applied on
restart, and database.Migrate had no logger, so a successful upgrade was
silent — the one moment the schema moves was the one moment nothing was said.
- One
migration_appliedrecord per applied migration, atinfo, in the
order they were applied, carrying the migration'sversionandname
(nameis the filename, which is also its description; no new column was
added). - A restart with nothing pending logs nothing, so "no migration lines" means
the schema did not move rather than that nobody looked. - A failed migration is named before the process exits:
migration_failed
aterror, with the version and name, and the schema stays at the last
version that committed. A repaired migration can be shipped and the next start
continues from there.
You can operate it unattended
/health answered "how many?" and none of "how old?", "which job?", "did it
stop succeeding?", "is the disk filling?". One integer of queue depth cannot tell
a busy queue from one stuck since yesterday, and a job failing since its last
success still showed healthy totals.
For an authenticated caller only, /health now carries:
queue— the age of the oldest run that is claimable now
(available_at <= now), the queue depth per job, the count of runs parked by
retry backoff, and when the soonest of them is due. Retries are reported
separately from the age on purpose: they wait on a clock, not on capacity.freshness— each job's last success and its age, from one grouped read.
A job that has never succeeded omits both fields; that absence is the signal.storage—db_bytes(page_count * page_size, so the WAL is excluded)
and the data directory's free and total bytes. The guide previously made
"alert whenotter.dbexceeds about 2 GB" a host-level check, which a remote
operator cannot perform; now they can.
Each block degrades on its own and is logged when it does. The unauthenticated
liveness body is unchanged, field for field, so a caller without a token learns
nothing new. otter status prints the same signals, and otter jobs --schedule
renders the daemon's last success instead of listing runs once per job.
Retention is automatic, and a deploy prunes releases
- Runs and run logs expire on their own windows (
--run-retention,
--log-retention), with--capture-retentionalready covering HTTP payloads.
Every window is opt-in and defaults to retaining forever, and the tracked
windows are committed inotter.deploy.yamlso a rebuilt host cannot silently
retain forever. The documented manual-pruning SQL is corrected to filter on
runs.created_at; it was filtering on a column that does not exist. otter deployprunes inactive releases with the same--keepwindow a
release uses (default 3,0keeps everything), and every prune path refuses
to act when the pin set is unreadable. A queued or retried attempt keeps its
bound release whatever happens to the release directory.
The backlog answer is written down
"What happens if a run outlasts its next scheduled run?" runs opposite to
intuition, and it was written nowhere. operations.md §"Backlog behavior" now
states all five consequences: every cron occurrence becomes its own run
unconditionally; with concurrency: 1 they serialize into a catch-up backlog;
downtime skips while slowness accumulates (occurrences during downtime are
never replayed — the linked contract section is an explicit non-guarantee);
queued work executes the release bound at submission, so a deep backlog can run
older code than what is active; and raising concurrency produces genuinely
concurrent executions. No behavior changed.
A downgrade is refused, and the procedures are drills
- A database migrated by a newer binary is refused. Migrations are
append-only, so a binary that does not embed the database's newest migration
exits withmigration_failednaming both versions, before anything reads or
writes the schema. Before this release, "downgrades are not supported" meant
"starts, then fails on the first query that touches a renamed column".
Rolling back remains a restore of the pre-upgrade backup. - The operating procedures are executable.
make drillruns drills that
assert their own outcome:backup-restore(a hot backup restores onto a
different, empty data directory and the job runs again),upgrade(the
previous release's schema upgrades observably, and the runtime works after),
downgrade-refusal,secret-rotation,fresh-archive(a release archive
installs outside the checkout and runs a job),deploy-failure,unit-caps,
disk-pressureandliveness. Eight of the nine document aDRILL_SABOTAGE
mode that removes the behavior under test, so a green run is demonstrably
falsifiable;unit-capsis the exception, falsified instead by theUnitFile
render-test mutation in
evidence/phase-0/2026-09-29-p0-04-unit-caps-mutation.txt,
with a drill-level mode tracked asOT-030. - A retry keeps its bound release and environment. Activating a newer
release, changing the interpreter, or changing the retry policy while a retry
is pending cannot move that retry onto different code, a different
environment, or a different policy; a pending backlog also keeps its digest
through a release prune. This is now a guarantee in
runtime-contract.md,
pinned byTestRetryExecutesTheParentsReleaseSnapshot,
TestRetryResolvesTheParentsManagedEnvironment,
TestPendingBacklogKeepsItsDigestThroughAReleasePruneand
TestRecoveryPlansRetriesFromTheBoundRelease. - The trust-boundary claims are asserted, not restated: the data directory
is created owner-only, a non-loopback bind without a token is refused, the
unauthenticated/healthbody carries only status, version and uptime, a
per-run token cannot read another job's run, logs or state, and a webhook
token authorises only its own job's hook and is never recorded on the run.
Also in this release
- A detached start forwards the flags it accepts.
otter start --detach
rebuilt the child's arguments from--jobs,--dataand--listenonly, so
every other flag was silently dropped — including--api-token, which brought
the runtime up with its API unauthenticated while--helppromised the flag
was forwarded.--workers,--log-formatand the rest were dropped the same
way. Present sincev0.2.0; found by the v0.3.0 exit gate's document checks.
The detached path now builds its arguments the way the foreground path always
has. - Schedules are runtime state.
otter schedule set|clear|show(and
PUT/DELETE /v1/jobs/{id}/schedule) change a job's cadence without editing
the manifest or restarting; a cleared schedule is deliberate and does not fall
back to the manifest (migration0012). - A deploy can bind the API to a private address, not only loopback.
- Managed Python is harder to misconfigure: a lock the configured package
index cannot satisfy is refused, preparation has an explicit route with an
egress preflight, and the generated unit defaultsMemoryHighto off so
MemoryMaxis what kills a runaway.
Upgrade from v0.2.0
- Back up first — the complete backup, not just the database
(Backups). An upgrade runs migrations on first
start. - Stop the daemon, replace both binaries, start it, and watch the journal for
onemigration_appliedper migration (0011,0012). otter status,otter jobs,otter runs --all --limit 10.- Release every job once. The release layout is versioned and the new
implementation deliberately does not reuse a snapsh...
v0.2.0
Otter v0.2.0 — Dependable execution
Status: release notes. Date: 2026-09-28. Previous release: v0.1.19.
v0.2.0 is a minor release. It closes Phase 1 of the
product roadmap: accepted work is accounted for, even
after a hard kill. It changes runtime behavior and publishes a new document;
it ships no schema migration and no manifest, CLI-JSON or HTTP API shape
change. Read the semantics below before upgrading.
Recovery is complete
v0.1.19 stopped recovery from losing accepted work; v0.2.0 closes the
remaining gaps, so interrupted or newly claimed runs can no longer be stranded.
- Every interrupted run is recovered. Startup reads all
runningrows,
not the first 50. The old listing limit silently capped recovery at 50 runs,
so a busy crash left the rest permanently non-terminal. Recovery now pages the
whole status. (v0.1.19;TestCrashRecoveryHandlesMoreThanOneListingPage,
TestReconcileQueueReenqueuesMoreThanOneListingPage.) - A claimed run that cannot start returns to the queue. If loading the run
or marking itrunningfails on a transient database error, the attempt is
re-enqueued with a 5s backoff instead of sitting asqueuedwith no queue row
until the next restart. A run whose record no longer exists is dropped once,
atwarn. (v0.1.19;TestClaimedRunIsRequeuedWhenMarkRunningFails.) - Capacity is not leaked. A claim that fails to commit releases the worker
slot it reserved, so a daemon no longer quietly runs fewer workers than
configured. (v0.1.19;TestClaimCommitFailureReleasesCapacity.) - Finish and retry are one durable decision. The terminal status, the
successor attempt and its queue row are written in a single transaction. A
crash between them can no longer leave a terminal failure whose retry is
missing, and aFinishwrite that fails after a successful child is recorded
in a fallback journal and applied on the next start rather than re-running the
child. (5481e1c;TestFinishCommitFailureLeavesNoPartialOutcome,
TestFallbackJournalIsAppliedOnRestart.)
Startup can now refuse
A startup that could not repair interrupted work used to log the error and open
for new work, turning one transient database error into permanently stranded
runs. otterd is now fail-closed: if crash recovery or queue reconciliation
cannot complete, startup returns an error and the process exits.
- The safe failure is a daemon that stays down and visible, not one that serves
while accepted work sits stranded. This is the user-visible change: a bad
database now stops the daemon instead of letting it run degraded. - The one escape hatch is explicit:
--allow-incomplete-recovery
(OTTER_ALLOW_INCOMPLETE_RECOVERY=true) starts anyway and logs the error at
errorlevel, for an operator who must bring a daemon up to inspect or repair
a database it cannot read. - A failure to persist the repair of any interrupted run counts, not only a
failure of the initial read.
(ec01900; TestStartupFailClosedOnIncompleteRecovery.)
Child lifetime is platform-specific
- Linux: a child cannot outlive an abruptly killed daemon. The child is
launched withPdeathsig: SIGKILL, sokill -9, an OOM kill or a panic kills
the direct child before its retry can overlap it. This is the guarantee that
stops duplicated external effects after an abrupt death. - macOS: the exposure is now stated, not implied. Darwin has no
parent-death signal. After an abrupt daemon death the in-flight child is
reparented and keeps running while startup marks the run failed and retries
it, so non-idempotent work can run twice. Closing this needs a supervisor
process or a death-watch pipe inside the child; it remains unscheduled. - The graceful paths — timeout, cancellation and daemon shutdown — are
unchanged and still signal the whole process group on every platform.
(2220509; TestChildDiesWhenDaemonIsKilled, Linux only.)
The runtime contract is published
runtime-contract.md (contract version 1) is the
versioned statement of what the runtime promises and, just as importantly, what
it does not. It pins the attempt state machine, what an accepted request
promises, missed cron windows, duplicate webhook delivery, state-write
concurrency, retry effect duplication, and the platform-specific guarantees. It
is backed by the WS4 fault matrix, whose evidence status is stated per scenario.
Two honest non-guarantees worth reading before relying on them:
- Exactly-once is not promised. Execution is at-least-once in effect; a
retry can repeat external effects. - A retry's bound release and environment are implemented but not yet
independently proven. A retry is recorded with its parent's release and
environment and is expected to re-use them, but no automated scenario proves
the retry half yet; it is tracked asOT-011in
open-work.md.
(b5a4001.)
Upgrade from v0.1.19
- No schema migration. No migration shipped between
v0.1.19andv0.2.0,
so upgrade and downgrade are a binary swap. Keep the usual database backup for
any operational change, but no data conversion runs. - No manifest, CLI-JSON or HTTP API shape change. The only new flag is
--allow-incomplete-recovery/OTTER_ALLOW_INCOMPLETE_RECOVERY. - Startup can now fail where it previously warned. A daemon whose database
cannot be repaired will not start. That is the intended change; use
--allow-incomplete-recoveryonly to inspect or repair, not to keep serving. - Linux binaries now use
Pdeathsig. The behavior change is visible only
after an abrupt daemon death.
Verification
make test, make smoke, make lint, goreleaser check and
goreleaser release --snapshot --clean pass on the tag commit. The release
workflow runs the same test and smoke jobs before GoReleaser publishes, so a
tag cannot publish a binary that fails the loop a user performs.
v0.1.19
v0.1.18
Changelog
- 98df724 Add integration pause and refine run inspection
{ "runtime_id": "…", "generation": 12, "snapshot": true, "jobs": [ { "job": "<durable job id>", "digest": "sha256:…", "url": "…" } ] }