Skip to content

Releases

v0.5.4

Choose a tag to compare

@github-actions github-actions released this 08 Oct 04:45

Otter v0.5.4 — Per-job desired state, and a delete that sticks

Status: release notes. Date: 2026-10-08. Previous release: v0.5.3.

v0.5.3 added otter delete --cloud and it did not work. The delete removed the
release on the runtime, the control plane still desired it, and the agent
converged the runtime back onto it within four seconds — so the job reappeared,
while the command reported success and a note claiming every release was removed.
This release fixes the cause: a runtime's desired state is now a set of jobs, one
entry per job, instead of a single release.

Read this first:

  • Deleting a job now removes it from Cloud's desired state, which is what
    stops the runtime being reconciled back onto it. Until this release, deletion
    and convergence were fighting and convergence always won, because it retries.
  • One job's deploy no longer disturbs another's. The desired state was a
    single digest per runtime, so deploying job B overwrote what job A was relying
    on. Each job is now addressed by its stable job id.
  • Cloud only reports a delete as done once the runtime confirms it. A
    removal that has not been confirmed is pending, not deleted.
  • The agent protocol gained a capability — see below. It is additive and
    gated, so nothing breaks during a rollout.

What changed

The desired-state answer from the control plane is now a snapshot:

{ "runtime_id": "…", "generation": 12, "snapshot": true,
  "jobs": [ { "job": "<durable job id>", "digest": "sha256:…", "url": "…" } ] }

The agent converges the set: it installs and activates what the snapshot
names, and removes managed jobs the snapshot has dropped. Removals are not gated,
so deleting one job never pauses its siblings. It only ever removes jobs the
runtime reports as managed, so a job whose source lives in the workspace is out
of reach by construction rather than by a rule someone has to remember.

The runtime-level generation fence is unchanged. Jobs are addressed by their
stable id.

The capability: desired.snapshot.v1

Cloud and agents deploy independently — Cloud on a push, an agent only when a
tenant image is rebuilt — so the shape is dual-served, chosen by an explicit
capability:

  • the agent declares desired.snapshot.v1 and reports the releases it holds;
  • Cloud serves the per-job shape only when the agent declares the capability
    and the runtime says it can identify and remove managed jobs;
  • otherwise the previous single-release shape is served, unchanged, and an agent
    that receives no snapshot marker removes nothing.

Adoption of per-job state is seeded from what the agent reports it already
runs
, never from an empty table — an empty snapshot means "delete every managed
job", so seeding from nothing would have ordered the release store emptied.

Three ways this could have failed silently

Recorded because each was caught late and each would have looked like success:

  • A missing snapshot: true marker. The agent keys on it, so a per-job answer
    without it decodes as the legacy shape: nothing deployed, nothing removed.
  • Confirmed by silence. An empty release list in a report could wipe the
    stored observation and read as confirmation that a removal had happened. The
    stored set is now only replaced when the field was present, it is nullable so
    "never reported" is not "reported empty", and a removal requires an observed set.
  • A stale single-release projection. After a snapshot delete, the legacy
    projection still named the deleted digest, so one poll from a not-yet-upgraded
    agent could re-install it. It is cleared before the delete.

Installing

brew upgrade otter

Linux packages are on the release page, per architecture. Upgrading the client
before the runtime matters: a runtime that becomes snapshot-capable expects the
deploy to name a job id, which older clients do not send.

Compatibility

This is the first agent-protocol change since the compatibility policy in
v0.4.0. It is additive: a new optional capability, dual-served, with old agents
receiving exactly today's shape and new agents falling back safely. No manifest,
SDK, CLI-JSON or HTTP shape that the policy froze is changed.

v0.5.3

Choose a tag to compare

@github-actions github-actions released this 08 Oct 03:50

Otter v0.5.3 — Delete a job, from the command line

Status: release notes. Date: 2026-10-08. Previous release: v0.5.2.

v0.5.3 is the client half of job deletion. v0.5.2 shipped the runtime side —
a job known only from its releases can be deleted, and the agent accepts a delete
command — but nothing in the CLI could ask for one. This release adds the verb, a
confirmation in front of it, and the rule that it names a job rather than a
directory.

Read this first:

  • Deletion is irreversible and the command now asks. It prints what it will
    destroy, names the job, and waits for y/yes. Anything else — n, blank, end
    of input — aborts with aborted; nothing was deleted and exit 1. The default is
    No, so a stray Enter does nothing.
  • A scripted delete needs --yes. Non-interactive input without it is
    refused rather than assumed: the prompt never reads a piped answer as consent,
    and it never hangs waiting for one.
  • The argument is a job name or id, never a directory. otter delete . and
    otter delete jobs/counter are refused with a usage error (exit 2), before the
    prompt and before any request. The confirmation can only name something an
    operator recognises, and a path is not that.

Deleting a job

otter delete counter                    # the local runtime
otter delete --cloud counter            # a pooled runtime, through Otter Cloud
otter delete --cloud counter --runtime rt-pool-2

--cloud resolves the credential from otter login (or OTTER_CLOUD_TOKEN),
finds the job by case-insensitive name or by id, and requires a deploy-scoped
token: a read token is refused. Deleting a job with a run in flight is refused
with the run named; cancel it first.

The runtime reports what it could not remove, and that report is printed rather
than swallowed. The case that matters: a job known only from its releases has no
tombstone, so releasing the same job id again brings it back.

What a delete removes

Every run of the job and the logs those runs produced, every captured request and
response payload, the job's stored state, every schedule it owns, its
configuration, and every release including the active one. Source files in the
jobs directory are left in place.

Limitations

  • A dedicated (non-pooled) runtime's job cannot be deleted from Cloud. The
    runtime's own delete is admin-only, and Cloud deliberately holds no admin
    credential, so the request is refused rather than half-performed.
  • The note is the only warning you get that a release-only job can reappear.
    It is carried from the runtime through the control plane to the terminal, which
    cost three hops to get right and is worth knowing about if it ever goes quiet.

Installing

brew upgrade otter

The .deb for Linux is on the release page as usual, per architecture.

v0.5.2

Choose a tag to compare

@github-actions github-actions released this 08 Oct 02:50

Otter v0.5.2 — Deploy what you released

Status: release notes. Date: 2026-10-07. Previous release: v0.5.1.

v0.5.2 makes otter deploy --cloud promote a release you already made, rather
than packaging the working directory as it goes, and adds job deletion. Deploy
stops being a build; release becomes the build.

Read this first:

  • otter deploy --cloud changed meaning. It promotes the latest local
    release
    for the job. Editing a job and deploying no longer ships the edit —
    run otter release first. --build restores the previous behaviour, packaging
    the workspace as part of the deploy.
  • Name a release when you don't want the latest. --release <digest> takes
    bare hex or sha256:<hex>; --release latest is the default spelled out.
    otter release --list shows the digests with their creation times, and "latest"
    is the newest by creation time.
  • The prepared environment is frozen at release time. A job using
    python.mode: managed prepares its interpreter and locked dependencies when you
    release, and those ship inside the release. Release on the platform the runtime
    runs on — a release built on macOS carries a macOS environment.
  • Deleting a job is new, and it is destructive. It removes the job's state,
    runs and logs, captured payloads, schedules, configuration and every release,
    including the active one. It refuses while a run is in flight.

Deploying

cd <job directory>
otter release                 # build the artifact; prints its digest
otter deploy --cloud          # promote the latest release
otter deploy --cloud --dry-run   # say what would go, send nothing

otter deploy --cloud --release <digest> promotes a specific release, which is
also how a rollback is expressed: promote the digest you were on before. A release
Cloud does not already hold is uploaded first — a copy, not a build, so no
preparation is involved and the digest cannot change on the way.

otter deploy --host <user@host> is unchanged: it is a converge that ships the
source tree for every job, and it neither requires nor produces a release.

Deleting a job

otter delete <job>            # locally

This is also available against a pooled runtime from Otter Cloud
(DELETE /api/jobs/{id}, a deploy-scoped operation). What it removes: state,
runs and logs, captured payloads, the webhook token, pause, schedules, job
configuration, and every staged release including the active link.

Two things worth knowing before using it:

  • A job with a run in flight is refused (409, naming the run) and nothing is
    removed. Cancel the run first. The check runs before any store write, so a
    refusal is never a partial delete.
  • A job that exists only as a release has no tombstone. A workspace job keeps
    its source and is remembered as deleted; a job known only from its releases is
    removed outright, so staging and activating a new release for the same job id
    makes it reappear. The response says so rather than leaving you to discover it.

Deleting a dedicated (non-pooled) runtime's job is refused: the runtime's own
delete is admin-only, and Cloud holds no admin credential by design.

Fixes

  • Job detail pages in Otter Cloud reported every job as unavailable. The
    transport asked the agent for a job.get read, which is not one of the reads
    the agent is allowed to perform; a single job is meant to be narrowed out of
    jobs.list. The job list worked throughout, which is why only the detail page
    was empty. Fixed in Cloud; no CLI change.
  • The README's install section and the changelog index had fallen behind the
    releases: the section offered only Homebrew, with no Linux path and no mention
    that apt install otter by name does not exist yet.

Known limitations

  • otter deploy --cloud reports upload: needed even for a release Cloud
    already holds. The presence probe (GET /api/releases/<digest>) is not served
    by the control plane yet, so the client cannot confirm presence and uploads
    idempotently. The upload is harmless; the report is imprecise.

v0.5.1

Choose a tag to compare

@github-actions github-actions released this 07 Oct 22:31

Otter v0.5.1 — Ubuntu packages

Status: release notes. Date: 2026-10-07. Previous release: v0.5.0.

v0.5.1 adds a Debian package to the release artifacts. v0.5.0 shipped
tarballs only, so a Linux install meant downloading an archive and placing two
binaries by hand: nothing could upgrade them and nothing could remove them.

Read this first:

  • The .deb installs otter and otterd into /usr/bin. If you already
    put them on your PATH from a tarball — ~/.local/bin, most likely — your
    shell may keep resolving that copy, because it comes first in PATH. Remove
    it, or leave its directory later in PATH, or otter version will disagree
    with dpkg -l otter.
  • macOS is unchanged. The cask in the tap still installs the tarballs, and
    this release updates it to 0.5.1 in step.

Installing on Ubuntu

curl -LO https://github.com/tkoizumi/otter/releases/download/v0.5.1/otter_0.5.1_linux_amd64.deb
sudo apt install ./otter_0.5.1_linux_amd64.deb

apt records the package, so apt remove otter works, and the two binaries are
versioned with the rest of the system rather than living in a home directory.

Nothing else changes in this release: no runtime, protocol or manifest surface
is touched, and no migration runs.

What this is not yet

A hosted APT repository. There is no otter package in an archive you can
apt install otter from by name, and no signing key to trust — so upgrades do
not flow through apt upgrade yet, and each release is still a download from the
release page. That repository is the next step; this release is the artifact it
will carry.

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 07 Oct 22:21

Otter v0.5.0 — Self-serve deploys to Otter Cloud

Status: release notes. Date: 2026-10-07. Previous release: v0.4.0.

v0.5.0 makes a job deployable by the person who wrote it: otter login against
Otter Cloud, otter deploy --cloud --job <name> to package and promote one job
to a runtime the organization owns, and a pooled tenant that runs it. The runtime
half of the pipeline already existed; this release is the client half and the
hosting that makes it real.

Read this first:

  • The tenant container changed shape. A pooled tenant is now one
    container running two processes: otterd as uid 10001 and the agent as uid
    10002, over loopback. The separate sidecar container is gone, and an existing
    tenant must be re-provisioned rather than upgraded in place.
  • otter login needs a token minted in Otter Cloud; there is no browser or
    device flow. It is read from --token, OTTER_CLOUD_TOKEN, or a terminal
    prompt, and stored per machine under ~/.otter/cloud.json (0600).
  • Assigning a runtime is an API call. POST /api/runtimes creates the
    organization's assignment and returns the per-runtime bootstrap secret once;
    it is session-authenticated and has no dashboard page yet.

Deploying a job

otter login --cloud https://app.runotter.dev --token <token>
cd my-workspace
otter deploy --cloud --job <name>

deploy --cloud packages one job with the same release machinery otter release uses, so the digest it promotes is the canonical release identity, and
asks the control plane to move the assigned runtime onto it. --dry-run and
--runtime are honoured; otter login --status reports the stored identity
without deploying.

A deploy installs and activates a release. It does not run anything: the run is
a separate act, and on a pooled runtime it is triggered from the job's page in
Otter Cloud.

One container per tenant, two uids

The agent must reach otterd over loopback, and gVisor cannot give it that
across containers: it does not honour --network=container:<other> (a joining
container comes up with only lo), and a container's listener is not reachable
across a gVisor bridge either. So both processes live in one container and
share a network namespace by construction. otterd is loopback-only and needs
no API token, and nothing is exposed on any bridge.

The isolation the separate sidecar existed for is kept by uid separation, and it
is why the container starts as root:

Process uid Access
otterd 10001 owns the tenant volume, mode 0700
agent 10002 its own tmpfs, cannot read or write the tenant volume

Root is needed only to drop to those two uids; the container is granted exactly
SETUID, SETGID and CHOWN, and runs with no-new-privileges. After the drop
no process holds a capability. The agent's Cloud credential is stripped from
otterd's environment, so the tenant process cannot read it.

What the agent does differently

  • Released jobs are discovered. A runtime whose workspace is empty still
    resolves the job a release was installed for, by name and by id, so a deploy
    to a pooled tenant is runnable without any source in /workspace.
  • Applying is idempotent. A control plane publishes desired state and keeps
    returning the same generation; the agent now compares the content it observes
    against what is desired and does nothing when they match. Before this, a stable
    desired state re-downloaded and re-activated the same release every few
    seconds and leaked a staging directory on each pass.
  • Polling is paced, and its decisions are logged to stderr — which in a
    container is the container log. The agent used to be silent, which made a
    stalled deploy indistinguishable from a working one.

Fixes worth naming

  • Installing a release that was already in the store left its staging directory
    behind; cleanup is now unconditional.
  • The pool resolver now binds and checks every tenant gateway. It previously
    reloaded a unit that declares no ExecReload, so a newly added tenant's
    gateway was never bound, and the health check probed only the first gateway —
    reporting healthy while the new tenant's agent could not resolve anything.
  • The agent data directory is chmoded before it is chowned; the other order
    needs CAP_FOWNER, which this container deliberately does not have.

Installing

Homebrew (macOS):

brew install tkoizumi/tap/otter

The cask installs the two prebuilt binaries, otter and otterd; no Go
toolchain is needed.

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 04 Oct 17:14

Otter v0.4.0 — The interface freeze

Status: release notes. Date: 2026-10-03. Previous release: v0.3.0.

v0.4.0 completes the runtime half of Track R: a scoped control credential,
first-class dynamic schedules, a reachable-but-scoped remote API, pinned job
configuration, and the published compatibility policy that freezes the manifest
schema, the Python SDK, CLI JSON and the HTTP API.

Read this first:

  • It ships three schema migrations (0013_api_tokens,
    0014_dynamic_schedules, 0015_job_configs). An upgrade runs them on first
    start, and a rollback is a restore, not a binary swap: a newer schema is
    refused by an older binary by design.
  • Two shapes break. A cron expression now means UTC rather than the host's
    local time, and every list response is an object rather than a bare array.
    Both changed deliberately, before the compatibility promise takes effect, and
    both are recorded in compatibility.md.
  • The runtime contract is version 2 and reached from
    GET /v1/version or otter --json version.

What breaks, and what does not

Surface Before v0.4.0
trigger.cron time zone the host's local time UTC; use a schedule's timezone for a local wall-clock meaning
GET /v1/jobs, /v1/runs, /v1/runs/{id}/logs, /v1/tokens, capture requests a bare array, or an object with no version {"schema_version": 1, "jobs": [...]} and the matching key per endpoint
otter jobs --json, otter runs --json a bare array the same versioned envelope as the HTTP API
PUT/DELETE /v1/jobs/{id}/schedule, otter schedule set|clear the supported cadence surface deprecated; still work, removed in the next major
A manifest-declared trigger.cron the API could overwrite it it is manifest-owned; PATCH/DELETE/legacy PUT refuse it with 409

A job that reads ctx.trigger, ctx.state or its environment needs no change.
A script that parses a list response does: add .jobs / .runs / .logs /
.tokens / .requests / .schedules. The old line-oriented CLI output is
unchanged.

Scoped control credential (CL-21)

Named API tokens with read, control and capture scopes, minted by
otter token create, stored as a SHA-256, revocable with immediate effect (the
row is read on every request). Full authority stays the static OTTER_API_TOKEN;
a named admin token is deliberately not offered. capture is control plus the
two payload routes, so a control plane holds one credential that can both
re-run a job and read what it sent; it is deliberately not implied by control,
so a credential minted before the scope existed does not silently gain access to
the client's traffic. The route-by-route boundary is in
api-reference.md.

Dynamic schedules (CL-22)

A schedule is now a first-class, id-keyed runtime object, not one cadence per
job buried in a manifest.

  • GET/POST /v1/jobs/{id}/schedules, GET/PATCH/DELETE /v1/schedules/{id}, and pause/resume; otter schedule list|add|update|remove|pause|resume.
  • A per-occurrence payload becomes ctx.trigger.body, so "this job, for
    dataset 42" is expressible without a release.
  • Idempotency-Key on create: a retry returns the existing schedule.
  • Exactly one run per occurrence, across restarts and duplicate wake-ups: the
    occurrence ledger (schedule_fires) is written in the same transaction as the
    run.
  • A manifest's trigger.cron reconciles as a manifest-owned row that the API
    cannot change; an API-created schedule is never touched by a reload.

Known limit: the missed-occurrence policy accepts only skip.
coalesce and catch_up are reserved names and are refused with 400 rather
than accepted and ignored (OT-007).

Reachable, scoped remote surface (CL-23)

otter deploy accepts a wildcard or public --listen behind an explicit
--allow-remote-bind and prints a warning; a private bind warns without the
opt-in. assert-host-permissions.sh asserts the configured posture and
requires OTTER_API_TOKEN for any non-loopback API. TLS remains a documented
reverse-proxy pattern (security.md);
the daemon still serves plain HTTP behind it.

Job configuration

A job's configuration is the layer between the release and the run: a store
name, a dataset id, a backfill date. It is versioned and immutable, and every
run pins the version it resolved at submission, so a queued, retrying or
backlogged attempt keeps its values.

  • GET/PUT /v1/jobs/{id}/config; admin-only to write in v0.4.0.
  • The child reads ctx.config (a dict; {} when unconfigured), passed as
    OTTER_CONFIG.
  • Not a secret store: values are returned to any scoped reader and are not
    projected into named environment variables in this release.

Interface freeze (R4)

compatibility.md publishes what makes a change breaking
for each frozen interface, how a deprecation is retired (one minor release
minimum, removal only in a major), and the current deprecation table.
GET /v1/version and otter --json version report schema_version,
contract_version, the manifest schema, the SDK version and the supported
platforms in one document. Contract tests pin the version document, the platform
list and the frozen JSON keys of every machine document.

Also in this release

  • OT-010: release retention and a submission binding a release now serialize
    on the database write lock, so a prune cannot strand a queued run.
  • OT-024: the CloudWatch metric contract is one file,
    monitoring-contract.json, asserted on both the runtime and the platform
    side.
  • OT-029: a retired job whose source is still in the repository no longer
    fails a full deploy; the release step skips and reports the suppressed path.
  • The runtime contract gains §7 "Honest limits": uncoordinated concurrent
    updates, retry effect duplication, no downgrade, plain-HTTP transport, and
    configuration not being a secret store.

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 02 Oct 18:31

Otter v0.3.0 — Operable unattended

Status: release notes. Date: 2026-10-02. Previous release: v0.2.0.

v0.3.0 closes the first half of Phase 2. It is a minor release: it adds
capabilities and documented semantics and corrects documents. Two facts a reader
must know before upgrading:

  • It ships two schema migrations (0011_job_terminology, 0012_job_schedules).
    An upgrade therefore runs migrations on first start, and a rollback is a
    restore, not a binary swap.
  • The integration → job rename is public. The manifest file and every
    manifest field are unchanged, but the names a client or a child process reads
    are not.

Read this first: the rename is on the wire

Surface v0.2.0 v0.3.0
HTTP and CLI JSON integration_id job_id
Child environment OTTER_INTEGRATION_ID, OTTER_INTEGRATION_NAME, OTTER_INTEGRATION_DIR OTTER_JOB_ID, OTTER_JOB_NAME, OTTER_JOB_DIR
Daemon configuration OTTER_INTEGRATIONS_DIR OTTER_JOBS_DIR
Manifest file and fields otter.yaml, name/entrypoint/… unchanged

What actually breaks, and what does not:

  • JSON keys changed. A dashboard or script that parses integration_id out
    of otter runs --json or the HTTP API must read job_id. Migration 0011
    renames the affected database columns; the rows are preserved.
  • The child environment did not lose the old names. The executor exports
    OTTER_JOB_ID/_NAME/_DIR and OTTER_INTEGRATION_ID/_NAME/_DIR, the
    latter deliberately, so a job inside an immutable release written before the
    rename keeps working. The old names are no longer the documented ones, but a
    job that reads them needs no change.
  • The daemon configuration names did change. OTTER_INTEGRATIONS_DIR and the
    --integrations flag are no longer read; use OTTER_JOBS_DIR / --jobs.

The embedded Python SDK reads the new names, so a job that uses otter.Context
needs no change either way.

You can watch an upgrade

The operations guide told an operator to watch for migration_applied on
restart, and database.Migrate had no logger, so a successful upgrade was
silent — the one moment the schema moves was the one moment nothing was said.

  • One migration_applied record per applied migration, at info, in the
    order they were applied, carrying the migration's version and name
    (name is the filename, which is also its description; no new column was
    added).
  • A restart with nothing pending logs nothing, so "no migration lines" means
    the schema did not move rather than that nobody looked.
  • A failed migration is named before the process exits: migration_failed
    at error, with the version and name, and the schema stays at the last
    version that committed. A repaired migration can be shipped and the next start
    continues from there.

You can operate it unattended

/health answered "how many?" and none of "how old?", "which job?", "did it
stop succeeding?", "is the disk filling?". One integer of queue depth cannot tell
a busy queue from one stuck since yesterday, and a job failing since its last
success still showed healthy totals.

For an authenticated caller only, /health now carries:

  • queue — the age of the oldest run that is claimable now
    (available_at <= now), the queue depth per job, the count of runs parked by
    retry backoff, and when the soonest of them is due. Retries are reported
    separately from the age on purpose: they wait on a clock, not on capacity.
  • freshness — each job's last success and its age, from one grouped read.
    A job that has never succeeded omits both fields; that absence is the signal.
  • storage — db_bytes (page_count * page_size, so the WAL is excluded)
    and the data directory's free and total bytes. The guide previously made
    "alert when otter.db exceeds about 2 GB" a host-level check, which a remote
    operator cannot perform; now they can.

Each block degrades on its own and is logged when it does. The unauthenticated
liveness body is unchanged, field for field, so a caller without a token learns
nothing new. otter status prints the same signals, and otter jobs --schedule
renders the daemon's last success instead of listing runs once per job.

Retention is automatic, and a deploy prunes releases

  • Runs and run logs expire on their own windows (--run-retention,
    --log-retention), with --capture-retention already covering HTTP payloads.
    Every window is opt-in and defaults to retaining forever, and the tracked
    windows are committed in otter.deploy.yaml so a rebuilt host cannot silently
    retain forever. The documented manual-pruning SQL is corrected to filter on
    runs.created_at; it was filtering on a column that does not exist.
  • otter deploy prunes inactive releases with the same --keep window a
    release uses (default 3, 0 keeps everything), and every prune path refuses
    to act when the pin set is unreadable
    . A queued or retried attempt keeps its
    bound release whatever happens to the release directory.

The backlog answer is written down

"What happens if a run outlasts its next scheduled run?" runs opposite to
intuition, and it was written nowhere. operations.md §"Backlog behavior" now
states all five consequences: every cron occurrence becomes its own run
unconditionally; with concurrency: 1 they serialize into a catch-up backlog;
downtime skips while slowness accumulates (occurrences during downtime are
never replayed — the linked contract section is an explicit non-guarantee);
queued work executes the release bound at submission, so a deep backlog can run
older code than what is active; and raising concurrency produces genuinely
concurrent executions. No behavior changed.

A downgrade is refused, and the procedures are drills

  • A database migrated by a newer binary is refused. Migrations are
    append-only, so a binary that does not embed the database's newest migration
    exits with migration_failed naming both versions, before anything reads or
    writes the schema. Before this release, "downgrades are not supported" meant
    "starts, then fails on the first query that touches a renamed column".
    Rolling back remains a restore of the pre-upgrade backup.
  • The operating procedures are executable. make drill runs drills that
    assert their own outcome: backup-restore (a hot backup restores onto a
    different, empty data directory and the job runs again), upgrade (the
    previous release's schema upgrades observably, and the runtime works after),
    downgrade-refusal, secret-rotation, fresh-archive (a release archive
    installs outside the checkout and runs a job), deploy-failure, unit-caps,
    disk-pressure and liveness. Eight of the nine document a DRILL_SABOTAGE
    mode that removes the behavior under test, so a green run is demonstrably
    falsifiable; unit-caps is the exception, falsified instead by the UnitFile
    render-test mutation in
    evidence/phase-0/2026-09-29-p0-04-unit-caps-mutation.txt,
    with a drill-level mode tracked as OT-030.
  • A retry keeps its bound release and environment. Activating a newer
    release, changing the interpreter, or changing the retry policy while a retry
    is pending cannot move that retry onto different code, a different
    environment, or a different policy; a pending backlog also keeps its digest
    through a release prune. This is now a guarantee in
    runtime-contract.md,
    pinned by TestRetryExecutesTheParentsReleaseSnapshot,
    TestRetryResolvesTheParentsManagedEnvironment,
    TestPendingBacklogKeepsItsDigestThroughAReleasePrune and
    TestRecoveryPlansRetriesFromTheBoundRelease.
  • The trust-boundary claims are asserted, not restated: the data directory
    is created owner-only, a non-loopback bind without a token is refused, the
    unauthenticated /health body carries only status, version and uptime, a
    per-run token cannot read another job's run, logs or state, and a webhook
    token authorises only its own job's hook and is never recorded on the run.

Also in this release

  • A detached start forwards the flags it accepts. otter start --detach
    rebuilt the child's arguments from --jobs, --data and --listen only, so
    every other flag was silently dropped — including --api-token, which brought
    the runtime up with its API unauthenticated while --help promised the flag
    was forwarded. --workers, --log-format and the rest were dropped the same
    way. Present since v0.2.0; found by the v0.3.0 exit gate's document checks.
    The detached path now builds its arguments the way the foreground path always
    has.
  • Schedules are runtime state. otter schedule set|clear|show (and
    PUT/DELETE /v1/jobs/{id}/schedule) change a job's cadence without editing
    the manifest or restarting; a cleared schedule is deliberate and does not fall
    back to the manifest (migration 0012).
  • A deploy can bind the API to a private address, not only loopback.
  • Managed Python is harder to misconfigure: a lock the configured package
    index cannot satisfy is refused, preparation has an explicit route with an
    egress preflight, and the generated unit defaults MemoryHigh to off so
    MemoryMax is what kills a runaway.

Upgrade from v0.2.0

  1. Back up first — the complete backup, not just the database
    (Backups). An upgrade runs migrations on first
    start.
  2. Stop the daemon, replace both binaries, start it, and watch the journal for
    one migration_applied per migration
    (0011, 0012).
  3. otter status, otter jobs, otter runs --all --limit 10.
  4. Release every job once. The release layout is versioned and the new
    implementation deliberately does not reuse a snapsh...
Read more

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 28 Sep 21:02

Otter v0.2.0 — Dependable execution

Status: release notes. Date: 2026-09-28. Previous release: v0.1.19.

v0.2.0 is a minor release. It closes Phase 1 of the
product roadmap: accepted work is accounted for, even
after a hard kill. It changes runtime behavior and publishes a new document;
it ships no schema migration and no manifest, CLI-JSON or HTTP API shape
change. Read the semantics below before upgrading.

Recovery is complete

v0.1.19 stopped recovery from losing accepted work; v0.2.0 closes the
remaining gaps, so interrupted or newly claimed runs can no longer be stranded.

  • Every interrupted run is recovered. Startup reads all running rows,
    not the first 50. The old listing limit silently capped recovery at 50 runs,
    so a busy crash left the rest permanently non-terminal. Recovery now pages the
    whole status. (v0.1.19; TestCrashRecoveryHandlesMoreThanOneListingPage,
    TestReconcileQueueReenqueuesMoreThanOneListingPage.)
  • A claimed run that cannot start returns to the queue. If loading the run
    or marking it running fails on a transient database error, the attempt is
    re-enqueued with a 5s backoff instead of sitting as queued with no queue row
    until the next restart. A run whose record no longer exists is dropped once,
    at warn. (v0.1.19; TestClaimedRunIsRequeuedWhenMarkRunningFails.)
  • Capacity is not leaked. A claim that fails to commit releases the worker
    slot it reserved, so a daemon no longer quietly runs fewer workers than
    configured. (v0.1.19; TestClaimCommitFailureReleasesCapacity.)
  • Finish and retry are one durable decision. The terminal status, the
    successor attempt and its queue row are written in a single transaction. A
    crash between them can no longer leave a terminal failure whose retry is
    missing, and a Finish write that fails after a successful child is recorded
    in a fallback journal and applied on the next start rather than re-running the
    child. (5481e1c; TestFinishCommitFailureLeavesNoPartialOutcome,
    TestFallbackJournalIsAppliedOnRestart.)

Startup can now refuse

A startup that could not repair interrupted work used to log the error and open
for new work, turning one transient database error into permanently stranded
runs. otterd is now fail-closed: if crash recovery or queue reconciliation
cannot complete, startup returns an error and the process exits.

  • The safe failure is a daemon that stays down and visible, not one that serves
    while accepted work sits stranded. This is the user-visible change: a bad
    database now stops the daemon instead of letting it run degraded.
  • The one escape hatch is explicit: --allow-incomplete-recovery
    (OTTER_ALLOW_INCOMPLETE_RECOVERY=true) starts anyway and logs the error at
    error level, for an operator who must bring a daemon up to inspect or repair
    a database it cannot read.
  • A failure to persist the repair of any interrupted run counts, not only a
    failure of the initial read.

(ec01900; TestStartupFailClosedOnIncompleteRecovery.)

Child lifetime is platform-specific

  • Linux: a child cannot outlive an abruptly killed daemon. The child is
    launched with Pdeathsig: SIGKILL, so kill -9, an OOM kill or a panic kills
    the direct child before its retry can overlap it. This is the guarantee that
    stops duplicated external effects after an abrupt death.
  • macOS: the exposure is now stated, not implied. Darwin has no
    parent-death signal. After an abrupt daemon death the in-flight child is
    reparented and keeps running while startup marks the run failed and retries
    it, so non-idempotent work can run twice. Closing this needs a supervisor
    process or a death-watch pipe inside the child; it remains unscheduled.
  • The graceful paths — timeout, cancellation and daemon shutdown — are
    unchanged and still signal the whole process group on every platform.

(2220509; TestChildDiesWhenDaemonIsKilled, Linux only.)

The runtime contract is published

runtime-contract.md (contract version 1) is the
versioned statement of what the runtime promises and, just as importantly, what
it does not. It pins the attempt state machine, what an accepted request
promises, missed cron windows, duplicate webhook delivery, state-write
concurrency, retry effect duplication, and the platform-specific guarantees. It
is backed by the WS4 fault matrix, whose evidence status is stated per scenario.

Two honest non-guarantees worth reading before relying on them:

  • Exactly-once is not promised. Execution is at-least-once in effect; a
    retry can repeat external effects.
  • A retry's bound release and environment are implemented but not yet
    independently proven.
    A retry is recorded with its parent's release and
    environment and is expected to re-use them, but no automated scenario proves
    the retry half yet; it is tracked as OT-011 in
    open-work.md.

(b5a4001.)

Upgrade from v0.1.19

  • No schema migration. No migration shipped between v0.1.19 and v0.2.0,
    so upgrade and downgrade are a binary swap. Keep the usual database backup for
    any operational change, but no data conversion runs.
  • No manifest, CLI-JSON or HTTP API shape change. The only new flag is
    --allow-incomplete-recovery / OTTER_ALLOW_INCOMPLETE_RECOVERY.
  • Startup can now fail where it previously warned. A daemon whose database
    cannot be repaired will not start. That is the intended change; use
    --allow-incomplete-recovery only to inspect or repair, not to keep serving.
  • Linux binaries now use Pdeathsig. The behavior change is visible only
    after an abrupt daemon death.

Verification

make test, make smoke, make lint, goreleaser check and
goreleaser release --snapshot --clean pass on the tag commit. The release
workflow runs the same test and smoke jobs before GoReleaser publishes, so a
tag cannot publish a binary that fails the loop a user performs.

v0.1.19

Choose a tag to compare

@github-actions github-actions released this 28 Sep 16:26

Changelog

  • f066fa3 Fix capacity slot leak
  • ffb5940 Fix default limit of 50 runs for recovery
  • 280c312 Re-enqueue claimed runs when an attempt cannot start

v0.1.18

Choose a tag to compare

@github-actions github-actions released this 28 Sep 04:54

Changelog

  • 98df724 Add integration pause and refine run inspection