Skip to content

Satchel v0.2.0

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 12 Nov 00:08
· 37 commits to main since this release
97f2354

Satchel 0.2.0

Eval suites can now run on a schedule, and a GitHub Action previews context repo pull requests,
runs the publish gate on them and publishes on merge.

Status: pre-alpha. Expect breaking changes between 0.x releases.

Install

uv tool install satchel-server     # or: uv tool upgrade satchel-server
satchel init --example jaffle
satchel up

Container image and Helm chart:

docker pull ghcr.io/getsatchel/satchel:0.2.0
helm install satchel oci://ghcr.io/getsatchel/charts/satchel --version 0.2.0 -n satchel --create-namespace -f my-values.yaml

This release ships with satchel-adapter-sdk 1.2.1 (IR 1.0.0).

New

  • Scheduled eval runs. Add schedule: { interval_hours: N } to a suite and satchel worker
    runs it against the latest published release every N hours, on its own thread
    (SATCHEL_EVAL_TICK_MIN, default 15; 0 turns it off). The Suites tab shows the schedule and
    the next run, Run history labels scheduled and gate runs, and a scheduled run links
    Compare with previous. A suite that needs an LLM is skipped while no model is configured.
    See Evals.
  • GitHub Action (getsatchel/satchel/action@v0.2.0). On a pull request it uploads the context
    folder, gets a preview, runs the gate's suites, comments the comparison and fails the check on
    a regression. On a push to the default branch it imports and publishes. See
    GitHub Action.
  • Context repo uploads.
    • POST /v1/sync/preview (curator) builds a pull-request preview without changing drafts or
      what is served. Such previews are published by merging, not from the UI.
    • POST /v1/sync/import (admin) is satchel sync import over HTTP. Sources declared in the
      config are not synced from an upload.
    • Both calls are audited as sync.preview and sync.import.
  • Evals documentation. Evals covers suites, graders, runs, the
    publish gate and schedules.

Changes

  • Eval suite files whose names start with . (for example macOS ._* metadata) are ignored on
    import.
  • New context versions are numbered after the highest existing one, so they never collide with
    versions a pull-request preview created.
  • CI runs only the jobs a pull request can affect (contributors).

Upgrading from 0.1.1

uv tool upgrade satchel-server
satchel db upgrade

The migration 0013 adds release.ref. Run satchel db upgrade (or let the Helm chart's
migration job run) before starting the new version. The worker starts the eval schedule thread
by default; suites run on a schedule only once you add schedule: to them, so nothing changes
until then.