Skip to content

Release v0.2.0

Choose a tag to compare

@homer6 homer6 released this 25 Oct 21:07
· 19 commits to main since this release
8b40218

agent-eval-generator v0.2.0

Released 2026-10-25. A minor release. Two security fixes since v0.1.0, from the
2026-10-21 security review: the service token is compared in constant time and turns are refused
when it is unset, and a turn's Lake credential goes only to the service's own LAKE_URL. Private
repository and image, docker.io/oramasearch/agent-eval-generator:v0.2.0. Tracking issue:
#15.

This release is the tree of dev at d174d10 with these notes, the version and two README
corrections.

What it includes

Every pull request merged into dev since v0.1.0:

PR What it changes
#11 Docs: records the v0.1.0 image digest in the v0.1.0 notes
#12 POST /v1/turn compares the bearer with hmac.compare_digest and fails closed: with AGENT_SERVICE_TOKEN unset or empty every turn is refused with 503; a wrong or missing bearer is still 401. serve() warns at start when the token is missing (#8)
#14 A turn's lake_token is sent only to the service's own LAKE_URL (an absolute http(s) URL with no user info): the registry reads (<LAKE_URL>/v1/workspace_query) and the model gateway (<LAKE_URL>/openai/v1) both use it, and a turn's lake_url is ignored. With LAKE_URL unset or malformed, a turn is refused before any request (#13)

The release branch also corrects two README lines: the command-line example uses --cases (it said
--profiles), and the layout no longer lists a grader under evals/ or a reports/ folder, which do
not exist (v0.1.0 known issue 5).

Before it rolls: the settings it needs

  • AGENT_SERVICE_TOKEN must be set, or POST /v1/turn refuses every turn with 503.
  • LAKE_URL must be set to Lake's address, or every turn is refused. A turn's lake_url is no
    longer read.

The Orama operator already renders both for every library agent (with AGENT_MODEL), so a Lake run
by the operator needs no new setting. A service run by hand needs both.

What it does not include

Unchanged from v0.1.0: no grading by Zetta's deterministic
graders (zetta#987), no capture-state evidence, no
model tools, no Hugging Face upload from a turn, and committed profiles that mostly predate the
registry.

Turn traces are still written to the pod's disk (TRACE_DIR, /tmp/traces under the operator).
The other agents' "no turn trace on disk" fix from the same review was not made in this repository;
the traces are lost on restart and nothing stores them in the workspace Zetta.

Known issues

Security-relevant first. All open at this tag.

  1. Workspace identity is not passed through
    (#6, part of
    lake#311). Registry reads send only customer_id;
    the task's workspace_id is ignored. The same issue asks for idle backoff on the bus.
  2. Turn traces on the pod's disk (above): a trace holds the model's output for a person's request
    and stays in the pod's /tmp until it restarts.
  3. The lease is kept by activity only: no heartbeat of its own; a single model call longer than
    lease_seconds (600 by default) could let the turn be reclaimed. Not observed.
  4. Library agent page and chat in Lake are still open
    (lake#263).

v0.1.0's known issue 2 (the bearer compared with !=, open when unset) is fixed by #12, and issue 5
(the README's example) by this release.

Where it has run

The v0.2.0 tree has not run on the dev environment. Every Lake runs the operator's default,
oramasearch/agent-eval-generator:v0.1.0 (orama-operator internal/controller/lake_defaults.go on
dev); no LakeSet member pins a dev build of this agent. After the tag, the release image goes to
testing first, behind its gate, then staging and prod.

The image

Built from the tag v0.2.0 (8b40218) on the cluster BuildKit with build-agent.sh (linux/amd64, the repository's Dockerfile), then tagged :v0.2.0 on that digest without rebuilding:

docker.io/oramasearch/agent-eval-generator:8b40218@sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284, also tagged :v0.2.0 (same digest). The Docker Hub repository answers 404 anonymously (private).

It is not rolled anywhere yet: pinning it on an environment is a separate change to the operator's set files, through dev, testing, staging and prod in order.

Verification

Proof Result
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) 36 tests: 36 passed, 0 failed, 0 skipped. tests/test_service.py 8, tests/test_service_auth.py 13 (#12), tests/test_turn_destinations.py 8 (#14), tests/test_lake_bus.py 4, tests/test_release.py 3 (the installed package is 0.2.0, these notes exist and are indexed, /healthz reports it)
A live turn (registry reads, model gateway, answer on the bus) Not run for this release (needs a live model and a Lake)
Linux image build (cluster BuildKit, from the tag) Built from v0.2.0 with build-agent.sh: oramasearch/agent-eval-generator:8b40218, also :v0.2.0, sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284; private (404 anonymously)