Repository navigation
Release v0.2.0
agent-eval-generator v0.2.0
Released 2026-10-25. A minor release. Two security fixes since v0.1.0, from the
2026-10-21 security review: the service token is compared in constant time and turns are refused
when it is unset, and a turn's Lake credential goes only to the service's own LAKE_URL. Private
repository and image, docker.io/oramasearch/agent-eval-generator:v0.2.0. Tracking issue:
#15.
This release is the tree of dev at d174d10 with these notes, the version and two README
corrections.
What it includes
Every pull request merged into dev since v0.1.0:
| PR | What it changes |
|---|---|
| #11 | Docs: records the v0.1.0 image digest in the v0.1.0 notes |
| #12 | POST /v1/turn compares the bearer with hmac.compare_digest and fails closed: with AGENT_SERVICE_TOKEN unset or empty every turn is refused with 503; a wrong or missing bearer is still 401. serve() warns at start when the token is missing (#8) |
| #14 | A turn's lake_token is sent only to the service's own LAKE_URL (an absolute http(s) URL with no user info): the registry reads (<LAKE_URL>/v1/workspace_query) and the model gateway (<LAKE_URL>/openai/v1) both use it, and a turn's lake_url is ignored. With LAKE_URL unset or malformed, a turn is refused before any request (#13) |
The release branch also corrects two README lines: the command-line example uses --cases (it said
--profiles), and the layout no longer lists a grader under evals/ or a reports/ folder, which do
not exist (v0.1.0 known issue 5).
Before it rolls: the settings it needs
AGENT_SERVICE_TOKENmust be set, orPOST /v1/turnrefuses every turn with503.LAKE_URLmust be set to Lake's address, or every turn is refused. A turn'slake_urlis no
longer read.
The Orama operator already renders both for every library agent (with AGENT_MODEL), so a Lake run
by the operator needs no new setting. A service run by hand needs both.
What it does not include
Unchanged from v0.1.0: no grading by Zetta's deterministic
graders (zetta#987), no capture-state evidence, no
model tools, no Hugging Face upload from a turn, and committed profiles that mostly predate the
registry.
Turn traces are still written to the pod's disk (TRACE_DIR, /tmp/traces under the operator).
The other agents' "no turn trace on disk" fix from the same review was not made in this repository;
the traces are lost on restart and nothing stores them in the workspace Zetta.
Known issues
Security-relevant first. All open at this tag.
- Workspace identity is not passed through
(#6, part of
lake#311). Registry reads send onlycustomer_id;
the task'sworkspace_idis ignored. The same issue asks for idle backoff on the bus. - Turn traces on the pod's disk (above): a trace holds the model's output for a person's request
and stays in the pod's/tmpuntil it restarts. - The lease is kept by activity only: no heartbeat of its own; a single model call longer than
lease_seconds(600 by default) could let the turn be reclaimed. Not observed. - Library agent page and chat in Lake are still open
(lake#263).
v0.1.0's known issue 2 (the bearer compared with !=, open when unset) is fixed by #12, and issue 5
(the README's example) by this release.
Where it has run
The v0.2.0 tree has not run on the dev environment. Every Lake runs the operator's default,
oramasearch/agent-eval-generator:v0.1.0 (orama-operator internal/controller/lake_defaults.go on
dev); no LakeSet member pins a dev build of this agent. After the tag, the release image goes to
testing first, behind its gate, then staging and prod.
The image
Built from the tag v0.2.0 (8b40218) on the cluster BuildKit with build-agent.sh (linux/amd64, the repository's Dockerfile), then tagged :v0.2.0 on that digest without rebuilding:
docker.io/oramasearch/agent-eval-generator:8b40218@sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284, also tagged :v0.2.0 (same digest). The Docker Hub repository answers 404 anonymously (private).
It is not rolled anywhere yet: pinning it on an environment is a separate change to the operator's set files, through dev, testing, staging and prod in order.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) |
36 tests: 36 passed, 0 failed, 0 skipped. tests/test_service.py 8, tests/test_service_auth.py 13 (#12), tests/test_turn_destinations.py 8 (#14), tests/test_lake_bus.py 4, tests/test_release.py 3 (the installed package is 0.2.0, these notes exist and are indexed, /healthz reports it) |
| A live turn (registry reads, model gateway, answer on the bus) | Not run for this release (needs a live model and a Lake) |
| Linux image build (cluster BuildKit, from the tag) | Built from v0.2.0 with build-agent.sh: oramasearch/agent-eval-generator:8b40218, also :v0.2.0, sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284; private (404 anonymously) |