Repository navigation
Releases
Release v0.3.0
Released 2026-11-16. A minor release. Since v0.2.0: model-written profiles are checked before anything is written (safe names, bounded size), the licence, the security policy and base images pinned by digest. Image: docker.io/oramasearch/agent-eval-generator:v0.3.0, not published (see Known issues).
This release is the tree of dev with these notes and the version.
What it includes
Every pull request merged into dev since the v0.2.0 tag:
| PR | What it does |
|---|---|
| #27 | MIT LICENSE and COPYRIGHT (OramaSearch, Inc.), an SPDX-License-Identifier: MIT line in every source file (tools/spdx-headers.py), THIRD_PARTY_LICENSES, CONTRIBUTING.md, CLA.md, CODE_OF_CONDUCT.md, and the SBOM (sbom.spdx.json, sbom.cdx.json, tools/sbom.py) |
| #24 | Security: a model-written profile is checked before anything is written or generated (safe names, bounded size) |
| #25 | The Dockerfile names its base images by digest as well as by tag (python:3.12-slim, ghcr.io/astral-sh/uv:0.11.7), so a later build of the same source gets the same base |
| #23 | SECURITY.md: how to report a vulnerability |
| #19 | Records the v0.2.0 image digest in the v0.2.0 notes (documentation only) |
| #18 | The v0.2.0 release promotion (dev → main), merged back into dev |
A model-written profile is checked before anything is written — #24
- The model's
profile_idand table names were used as file paths, so a turn could write and delete outside its temporary folder in a pod every workspace shares (#21).synthesize()had no limit on tables or columns, so one turn could use up the pod's memory (#22). - New
limits.check_profile()runs right after parsing, inservice.run_turn, in the batch command line and again insynthesize(). Names that become paths must match^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$; a profile has at most 12 tables, 64 columns per table, 128-character column names, no table declared twice, and at most 5,000,000 generated cells. - A refused profile is answered
invalid profile: <reason>and its trace graded failed. records.package()andTable.write_jsonl()resolve every path inside their directory and refuse one outside it; an oldrecords/is removed only when it is a real directory inside the output folder, never through a symbolic link.
The licence — #27
Everything in this repository that Orama wrote is under the MIT licence. Contributors sign CLA.md once. THIRD_PARTY_LICENSES lists every runtime dependency with its licence text; none is incompatible with MIT distribution.
Before it rolls: the settings it needs
Nothing new. The variables are unchanged from v0.2.0.
What it does not include
Deferred, not in this release:
- #26: claiming bus tasks by reference and naming the task on every call; it needs the matching Lake change.
- #28: the public-tree cleanup, which also makes the Hugging Face upload target a setting (
EVAL_PROFILES_DATASET) and so changes the batch command line's behaviour.
Otherwise unchanged from v0.2.0.
Known issues
Security-relevant first. All open at this tag.
- #29 (release blocker for publishing): the image is built on
python:3.12-slim, which bundles GPL and LGPL userland, so the image is not published. It is built and kept in the project's non-public Docker Hub repository until it moves to a scratch base with static binaries. - The public-tree cleanup (#28) is not in this release: the batch command line still uploads to the Hugging Face dataset named in the code, and some files still carry internal references. This tree is not the one to publish.
- #6: the task's workspace is not passed explicitly on data-plane calls. Lake scopes the task credential to the task's workspace.
Where it has run
The v0.3.0 tree had not run on a development environment when it was released. Lakes run the operator's default image of this agent; the changes since v0.2.0 are proven by the test suite only. Moving a Lake to :v0.3.0 is a separate operator change.
The image
Built from the tag v0.3.0 (28eded2) with BuildKit (linux/amd64, the repository's Dockerfile), then tagged :v0.3.0 on that digest without rebuilding:
docker.io/oramasearch/agent-eval-generator:28eded2@sha256:74c5b48f80a6eec2274d3f092e1bd5ac64523dc79b05af111ab1992095c3f440, also tagged :v0.3.0 (same digest).
It is pushed only to the project's non-public Docker Hub repository, which answers 404 anonymously. Pinning it on a Lake is a separate change.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) |
66 passed, 0 failed, 0 skipped; 66 tests. Includes tests/test_release.py (the declared version has these notes, indexed) |
SPDX headers (python3 tools/spdx-headers.py --check) |
Every source file carries the header |
SBOM (python3 tools/sbom.py --version v0.3.0, then --check) |
Regenerated on the release commit; --check passes |
| Batch generation, and a live turn through Lake's model gateway | Not run (needs a live model and a Lake) |
| Linux image build (BuildKit, linux/amd64, from the tag) | Built from v0.3.0: oramasearch/agent-eval-generator:28eded2, also :v0.3.0, sha256:74c5b48f80a6eec2274d3f092e1bd5ac64523dc79b05af111ab1992095c3f440 |
Release v0.2.0
agent-eval-generator v0.2.0
Released 2026-10-25. A minor release. Two security fixes since v0.1.0, from the
2026-10-21 security review: the service token is compared in constant time and turns are refused
when it is unset, and a turn's Lake credential goes only to the service's own LAKE_URL. Private
repository and image, docker.io/oramasearch/agent-eval-generator:v0.2.0. Tracking issue:
#15.
This release is the tree of dev at d174d10 with these notes, the version and two README
corrections.
What it includes
Every pull request merged into dev since v0.1.0:
| PR | What it changes |
|---|---|
| #11 | Docs: records the v0.1.0 image digest in the v0.1.0 notes |
| #12 | POST /v1/turn compares the bearer with hmac.compare_digest and fails closed: with AGENT_SERVICE_TOKEN unset or empty every turn is refused with 503; a wrong or missing bearer is still 401. serve() warns at start when the token is missing (#8) |
| #14 | A turn's lake_token is sent only to the service's own LAKE_URL (an absolute http(s) URL with no user info): the registry reads (<LAKE_URL>/v1/workspace_query) and the model gateway (<LAKE_URL>/openai/v1) both use it, and a turn's lake_url is ignored. With LAKE_URL unset or malformed, a turn is refused before any request (#13) |
The release branch also corrects two README lines: the command-line example uses --cases (it said
--profiles), and the layout no longer lists a grader under evals/ or a reports/ folder, which do
not exist (v0.1.0 known issue 5).
Before it rolls: the settings it needs
AGENT_SERVICE_TOKENmust be set, orPOST /v1/turnrefuses every turn with503.LAKE_URLmust be set to Lake's address, or every turn is refused. A turn'slake_urlis no
longer read.
The Orama operator already renders both for every library agent (with AGENT_MODEL), so a Lake run
by the operator needs no new setting. A service run by hand needs both.
What it does not include
Unchanged from v0.1.0: no grading by Zetta's deterministic
graders (zetta#987), no capture-state evidence, no
model tools, no Hugging Face upload from a turn, and committed profiles that mostly predate the
registry.
Turn traces are still written to the pod's disk (TRACE_DIR, /tmp/traces under the operator).
The other agents' "no turn trace on disk" fix from the same review was not made in this repository;
the traces are lost on restart and nothing stores them in the workspace Zetta.
Known issues
Security-relevant first. All open at this tag.
- Workspace identity is not passed through
(#6, part of
lake#311). Registry reads send onlycustomer_id;
the task'sworkspace_idis ignored. The same issue asks for idle backoff on the bus. - Turn traces on the pod's disk (above): a trace holds the model's output for a person's request
and stays in the pod's/tmpuntil it restarts. - The lease is kept by activity only: no heartbeat of its own; a single model call longer than
lease_seconds(600 by default) could let the turn be reclaimed. Not observed. - Library agent page and chat in Lake are still open
(lake#263).
v0.1.0's known issue 2 (the bearer compared with !=, open when unset) is fixed by #12, and issue 5
(the README's example) by this release.
Where it has run
The v0.2.0 tree has not run on the dev environment. Every Lake runs the operator's default,
oramasearch/agent-eval-generator:v0.1.0 (orama-operator internal/controller/lake_defaults.go on
dev); no LakeSet member pins a dev build of this agent. After the tag, the release image goes to
testing first, behind its gate, then staging and prod.
The image
Built from the tag v0.2.0 (8b40218) on the cluster BuildKit with build-agent.sh (linux/amd64, the repository's Dockerfile), then tagged :v0.2.0 on that digest without rebuilding:
docker.io/oramasearch/agent-eval-generator:8b40218@sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284, also tagged :v0.2.0 (same digest). The Docker Hub repository answers 404 anonymously (private).
It is not rolled anywhere yet: pinning it on an environment is a separate change to the operator's set files, through dev, testing, staging and prod in order.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) |
36 tests: 36 passed, 0 failed, 0 skipped. tests/test_service.py 8, tests/test_service_auth.py 13 (#12), tests/test_turn_destinations.py 8 (#14), tests/test_lake_bus.py 4, tests/test_release.py 3 (the installed package is 0.2.0, these notes exist and are indexed, /healthz reports it) |
| A live turn (registry reads, model gateway, answer on the bus) | Not run for this release (needs a live model and a Lake) |
| Linux image build (cluster BuildKit, from the tag) | Built from v0.2.0 with build-agent.sh: oramasearch/agent-eval-generator:8b40218, also :v0.2.0, sha256:7f72fe02cb28de35b12c45f004609779d093208c1661a68a82756bc462fc3284; private (404 anonymously) |
Release v0.1.0
agent-eval-generator v0.1.0
Released 2026-10-17. The first release. One Python package, agent-eval-generator, and one
private image, docker.io/oramasearch/agent-eval-generator:v0.1.0. Proprietary to Orama; private
repository and image.
The Eval Generator Agent writes eval profiles: a situation (who someone is, what they are
responsible for, what systems they can reach) plus the tables and records that go with it, used to
score Orama's other agents (Auto-Suggest, the Project Agent) without a running Zetta node. Each
profile names the catalogue solution, capability level and methodology it needs, copies that
solution's correctness checks and decision policy from the engine's registry, and says whether the
engine can run it today (runnable); a profile that cannot run is kept but marked not scoreable.
It runs two ways: as a Lake library agent that answers one profile per turn, and as a batch
command line that writes profiles to evals/profiles/ and uploads them to the private Hugging
Face dataset OramaSearch/agent-eval-profiles.
v0.1.0 is the same tree as commit e2a683c (the image Lake runs today), plus these notes, the
version test, the version field in /healthz and the package description.
What it includes
The Lake library agent (agent-eval-generator-serve)
| Part | What it does |
|---|---|
Bus worker (lake_bus.py) |
Started by serve() when LAKE_URL, AGENT_BUS_TOKEN and AGENT_BUS_AGENT are all set (otherwise it logs "not configured" and only the HTTP server runs). One background thread long-polls receive_agent_tasks (GET /v1/receive_agent_tasks, wait_ms 10000, starting at from_sequence 1 on every start), then calls claim_agent_task (POST) for each task id. A conflict or not_found claim means another replica won and the task is skipped. Up to four claimed turns run at once in a thread pool. Each turn publishes publish_agent_event progress ("Working on it"), then result carrying the whole answer, or failed with the error. A receive failure is logged and retried after 2 seconds |
GET /healthz |
{ok: true, agent: "agent-eval-generator", version, prompt_version}; version is the installed package's version |
POST /v1/turn |
The same turn, answered in the HTTP response, for a direct call. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (the operator sets it). 200 with the answer when ok, 422 when not, 400 for a body that is not JSON, 401 for a wrong bearer, 404 for any other route |
Turn (service.run_turn) |
Reads the registries, asks the model for one profile steered by the person's request, applies the registry facts, synthesizes the records, validates them, and answers. A failure is an answer ({ok: false, error}), never an exception |
The turn body it reads: request (or message), lake_url (or the LAKE_URL variable),
lake_token, customer_id, and optionally methodologies (a list_methodologies response; when
absent the agent reads it itself). Lake's dispatch also carries workspace_id, conversation_id
and page, which this release does not read (known issue 1).
The answer: {ok: true, reply, result: {kind: "eval_profile", profile, tables, problems, prompt_version, scoreable}}. tables lists each table as {table, rows, columns, sample} with five
sample rows; problems is the validator's list (empty means the profile passes); scoreable is
the profile's runnable. The reply opens with the situation's first sentence and says when the
profile fails validation, is declined for scoring, or rests on the M-1-only fallback.
A turn refuses before the model runs when the methodology or solution registry read fails, when
the solution registry is not deployed (unknown_capability: no solutions, no local fallback list),
or when the turn has no lake_url and lake_token. It refuses after the model when no parseable
profile came back, or when the model chose a methodology, solution or level that the turn's
registry does not have.
Nothing is published by a turn: no Hugging Face upload, no write to the workspace. The trace is
appended to TRACE_DIR inside the pod.
What it calls on Lake
| Call | With | Why |
|---|---|---|
receive_agent_tasks, claim_agent_task, publish_agent_event (<LAKE_URL>/v1/<capability>) |
Its own bus credential, AGENT_BUS_TOKEN |
Take turns and answer them |
workspace_query with capability list_methodologies, list_solutions and get_solution {solution} (POST <lake_url>/v1/workspace_query, customer_id from the turn) |
The turn's lake_token |
The registries, fresh on every turn: the lists go into the model's context, and the selected solution's detail is copied into the profile |
Model gateway <lake_url>/openai/v1 (chat.completions, OpenAI SDK) |
The turn's lake_token as the API key |
One model call per turn; Lake counts the tokens in the turn's workspace and enforces the account's limit (lake#248). The service holds no provider key |
Lake grants the turn's credential workspace_query, workspace_openapi and get_agent_task.
The model is given no tools (tools=[]): it invents the systems and tables, so a person's
workspace is never shown to it.
Prompt version: PROMPT_VERSION = "v3" (src/agent_eval_generator/prompts/v3.md); v1 and v2
remain in prompts/ for older traces. Every trace and every answer records the version.
Model: EVAL_GEN_MODEL, else AGENT_MODEL, else gpt-4o.
Profile generation (shared by the agent and the command line)
methodologies.py: normalizeslist_methodologies; an engine that answersunknown_capability
gets the documented M-1-only fallback; attachesrunnableandissuefor the chosen methodology.
Methodology-aware eval profilessolutions.py: normalizeslist_solutions, readsget_solution, refuses asketch, an
undeclared level, or a methodology that is not in the solution's composition at that level, and
copiessolution_checks(capability level, composition, pipeline order, required data roles,
measurement design, guardrails) anddecision_policyinto the profile;runnableis the
methodology's and the solution's together. Solution-aware eval profilessynth.py: makes the records from the declared schema, honours each column'snull_fraction,
caps a table at 50,000 rows and recordsrecords_sampledwhen the declared size is larger.records.pypackage(): writessituation.md,schema.jsonandrecords/*.jsonl, clearing
stale record files first.validate.py: profile fields,runnable/issue, the solution contract, stray or missing record
files, undeclared or absent columns, null rates within 0.15 of the declared ones, placeholder
values, declared versus packaged row counts, asituation.mdof at least 120 characters, and at
least onecapturedconnection.trace.py: one JSON line per run in<dir>/eval-generator-<day>.jsonl: model, prompt version,
tool calls, output and grade.
The command line
Entry point ([project.scripts]) |
What it does |
|---|---|
agent-eval-generator (agent:main) |
--cases N (default 1), --model, --traces DIR (default traces). Reads the registries directly from a Zetta node (GET <ZETTA_BASE_URL>/v1/<capability>), calls OpenAI directly with your own key, writes each passing profile to evals/profiles/<profile_id>/, and uploads it with hf upload OramaSearch/agent-eval-profiles … --repo-type=dataset as soon as it is validated. A rejected profile is not uploaded |
agent-eval-generator-serve (service:serve) |
The library agent above |
Environment variables
| Variable | Read by | Meaning |
|---|---|---|
PORT |
service | Listening port (default and image: 8080) |
AGENT_SERVICE_TOKEN |
service | Bearer required on POST /v1/turn when set |
LAKE_URL |
service, bus | Lake's address; also the fallback when a turn has no lake_url |
AGENT_BUS_TOKEN |
bus | This agent's own bus credential (operator Secret lake-<name>-agent-bus) |
AGENT_BUS_AGENT |
bus | The agent name it takes tasks under (default agent-eval-generator) |
EVAL_GEN_MODEL, AGENT_MODEL |
service, CLI (EVAL_GEN_MODEL only) |
Model name; default gpt-4o |
TRACE_DIR |
service | Where turn traces are appended (image default /tmp/agent-eval-generator-traces; the operator sets /tmp/traces) |
ZETTA_BASE_URL, ZETTA_TOKEN |
CLI | The Zetta node the CLI reads registries from |
OPENAI_API_KEY |
CLI (through the OpenAI SDK) | The CLI's own model key. The service never uses it, even when the operator sets it |
How Lake reaches it
- The Orama operator runs it as a library agent of every Lake: one Deployment and one Service,
port8080, health on/healthz, non-root with a read-only root filesystem and a writable
/tmp. The operator setsLAKE_URL,PORT,TRACE_DIR,AGENT_MODEL,AGENT_SERVICE_TOKEN,
AGENT_BUS_TOKENandAGENT_BUS_AGENT=agent-eval-generator. - Lake learns its address from
LAKE_AGENT_ENDPOINTS(agent-eval-generator=<url>, set by the
operator); Lake's agent library lists it as Eval Generator Agent, run as aservice, reached
from Ask Orama when a person asks for an eval. - Turns are dispatched on the agent bus:
agent_turnpublishes the task id on
agent-tasks.agent-eval-generatoron Lake's own Zetta (task ids only, never content); this agent
receives and claims it through Lake as above, and the claim's answer carries the body with the
task credential. Its events go, through Lake, to the task workspace'sagent-events. Lake's
agent bus,
[agent launches](https:/...