Repository navigation
Release v0.1.0
agent-eval-generator v0.1.0
Released 2026-10-17. The first release. One Python package, agent-eval-generator, and one
private image, docker.io/oramasearch/agent-eval-generator:v0.1.0. Proprietary to Orama; private
repository and image.
The Eval Generator Agent writes eval profiles: a situation (who someone is, what they are
responsible for, what systems they can reach) plus the tables and records that go with it, used to
score Orama's other agents (Auto-Suggest, the Project Agent) without a running Zetta node. Each
profile names the catalogue solution, capability level and methodology it needs, copies that
solution's correctness checks and decision policy from the engine's registry, and says whether the
engine can run it today (runnable); a profile that cannot run is kept but marked not scoreable.
It runs two ways: as a Lake library agent that answers one profile per turn, and as a batch
command line that writes profiles to evals/profiles/ and uploads them to the private Hugging
Face dataset OramaSearch/agent-eval-profiles.
v0.1.0 is the same tree as commit e2a683c (the image Lake runs today), plus these notes, the
version test, the version field in /healthz and the package description.
What it includes
The Lake library agent (agent-eval-generator-serve)
| Part | What it does |
|---|---|
Bus worker (lake_bus.py) |
Started by serve() when LAKE_URL, AGENT_BUS_TOKEN and AGENT_BUS_AGENT are all set (otherwise it logs "not configured" and only the HTTP server runs). One background thread long-polls receive_agent_tasks (GET /v1/receive_agent_tasks, wait_ms 10000, starting at from_sequence 1 on every start), then calls claim_agent_task (POST) for each task id. A conflict or not_found claim means another replica won and the task is skipped. Up to four claimed turns run at once in a thread pool. Each turn publishes publish_agent_event progress ("Working on it"), then result carrying the whole answer, or failed with the error. A receive failure is logged and retried after 2 seconds |
GET /healthz |
{ok: true, agent: "agent-eval-generator", version, prompt_version}; version is the installed package's version |
POST /v1/turn |
The same turn, answered in the HTTP response, for a direct call. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (the operator sets it). 200 with the answer when ok, 422 when not, 400 for a body that is not JSON, 401 for a wrong bearer, 404 for any other route |
Turn (service.run_turn) |
Reads the registries, asks the model for one profile steered by the person's request, applies the registry facts, synthesizes the records, validates them, and answers. A failure is an answer ({ok: false, error}), never an exception |
The turn body it reads: request (or message), lake_url (or the LAKE_URL variable),
lake_token, customer_id, and optionally methodologies (a list_methodologies response; when
absent the agent reads it itself). Lake's dispatch also carries workspace_id, conversation_id
and page, which this release does not read (known issue 1).
The answer: {ok: true, reply, result: {kind: "eval_profile", profile, tables, problems, prompt_version, scoreable}}. tables lists each table as {table, rows, columns, sample} with five
sample rows; problems is the validator's list (empty means the profile passes); scoreable is
the profile's runnable. The reply opens with the situation's first sentence and says when the
profile fails validation, is declined for scoring, or rests on the M-1-only fallback.
A turn refuses before the model runs when the methodology or solution registry read fails, when
the solution registry is not deployed (unknown_capability: no solutions, no local fallback list),
or when the turn has no lake_url and lake_token. It refuses after the model when no parseable
profile came back, or when the model chose a methodology, solution or level that the turn's
registry does not have.
Nothing is published by a turn: no Hugging Face upload, no write to the workspace. The trace is
appended to TRACE_DIR inside the pod.
What it calls on Lake
| Call | With | Why |
|---|---|---|
receive_agent_tasks, claim_agent_task, publish_agent_event (<LAKE_URL>/v1/<capability>) |
Its own bus credential, AGENT_BUS_TOKEN |
Take turns and answer them |
workspace_query with capability list_methodologies, list_solutions and get_solution {solution} (POST <lake_url>/v1/workspace_query, customer_id from the turn) |
The turn's lake_token |
The registries, fresh on every turn: the lists go into the model's context, and the selected solution's detail is copied into the profile |
Model gateway <lake_url>/openai/v1 (chat.completions, OpenAI SDK) |
The turn's lake_token as the API key |
One model call per turn; Lake counts the tokens in the turn's workspace and enforces the account's limit (lake#248). The service holds no provider key |
Lake grants the turn's credential workspace_query, workspace_openapi and get_agent_task.
The model is given no tools (tools=[]): it invents the systems and tables, so a person's
workspace is never shown to it.
Prompt version: PROMPT_VERSION = "v3" (src/agent_eval_generator/prompts/v3.md); v1 and v2
remain in prompts/ for older traces. Every trace and every answer records the version.
Model: EVAL_GEN_MODEL, else AGENT_MODEL, else gpt-4o.
Profile generation (shared by the agent and the command line)
methodologies.py: normalizeslist_methodologies; an engine that answersunknown_capability
gets the documented M-1-only fallback; attachesrunnableandissuefor the chosen methodology.
Methodology-aware eval profilessolutions.py: normalizeslist_solutions, readsget_solution, refuses asketch, an
undeclared level, or a methodology that is not in the solution's composition at that level, and
copiessolution_checks(capability level, composition, pipeline order, required data roles,
measurement design, guardrails) anddecision_policyinto the profile;runnableis the
methodology's and the solution's together. Solution-aware eval profilessynth.py: makes the records from the declared schema, honours each column'snull_fraction,
caps a table at 50,000 rows and recordsrecords_sampledwhen the declared size is larger.records.pypackage(): writessituation.md,schema.jsonandrecords/*.jsonl, clearing
stale record files first.validate.py: profile fields,runnable/issue, the solution contract, stray or missing record
files, undeclared or absent columns, null rates within 0.15 of the declared ones, placeholder
values, declared versus packaged row counts, asituation.mdof at least 120 characters, and at
least onecapturedconnection.trace.py: one JSON line per run in<dir>/eval-generator-<day>.jsonl: model, prompt version,
tool calls, output and grade.
The command line
Entry point ([project.scripts]) |
What it does |
|---|---|
agent-eval-generator (agent:main) |
--cases N (default 1), --model, --traces DIR (default traces). Reads the registries directly from a Zetta node (GET <ZETTA_BASE_URL>/v1/<capability>), calls OpenAI directly with your own key, writes each passing profile to evals/profiles/<profile_id>/, and uploads it with hf upload OramaSearch/agent-eval-profiles … --repo-type=dataset as soon as it is validated. A rejected profile is not uploaded |
agent-eval-generator-serve (service:serve) |
The library agent above |
Environment variables
| Variable | Read by | Meaning |
|---|---|---|
PORT |
service | Listening port (default and image: 8080) |
AGENT_SERVICE_TOKEN |
service | Bearer required on POST /v1/turn when set |
LAKE_URL |
service, bus | Lake's address; also the fallback when a turn has no lake_url |
AGENT_BUS_TOKEN |
bus | This agent's own bus credential (operator Secret lake-<name>-agent-bus) |
AGENT_BUS_AGENT |
bus | The agent name it takes tasks under (default agent-eval-generator) |
EVAL_GEN_MODEL, AGENT_MODEL |
service, CLI (EVAL_GEN_MODEL only) |
Model name; default gpt-4o |
TRACE_DIR |
service | Where turn traces are appended (image default /tmp/agent-eval-generator-traces; the operator sets /tmp/traces) |
ZETTA_BASE_URL, ZETTA_TOKEN |
CLI | The Zetta node the CLI reads registries from |
OPENAI_API_KEY |
CLI (through the OpenAI SDK) | The CLI's own model key. The service never uses it, even when the operator sets it |
How Lake reaches it
- The Orama operator runs it as a library agent of every Lake: one Deployment and one Service,
port8080, health on/healthz, non-root with a read-only root filesystem and a writable
/tmp. The operator setsLAKE_URL,PORT,TRACE_DIR,AGENT_MODEL,AGENT_SERVICE_TOKEN,
AGENT_BUS_TOKENandAGENT_BUS_AGENT=agent-eval-generator. - Lake learns its address from
LAKE_AGENT_ENDPOINTS(agent-eval-generator=<url>, set by the
operator); Lake's agent library lists it as Eval Generator Agent, run as aservice, reached
from Ask Orama when a person asks for an eval. - Turns are dispatched on the agent bus:
agent_turnpublishes the task id on
agent-tasks.agent-eval-generatoron Lake's own Zetta (task ids only, never content); this agent
receives and claims it through Lake as above, and the claim's answer carries the body with the
task credential. Its events go, through Lake, to the task workspace'sagent-events. Lake's
agent bus,
agent launches and
train agents.
The image
oramasearch/agent-eval-generatoron Docker Hub, private.- Two stages on
python:3.12-slim:uv0.11.7 runsuv sync --frozen --no-dev --no-editable; the
runtime copies only the virtual environment, runs as user10001(agent), exposes8080, and
startsagent-eval-generator-serve. - No named build contexts. The image carries
src/only: notevals/,traces/or thehf
command. - This repository's
evals/profiles/is itself theevalsnamed build context of the
oramasearch/agent-autosuggest-corpusimage, whose contamination guard checks generated examples
against these profiles. - Digest:
docker.io/oramasearch/agent-eval-generator:111ea5b@sha256:3b0ccf6132293ebd98c033191c461030b29a4b96497de2ea2a0dd557c348480c, also tagged:v0.1.0(same digest).
Where it runs
Every Lake runs it.
- The operator's default library image (
orama-operatorinternal/controller/lake_defaults.goon
dev) isoramasearch/agent-eval-generator:e2a683c. - Lake
prodon the managing clusterorama-production-cluster-01(namespacelake, the
development environment) pinsoramasearch/agent-eval-generator:e2a683cin itsspec.agents. - The LakeSets
lake-nonprod(membersdev,stagingandtestingonorama-nonprod-cluster-01)
andlake-prod(memberssharedonorama-production-cluster-02andshared-uson
orama-production-cluster-03) take the operator default.
e2a683c is the same tree as this release apart from these notes, the test and the version field
in /healthz. Moving the operator default to :v0.1.0 is a separate operator change.
What it does not include
- No grading. No generated profile has been scored by Zetta's deterministic graders; traces
recordgraded: false(zetta#987). - No capture-state evidence. No capability reports whether a source is captured, so the
discovery/capture distinction is stated by the model, not checked. - No model tools.
tools.pystill holds a hand-written Zetta tool table,derive_tool_specs
(from a node's OpenAPI) anddispatch, but both entry points pass the model no tools. The
seed-profile catalogue (catalog.py),records.coffee_subscriptionandrecords.land(loading a
profile into Zetta) are in the package and are not called by either entry point. - No upload from a turn. Publishing to Hugging Face is the command line's only.
- Turn traces are not kept. They are written to the pod's
/tmpand lost when it restarts;
nothing stores them in the workspace Zetta. - The committed profiles mostly predate the registry. Of the 264 profiles in
evals/profiles/,
263 carry nomethodology,solutionorrunnable; the one that does,
churn-chief-operating-officer-enterprise, is the tests' fixture. None carriesrecords/(the
records are on Hugging Face), so none passes today'svalidate().
Known issues
Security-relevant first.
- Workspace identity is not passed through
(agent-eval-generator#6, part of
lake#311). Registry reads send only
customer_id; the task'sworkspace_idis ignored, so the workspace is the one Lake derives
from the task credential rather than one the agent names. The same issue asks for idle backoff:
the bus worker polls again as soon as a long poll returns empty. POST /v1/turncompares the bearer with!=, not in constant time, and is open when
AGENT_SERVICE_TOKENis unset. A turn without a Lake credential is still refused before the
model runs. agent-eval-generator#8.- The lease is kept by activity only. The agent sends no heartbeat of its own. Lake renews a claim's lease on every accepted event and every capability call made with the task credential (lake
docs/api/agent-bus.md), so a turn can be reclaimed only if a single step between two such calls, such as one model call, outlastslease_seconds(600 seconds by default). Not observed. - Library agent page and chat in Lake are still open
(lake#263). - The README's command-line example uses
--profiles 5; the flag is--cases. Its layout lists a
grader underevals/and areports/folder, neither of which exists.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12) |
15 passed, 0 failed, 0 skipped: tests/test_service.py (8), tests/test_lake_bus.py (4), tests/test_release.py (3: the installed package's version, these notes for that version, /healthz reporting it) |
Committed profiles against validate() (offline, no model) |
264 checked, 0 pass: 263 lack methodology (made before the registry fields), the updated fixture has no records/ (the records are on Hugging Face). Not a release gate; see What it does not include |
| A live turn (registry reads, model gateway, answer on the bus) | Not run for this release (needs a live model) |
| Linux image build (cluster BuildKit, from the tag) | Built from the tag v0.1.0 with build-agent.sh: linux/amd64, pushed as oramasearch/agent-eval-generator:111ea5b |
| Image digest | docker.io/oramasearch/agent-eval-generator:111ea5b@sha256:3b0ccf6132293ebd98c033191c461030b29a4b96497de2ea2a0dd557c348480c, also tagged :v0.1.0 (same digest); the Docker Hub repository answers 404 anonymously (private) |