Skip to content

Release v0.1.0

Choose a tag to compare

@homer6 homer6 released this 17 Oct 23:01
· 28 commits to main since this release
111ea5b

agent-eval-generator v0.1.0

Released 2026-10-17. The first release. One Python package, agent-eval-generator, and one
private image, docker.io/oramasearch/agent-eval-generator:v0.1.0. Proprietary to Orama; private
repository and image.

The Eval Generator Agent writes eval profiles: a situation (who someone is, what they are
responsible for, what systems they can reach) plus the tables and records that go with it, used to
score Orama's other agents (Auto-Suggest, the Project Agent) without a running Zetta node. Each
profile names the catalogue solution, capability level and methodology it needs, copies that
solution's correctness checks and decision policy from the engine's registry, and says whether the
engine can run it today (runnable); a profile that cannot run is kept but marked not scoreable.
It runs two ways: as a Lake library agent that answers one profile per turn, and as a batch
command line
that writes profiles to evals/profiles/ and uploads them to the private Hugging
Face dataset OramaSearch/agent-eval-profiles.

v0.1.0 is the same tree as commit e2a683c (the image Lake runs today), plus these notes, the
version test, the version field in /healthz and the package description.

What it includes

The Lake library agent (agent-eval-generator-serve)

Part What it does
Bus worker (lake_bus.py) Started by serve() when LAKE_URL, AGENT_BUS_TOKEN and AGENT_BUS_AGENT are all set (otherwise it logs "not configured" and only the HTTP server runs). One background thread long-polls receive_agent_tasks (GET /v1/receive_agent_tasks, wait_ms 10000, starting at from_sequence 1 on every start), then calls claim_agent_task (POST) for each task id. A conflict or not_found claim means another replica won and the task is skipped. Up to four claimed turns run at once in a thread pool. Each turn publishes publish_agent_event progress ("Working on it"), then result carrying the whole answer, or failed with the error. A receive failure is logged and retried after 2 seconds
GET /healthz {ok: true, agent: "agent-eval-generator", version, prompt_version}; version is the installed package's version
POST /v1/turn The same turn, answered in the HTTP response, for a direct call. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (the operator sets it). 200 with the answer when ok, 422 when not, 400 for a body that is not JSON, 401 for a wrong bearer, 404 for any other route
Turn (service.run_turn) Reads the registries, asks the model for one profile steered by the person's request, applies the registry facts, synthesizes the records, validates them, and answers. A failure is an answer ({ok: false, error}), never an exception

The turn body it reads: request (or message), lake_url (or the LAKE_URL variable),
lake_token, customer_id, and optionally methodologies (a list_methodologies response; when
absent the agent reads it itself). Lake's dispatch also carries workspace_id, conversation_id
and page, which this release does not read (known issue 1).

The answer: {ok: true, reply, result: {kind: "eval_profile", profile, tables, problems, prompt_version, scoreable}}. tables lists each table as {table, rows, columns, sample} with five
sample rows; problems is the validator's list (empty means the profile passes); scoreable is
the profile's runnable. The reply opens with the situation's first sentence and says when the
profile fails validation, is declined for scoring, or rests on the M-1-only fallback.

A turn refuses before the model runs when the methodology or solution registry read fails, when
the solution registry is not deployed (unknown_capability: no solutions, no local fallback list),
or when the turn has no lake_url and lake_token. It refuses after the model when no parseable
profile came back, or when the model chose a methodology, solution or level that the turn's
registry does not have.

Nothing is published by a turn: no Hugging Face upload, no write to the workspace. The trace is
appended to TRACE_DIR inside the pod.

What it calls on Lake

Call With Why
receive_agent_tasks, claim_agent_task, publish_agent_event (<LAKE_URL>/v1/<capability>) Its own bus credential, AGENT_BUS_TOKEN Take turns and answer them
workspace_query with capability list_methodologies, list_solutions and get_solution {solution} (POST <lake_url>/v1/workspace_query, customer_id from the turn) The turn's lake_token The registries, fresh on every turn: the lists go into the model's context, and the selected solution's detail is copied into the profile
Model gateway <lake_url>/openai/v1 (chat.completions, OpenAI SDK) The turn's lake_token as the API key One model call per turn; Lake counts the tokens in the turn's workspace and enforces the account's limit (lake#248). The service holds no provider key

Lake grants the turn's credential workspace_query, workspace_openapi and get_agent_task.
The model is given no tools (tools=[]): it invents the systems and tables, so a person's
workspace is never shown to it.

Prompt version: PROMPT_VERSION = "v3" (src/agent_eval_generator/prompts/v3.md); v1 and v2
remain in prompts/ for older traces. Every trace and every answer records the version.

Model: EVAL_GEN_MODEL, else AGENT_MODEL, else gpt-4o.

Profile generation (shared by the agent and the command line)

  • methodologies.py: normalizes list_methodologies; an engine that answers unknown_capability
    gets the documented M-1-only fallback; attaches runnable and issue for the chosen methodology.
    Methodology-aware eval profiles
  • solutions.py: normalizes list_solutions, reads get_solution, refuses a sketch, an
    undeclared level, or a methodology that is not in the solution's composition at that level, and
    copies solution_checks (capability level, composition, pipeline order, required data roles,
    measurement design, guardrails) and decision_policy into the profile; runnable is the
    methodology's and the solution's together. Solution-aware eval profiles
  • synth.py: makes the records from the declared schema, honours each column's null_fraction,
    caps a table at 50,000 rows and records records_sampled when the declared size is larger.
  • records.py package(): writes situation.md, schema.json and records/*.jsonl, clearing
    stale record files first.
  • validate.py: profile fields, runnable/issue, the solution contract, stray or missing record
    files, undeclared or absent columns, null rates within 0.15 of the declared ones, placeholder
    values, declared versus packaged row counts, a situation.md of at least 120 characters, and at
    least one captured connection.
  • trace.py: one JSON line per run in <dir>/eval-generator-<day>.jsonl: model, prompt version,
    tool calls, output and grade.

The command line

Entry point ([project.scripts]) What it does
agent-eval-generator (agent:main) --cases N (default 1), --model, --traces DIR (default traces). Reads the registries directly from a Zetta node (GET <ZETTA_BASE_URL>/v1/<capability>), calls OpenAI directly with your own key, writes each passing profile to evals/profiles/<profile_id>/, and uploads it with hf upload OramaSearch/agent-eval-profiles … --repo-type=dataset as soon as it is validated. A rejected profile is not uploaded
agent-eval-generator-serve (service:serve) The library agent above

Environment variables

Variable Read by Meaning
PORT service Listening port (default and image: 8080)
AGENT_SERVICE_TOKEN service Bearer required on POST /v1/turn when set
LAKE_URL service, bus Lake's address; also the fallback when a turn has no lake_url
AGENT_BUS_TOKEN bus This agent's own bus credential (operator Secret lake-<name>-agent-bus)
AGENT_BUS_AGENT bus The agent name it takes tasks under (default agent-eval-generator)
EVAL_GEN_MODEL, AGENT_MODEL service, CLI (EVAL_GEN_MODEL only) Model name; default gpt-4o
TRACE_DIR service Where turn traces are appended (image default /tmp/agent-eval-generator-traces; the operator sets /tmp/traces)
ZETTA_BASE_URL, ZETTA_TOKEN CLI The Zetta node the CLI reads registries from
OPENAI_API_KEY CLI (through the OpenAI SDK) The CLI's own model key. The service never uses it, even when the operator sets it

How Lake reaches it

  • The Orama operator runs it as a library agent of every Lake: one Deployment and one Service,
    port 8080, health on /healthz, non-root with a read-only root filesystem and a writable
    /tmp. The operator sets LAKE_URL, PORT, TRACE_DIR, AGENT_MODEL, AGENT_SERVICE_TOKEN,
    AGENT_BUS_TOKEN and AGENT_BUS_AGENT=agent-eval-generator.
  • Lake learns its address from LAKE_AGENT_ENDPOINTS (agent-eval-generator=<url>, set by the
    operator); Lake's agent library lists it as Eval Generator Agent, run as a service, reached
    from Ask Orama when a person asks for an eval.
  • Turns are dispatched on the agent bus: agent_turn publishes the task id on
    agent-tasks.agent-eval-generator on Lake's own Zetta (task ids only, never content); this agent
    receives and claims it through Lake as above, and the claim's answer carries the body with the
    task credential. Its events go, through Lake, to the task workspace's agent-events. Lake's
    agent bus,
    agent launches and
    train agents.

The image

  • oramasearch/agent-eval-generator on Docker Hub, private.
  • Two stages on python:3.12-slim: uv 0.11.7 runs uv sync --frozen --no-dev --no-editable; the
    runtime copies only the virtual environment, runs as user 10001 (agent), exposes 8080, and
    starts agent-eval-generator-serve.
  • No named build contexts. The image carries src/ only: not evals/, traces/ or the hf
    command.
  • This repository's evals/profiles/ is itself the evals named build context of the
    oramasearch/agent-autosuggest-corpus image, whose contamination guard checks generated examples
    against these profiles.
  • Digest: docker.io/oramasearch/agent-eval-generator:111ea5b@sha256:3b0ccf6132293ebd98c033191c461030b29a4b96497de2ea2a0dd557c348480c, also tagged :v0.1.0 (same digest).

Where it runs

Every Lake runs it.

  • The operator's default library image (orama-operator internal/controller/lake_defaults.go on
    dev) is oramasearch/agent-eval-generator:e2a683c.
  • Lake prod on the managing cluster orama-production-cluster-01 (namespace lake, the
    development environment) pins oramasearch/agent-eval-generator:e2a683c in its spec.agents.
  • The LakeSets lake-nonprod (members dev, staging and testing on orama-nonprod-cluster-01)
    and lake-prod (members shared on orama-production-cluster-02 and shared-us on
    orama-production-cluster-03) take the operator default.

e2a683c is the same tree as this release apart from these notes, the test and the version field
in /healthz. Moving the operator default to :v0.1.0 is a separate operator change.

What it does not include

  • No grading. No generated profile has been scored by Zetta's deterministic graders; traces
    record graded: false (zetta#987).
  • No capture-state evidence. No capability reports whether a source is captured, so the
    discovery/capture distinction is stated by the model, not checked.
  • No model tools. tools.py still holds a hand-written Zetta tool table, derive_tool_specs
    (from a node's OpenAPI) and dispatch, but both entry points pass the model no tools. The
    seed-profile catalogue (catalog.py), records.coffee_subscription and records.land (loading a
    profile into Zetta) are in the package and are not called by either entry point.
  • No upload from a turn. Publishing to Hugging Face is the command line's only.
  • Turn traces are not kept. They are written to the pod's /tmp and lost when it restarts;
    nothing stores them in the workspace Zetta.
  • The committed profiles mostly predate the registry. Of the 264 profiles in evals/profiles/,
    263 carry no methodology, solution or runnable; the one that does,
    churn-chief-operating-officer-enterprise, is the tests' fixture. None carries records/ (the
    records are on Hugging Face), so none passes today's validate().

Known issues

Security-relevant first.

  1. Workspace identity is not passed through
    (agent-eval-generator#6, part of
    lake#311). Registry reads send only
    customer_id; the task's workspace_id is ignored, so the workspace is the one Lake derives
    from the task credential rather than one the agent names. The same issue asks for idle backoff:
    the bus worker polls again as soon as a long poll returns empty.
  2. POST /v1/turn compares the bearer with !=, not in constant time, and is open when
    AGENT_SERVICE_TOKEN is unset. A turn without a Lake credential is still refused before the
    model runs. agent-eval-generator#8.
  3. The lease is kept by activity only. The agent sends no heartbeat of its own. Lake renews a claim's lease on every accepted event and every capability call made with the task credential (lake docs/api/agent-bus.md), so a turn can be reclaimed only if a single step between two such calls, such as one model call, outlasts lease_seconds (600 seconds by default). Not observed.
  4. Library agent page and chat in Lake are still open
    (lake#263).
  5. The README's command-line example uses --profiles 5; the flag is --cases. Its layout lists a
    grader under evals/ and a reports/ folder, neither of which exists.

Verification

Proof Result
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12) 15 passed, 0 failed, 0 skipped: tests/test_service.py (8), tests/test_lake_bus.py (4), tests/test_release.py (3: the installed package's version, these notes for that version, /healthz reporting it)
Committed profiles against validate() (offline, no model) 264 checked, 0 pass: 263 lack methodology (made before the registry fields), the updated fixture has no records/ (the records are on Hugging Face). Not a release gate; see What it does not include
A live turn (registry reads, model gateway, answer on the bus) Not run for this release (needs a live model)
Linux image build (cluster BuildKit, from the tag) Built from the tag v0.1.0 with build-agent.sh: linux/amd64, pushed as oramasearch/agent-eval-generator:111ea5b
Image digest docker.io/oramasearch/agent-eval-generator:111ea5b@sha256:3b0ccf6132293ebd98c033191c461030b29a4b96497de2ea2a0dd557c348480c, also tagged :v0.1.0 (same digest); the Docker Hub repository answers 404 anonymously (private)