Skip to content

Release v0.1.0

Choose a tag to compare

@homer6 homer6 released this 17 Oct 22:59
· 43 commits to main since this release
41dcf92

agent-experiments v0.1.0

Released 2026-10-17. The first release. One Python service, the Experiments Agent, and one
private image, docker.io/oramasearch/agent-experiments:v0.1.0. Proprietary to Orama; private
repository and image.

The Experiments Agent designs AutoML experiments ("studies" in the engine's and the agent's own
words) for a dataset, starts one only after a person confirms the exact proposal, watches its trials
and their training metrics, stops trials that diverge, explains the result, and promotes the best
trial only after a second confirmation. It is a Lake library agent: Lake dispatches its turns
(Train → Experiments, and delegations from Ask Orama and the Project Agent), and every read and
action goes through Lake capabilities with the asking person's delegated credential. It has no
direct connection to Zetta, a database, a scheduler or an event broker. Epic:
lake#305.

The contracts: service and bus, designing and running
studies
, evaluation scenarios.

What it includes

Turn kinds

kind What it does Lake capabilities it may call
experiment (no context.confirmed_proposal) The model designs a study from the request and Lake facts, through one tool, lake_read. Every reference is resolved and the complete study is preflighted with validate_study; a defect is fed back to the model (up to 2 repairs, up to 10 tool rounds each). Answers result.kind proposal with confirm: true, or question. Performs no action reads only
experiment with context.confirmed_proposal Re-validates the reviewed proposal (fields, references, validate_study), then calls create_study with that exact study object. No model call. Answers result.kind: running reads, then create_study
experiment_status Reads the study (context.study, or context.study_id) and its trials, tails study-events.<study> and run-metrics.<run> of every running trial with bounded receive_events (at most 100 events, 20 s wait, per-topic cursors returned in result.cursors), and calls stop_trial for a trial whose batch has a non-finite value, a gradient norm of 1,000 or more, a tenfold gradient-norm jump to at least 100, or three worsening validation-loss points with a gap over training loss of at least 0.25. Answers result.kind: status reads, stop_trial
experiment_explain (no context.confirmed_promotion) Reads get_study and get_study_analysis; reports the best trial, parameter importance, the next search direction, trial-variance and validation-overuse caveats; returns a promotion proposal with confirm: true. Answers result.kind: explanation. Performs no action reads only
experiment_explain with context.confirmed_promotion Checks the confirmed study matches the turn's study and the confirmed run is still the best trial, then calls promote_trial. Answers result.kind: promoted get_study_analysis, promote_trial

Plain text such as "yes" never authorizes create_study or promote_trial; only the reviewed object
in context does.

Lake capabilities (exact allow-lists in src/agent_experiments/agent.py)

  • Reads (READ_CAPABILITIES, sent as GET <lake_url>/v1/<capability>): analyze_dataset,
    get_project, get_run_metrics, get_solution, get_study, get_study_analysis,
    list_methodologies, list_projects, list_studies, list_trials, plan_features,
    receive_events, validate_study. The model can call any of them through lake_read;
    get_run_metrics and list_studies are not called by the agent's own code.
  • Actions (ACTION_CAPABILITIES, sent as POST): create_study, promote_trial,
    stop_trial, stop_study. stop_study is allowed but never called by this release.
  • Any other capability is refused inside the agent (not_allowed) before a request is made.
  • These are the engine's names, called through Lake's workspace adapter; Lake's own names for them
    are the experiment ones (lake docs/api/experiments.md, lake#561).

The prompt

  • PROMPT_VERSION = "v1" (in service.py and agent.py); the system prompt is
    src/agent_experiments/prompts/v1.md, used only for the design turn.
  • Every turn's trace records prompt_version, and GET /healthz reports it.

HTTP API (port PORT, default 8080)

Route What
GET /healthz {ok, agent: "agent-experiments", version, model, prompt_version}. version is the installed package version (new in this release)
POST /v1/turn One turn. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (otherwise 401). Body: kind, message, lake_token (the person's delegated credential), lake_url (optional when LAKE_URL is set), optional customer_id, conversation_id, context. Answers 200 with {ok: true, conversation_id, trace_id, reply, result}, 422 with {ok: false, error}, 400 when the body is not JSON

Any other path is 404. Conversation history is kept per conversation_id in the process's memory.

The agent bus: how Lake reaches it

  • The operator runs it as a library agent of every Lake: one Deployment and one Service, port 8080,
    probed on /healthz, and gives Lake its address in LAKE_AGENT_ENDPOINTS
    (agent-experiments=<url>). The operator sets LAKE_URL, AGENT_SERVICE_TOKEN,
    AGENT_BUS_TOKEN and AGENT_BUS_AGENT=agent-experiments on it.
  • Lake dispatches each turn as a task on the topic agent-tasks.agent-experiments of Lake's own
    Zetta. The topic carries task ids only, never content (lake docs/api/agent-bus.md).
  • When LAKE_URL, AGENT_BUS_TOKEN and AGENT_BUS_AGENT are set, serve starts one bus thread
    (lake_bus.py). It long-polls receive_agent_tasks {agent, from_sequence, wait_ms: 10000}
    through Lake, starting at sequence 1 on every start and continuing from next_sequence. For each
    task it calls claim_agent_task {task_id, agent}; a conflict or not_found answer means
    another replica or attempt has it and the task is skipped. A claimed task's body (with its task
    credential as lake_token) runs on a pool of 4 worker threads through the same run_turn as
    POST /v1/turn. The worker publishes, through publish_agent_event, one progress event
    ("Handling the experiment turn"), then a result event with the whole answer or a failed event
    with the error. After a receive failure it waits 2 seconds and polls again.
  • The bus credential is used only for those three bus capabilities; the person's credential stays
    inside the claimed task body. Without the three variables the bus is off and only
    POST /v1/turn serves turns.

The model

Every model call goes through Lake's model gateway (lake#248, model.py): an OpenAI client at
<LAKE_URL>/openai/v1 (falling back to the turn's lake_url) with the turn's own lake_token as
the API key, so Lake counts the tokens in the turn's workspace and enforces the account's limit. The
service holds no provider key. Only the design turn calls the model.

Command-line entry points ([project.scripts])

  • agent-experiments-serve: the service (the image's entry point).
  • experiments-agent-evals: the three acceptance evals; writes dated JSON and Markdown reports under
    reports/ (git-ignored).

Environment variables

Name Meaning
PORT HTTP port; default 8080
AGENT_SERVICE_TOKEN Bearer token required on POST /v1/turn when set
LAKE_URL Lake's internal root URL: the bus calls and the model gateway
AGENT_BUS_TOKEN This agent's bus credential
AGENT_BUS_AGENT The agent key on the bus; default agent-experiments
EXPERIMENTS_AGENT_MODEL Model name; default gpt-5.5
TRACE_DIR Where turn traces are written; default /tmp/agent-experiments-traces
OPENAI_API_KEY (evals only) Read by the OpenAI client in experiments-agent-evals; the service never reads it

The image

oramasearch/agent-experiments on Docker Hub, private. The Dockerfile builds on
python:3.12-slim with uv 0.11.7 (uv sync --frozen --no-dev --no-editable), copies only the
virtual environment into a second python:3.12-slim stage, runs as UID 10001 (agent), exposes
8080 and starts agent-experiments-serve. It needs no named build contexts.

Where it runs

Every Lake runs it, deployed by the Orama operator as a library agent:

  • The operator's default library image (internal/controller/lake_defaults.go on dev) is
    oramasearch/agent-experiments:62a4d64. The LakeSets lake-nonprod (members dev, staging
    and testing on orama-nonprod-cluster-01) and lake-prod (members shared on
    orama-production-cluster-02 and shared-us on orama-production-cluster-03) take that default.
  • Lake prod on the managing cluster orama-production-cluster-01 (namespace lake, the
    development environment) pins the same oramasearch/agent-experiments:62a4d64 in its
    spec.agents.

v0.1.0 is the same tree as that running commit, 62a4d64, plus these notes, the release test and
the version field on /healthz. Moving the operator default to :v0.1.0 is a separate operator
change.

What it does not include

  • Lake's own experiment names (start_experiment, experiment_id): the agent speaks the engine's
    study names through Lake's workspace adapter, and Lake maps context.experiment_id to
    context.study.
  • Calls to stop_study, list_studies or get_run_metrics from its own code (allowed, unused).
  • Any model call outside the design turn; any model reached other than through Lake (the evals
    excepted).
  • Persistent conversations: history lives in one process's memory, is lost on restart and is not
    shared between replicas.
  • Delivery of its turn traces to the workspace Zetta: they are local JSON files (see Known issues).
  • A live-model eval run in this release's verification.

Known issues

Security-relevant first.

  1. Turn traces hold workspace content outside the workspace Zetta. Every turn writes its message
    and full answer (study definitions, Lake facts) as JSON to TRACE_DIR on the container's local
    disk (/tmp). The org rule puts all workspace data in its workspace Zetta. agent-experiments#11.
  2. The workspace is never named explicitly
    (agent-experiments#3): the agent does
    not pass workspace_id on its Lake calls or bus calls; it relies on Lake taking the task row's
    immutable workspace and the task credential being bound to it.
  3. experiment_status fails on Lake
    (lake#797): the status turn ends failed with
    receive_events: ... type must be number, but is string. The agent sends receive_events as a
    GET whose from_sequence, max and wait_ms travel as query strings.
  4. No backoff on an idle bus and no lease renewal while working
    (agent-experiments#3, last
    acceptance item): the bus thread long-polls every 10 seconds with no backoff when empty, and a
    turn publishes one progress event at its start and nothing more until its answer. Lake renews
    the claim's lease on every capability call made with the task credential, so a design turn is
    reclaimed only if one step between two calls, such as one model call, outlasts the lease (Lake's
    LAKE_AGENT_LEASE_SECONDS, default 600). Not observed.

Verification

Proof Result
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12) 29 passed, 0 failed, 0 skipped: tests/test_agent.py 17, tests/test_evals.py 4, tests/test_lake_bus.py 2, tests/test_service.py 4, tests/test_release.py 2. Includes test_release_notes_exist_for_the_project_version and test_health_reports_the_installed_version_and_prompt_version
Offline evals with no model (run_divergence_stop, run_promotion_confirmation from agent_experiments.evals) 2 of 2 passed
Eval known-good-search-space (experiments-agent-evals) Not run (needs a live model)
Linux image build (cluster BuildKit, from the tag) Built from the tag v0.1.0 with build-agent.sh: linux/amd64, pushed as oramasearch/agent-experiments:41dcf92
Image digest docker.io/oramasearch/agent-experiments:41dcf92@sha256:a20666296bd2d84e1dfd5f15446c7461d2d388e4394f0b45c3cf1c0e8aab56ac, also tagged :v0.1.0 (same digest); the Docker Hub repository answers 404 anonymously (private)