Repository navigation
Releases
Release v0.3.0
Released 2026-11-16. A minor release. Since v0.2.2: conversation history bound to its workspace and bounded, the licence, the security policy, base images pinned by digest and the public-tree cleanup. Image: docker.io/oramasearch/agent-experiments:v0.3.0, not published (see Known issues).
This release is the tree of dev with these notes and the version.
What it includes
Every pull request merged into dev since the v0.2.2 tag:
| PR | What it does |
|---|---|
| #41 | The public-tree cleanup (see below) |
| #40 | MIT LICENSE and COPYRIGHT (OramaSearch, Inc.), an SPDX-License-Identifier: MIT line in every source file (tools/spdx-headers.py), THIRD_PARTY_LICENSES, CONTRIBUTING.md, CLA.md, CODE_OF_CONDUCT.md, and the SBOM (sbom.spdx.json, sbom.cdx.json, tools/sbom.py) |
| #37 | Security: conversation history is kept per workspace and bounded |
| #38 | The Dockerfile names its base images by digest as well as by tag (python:3.12-slim, ghcr.io/astral-sh/uv:0.11.7), so a later build of the same source gets the same base |
| #36 | SECURITY.md: how to report a vulnerability |
Conversation history is bound to its workspace, and bounded — #37
- The pod serves every workspace and looked history up by conversation id alone, so another workspace that knew the id could read or write into it, and new ids grew memory without limit (#35).
- New
conversations.py: history is keyed by customer, the checked workspace path (lake_url),workspace_idanduser_idwhen the turn has them, and the conversation id. The store holds at most 256 conversations (least recently used dropped first), expires a conversation idle for an hour, and keeps each history to its system prompt plus at most 60 messages and 200,000 characters, dropping the oldest whole exchanges so a tool reply never loses its call. run_turnworks on a copy of the history and stores it back whole under the lock.
The public-tree cleanup — #41
Nothing internal is left in the files that will be published: .public-exclude was added and then removed again once the agent files were rewritten to ship publicly; issue links became plain text, the release notes name no environment, and "private" wording about the repository and its image is gone. Documentation and one Dockerfile comment only.
The licence — #40
Everything in this repository that Orama wrote is under the MIT licence. Contributors sign CLA.md once. THIRD_PARTY_LICENSES lists every runtime dependency with its licence text; none is incompatible with MIT distribution.
Before it rolls: the settings it needs
Nothing new. The variables are unchanged from v0.2.2.
What it does not include
Deferred, not in this release:
- #39: claiming bus tasks by reference and naming the task on every call; it needs the matching Lake change.
Otherwise unchanged from v0.2.2.
Known issues
Security-relevant first. All open at this tag.
- #42 (release blocker for publishing): the image is built on
python:3.12-slim, which bundles GPL and LGPL userland, so the image is not published. It is built and kept in the project's non-public Docker Hub repository until it moves to a scratch base with static binaries. - #3: the task's workspace is not passed explicitly on data-plane calls. Lake scopes the task credential to the task's workspace.
- #33: in an end-to-end run against lake v0.8.1 the final turn did not explain the experiment (empty answer).
- #20: the agent reads
EXPERIMENTS_AGENT_MODELand not the operator'sAGENT_MODEL.
Where it has run
The v0.3.0 tree had not run on a development environment when it was released. Lakes run the operator's default image of this agent; the changes since v0.2.2 are proven by the test suite only. Moving a Lake to :v0.3.0 is a separate operator change.
The image
Built from the tag v0.3.0 (3f83347) with BuildKit (linux/amd64, the repository's Dockerfile), then tagged :v0.3.0 on that digest without rebuilding:
docker.io/oramasearch/agent-experiments:3f83347@sha256:5d2290ba06fe0a60b04f9e6f28834da73b1ee3474c93e4af6fef45a9f899ec41, also tagged :v0.3.0 (same digest).
It is pushed only to the project's non-public Docker Hub repository, which answers 404 anonymously. Pinning it on a Lake is a separate change.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) |
67 passed, 0 failed, 0 skipped; 67 tests. Includes tests/test_release.py (the declared version has these notes, indexed) |
SPDX headers (python3 tools/spdx-headers.py --check) |
Every source file carries the header |
SBOM (python3 tools/sbom.py --version v0.3.0, then --check) |
Regenerated on the release commit; --check passes |
| Batch generation, and a live turn through Lake's model gateway | Not run (needs a live model and a Lake) |
| Linux image build (BuildKit, linux/amd64, from the tag) | Built from v0.3.0: oramasearch/agent-experiments:3f83347, also :v0.3.0, sha256:5d2290ba06fe0a60b04f9e6f28834da73b1ee3474c93e4af6fef45a9f899ec41 |
Release v0.2.2
agent-experiments v0.2.2
Released 2026-10-26. A patch release. Since v0.2.1: the Lake URL is never passed to the agent as its Lake client, so Experiments turns run again. Replaces v0.2.1, whose uv.lock did not parse (a release-script error), so it has no image.
Private repository and image, docker.io/oramasearch/agent-experiments:v0.2.2.
What it includes
| PR | Change |
|---|---|
| #27 | the Lake URL is never passed to the agent as its Lake client, so Experiments turns run again |
Found by lake v0.8.0's end-to-end gate on lake-testing.
Known issues
As in v0.2.1.
Verification
- The repository's test suite on #27, with a regression test; not re-run on the release branch.
- Image built from the tag on the cluster BuildKit; re-checked by the e2e gate on
lake-testing.
Release v0.2.1
Superseded by v0.2.2, never deployed. This tag's uv.lock does not parse (an error in the release script), so no image was built. Use v0.2.2, which carries the same change.
Release v0.2.0
agent-experiments v0.2.0
Released 2026-10-25. A minor release. Three security fixes since v0.1.0, from the
2026-10-21 security review. Private repository and image,
docker.io/oramasearch/agent-experiments:v0.2.0. Tracking issue:
#21.
This release is the tree of dev at 84fd433 (v0.1.0, the v0.1.0 image digest in its notes
(#14), and the three pull requests
below) with these notes and the version.
What it includes
Every pull request merged into dev since v0.1.0:
| PR | What |
|---|---|
| #13 | The v0.1.0 release pull request (dev → main), carried back into dev |
| #14 | The v0.1.0 image digest recorded in its notes |
| #16 | No turn trace on the pod's disk |
| #17 | The service token compared in constant time, and every turn refused when it is unset |
| #19 | A turn's credential goes only to Lake at LAKE_URL |
No turn trace on the pod's disk — #16
A turn no longer writes <TRACE_DIR>/<trace_id>.json. That file held the person's message and the
whole answer, workspace content on the disk of a pod every tenant shares
(#11, v0.1.0's first known issue).
Lake already keeps the whole answer in the task's sealed record in the turn's workspace Zetta, so
the local copy is dropped, not moved. The answer still carries trace_id. TRACE_DIR is no longer
read; the operator still sets it and the agent ignores it. The Dockerfile drops its default.
The service token: constant time, and required — #17
POST /v1/turn checks the bearer with hmac.compare_digest instead of !=
(#15). The route now fails closed:
with AGENT_SERVICE_TOKEN unset or empty, every turn is refused with 503 (v0.1.0 left it open). A
wrong or missing bearer is 401, as before. Turns from Lake's agent bus are unaffected.
A turn's credential only to Lake — #19
Neither POST /v1/turn nor a bus turn sends the turn's bearer to an address the caller names
(#18):
- the model gateway is always this service's
LAKE_URL; the fallback to the turn'slake_urlis
gone; - the turn's
lake_url(Lake's workspace path) is accepted only onLAKE_URL's origin and under
its path, with no user info, query, fragment, backslash or dot segments; without one the turn uses
LAKE_URL; - with
LAKE_URLunset, every turn is refused before any request goes out.
Before it rolls: settings
| Setting | Needed by | State |
|---|---|---|
AGENT_SERVICE_TOKEN |
#17: unset now refuses every turn (503) |
The operator renders it on every library agent |
LAKE_URL |
#19: unset now refuses every turn | The operator renders it |
TRACE_DIR |
#16: no longer read | The operator still sets /tmp/traces; harmless |
AGENT_MODEL |
Rendered by the operator, but not read by this agent (#20) | The agent reads EXPERIMENTS_AGENT_MODEL (default gpt-5.5), as in v0.1.0 |
A service run by hand needs AGENT_SERVICE_TOKEN and LAKE_URL set, or it refuses every turn.
What it does not include
Unchanged from v0.1.0, except that turn traces are no longer
kept anywhere by the agent:
- Lake's own experiment names (
start_experiment,experiment_id): the agent speaks the engine's
studynames through Lake's workspace adapter. - Calls to
stop_study,list_studiesorget_run_metricsfrom its own code (allowed, unused). - Persistent conversations: history lives in one process's memory.
- Reading the operator's
AGENT_MODEL(#20). - A live-model eval run in this release's verification.
Known issues
Security-relevant first. All open at this tag.
- Confirmation gates do not establish a separate human confirmer
(lake#989): the reviewed object incontext
authorizescreate_studyandpromote_trial, but Lake does not prove a person, rather than the
caller that asked, confirmed it. - Library agents hold live task credentials for every tenant they serve
(lake#871), and a delegated turn is not
attenuated to its parent task (lake#922). - The workspace is never named explicitly
(#3): noworkspace_idon Lake or bus
calls; the agent relies on the task credential being bound to the task's workspace. Also: no
backoff on an idle bus. - The agent ignores the operator's
AGENT_MODEL
(#20), found while cutting this
release. - The design turn asks instead of proposing on the e2e churn table
(lake#1184): the study validator finds no entity
column (lake e2eagents.pyT12). The e2e script also confirms withoutproposal_task_idand
proposal_sha256, so Lake refuses the turn
(lake#1183).
v0.1.0's trace issue (#11) is fixed by
#16. v0.1.0's experiment_status failure on Lake (lake#797)
is closed.
Where it has run
The v0.2.0 tree has not run on the dev environment. No dev build of this repo was pinned on the
dev member of lake-nonprod; every Lake still runs the operator's default
oramasearch/agent-experiments:v0.1.0. Moving that default to :v0.2.0 is a separate operator
change, rolled through dev → testing → staging → prod.
The image
Built from the tag v0.2.0 (3df9e05) on the cluster BuildKit with build-agent.sh (linux/amd64, the repository's Dockerfile), then tagged :v0.2.0 on that digest without rebuilding:
docker.io/oramasearch/agent-experiments:3df9e05@sha256:51906a21bed3b4a768e82d52545a1fb0607d0bae1ca99579b61b77cea349fc8a, also tagged :v0.2.0 (same digest). The Docker Hub repository answers 404 anonymously (private).
It is not rolled anywhere yet: pinning it on an environment is a separate change to the operator's set files, through dev, testing, staging and prod in order.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12) |
59 passed, 0 failed, 0 skipped. New since v0.1.0: tests/test_service_auth.py 13 (#17), tests/test_turn_destinations.py 17 (#19); tests/test_service.py asserts a turn writes no file (#16); tests/test_release.py 2 (the installed package is 0.2.0, these notes exist and are indexed) |
Eval known-good-search-space (experiments-agent-evals) |
Not run (needs a live model) |
| Linux image build (cluster BuildKit, from the tag) | Built from v0.2.0 with build-agent.sh: oramasearch/agent-experiments:3df9e05, also :v0.2.0, sha256:51906a21bed3b4a768e82d52545a1fb0607d0bae1ca99579b61b77cea349fc8a; private (404 anonymously) |
Release v0.1.0
agent-experiments v0.1.0
Released 2026-10-17. The first release. One Python service, the Experiments Agent, and one
private image, docker.io/oramasearch/agent-experiments:v0.1.0. Proprietary to Orama; private
repository and image.
The Experiments Agent designs AutoML experiments ("studies" in the engine's and the agent's own
words) for a dataset, starts one only after a person confirms the exact proposal, watches its trials
and their training metrics, stops trials that diverge, explains the result, and promotes the best
trial only after a second confirmation. It is a Lake library agent: Lake dispatches its turns
(Train → Experiments, and delegations from Ask Orama and the Project Agent), and every read and
action goes through Lake capabilities with the asking person's delegated credential. It has no
direct connection to Zetta, a database, a scheduler or an event broker. Epic:
lake#305.
The contracts: service and bus, designing and running
studies, evaluation scenarios.
What it includes
Turn kinds
kind |
What it does | Lake capabilities it may call |
|---|---|---|
experiment (no context.confirmed_proposal) |
The model designs a study from the request and Lake facts, through one tool, lake_read. Every reference is resolved and the complete study is preflighted with validate_study; a defect is fed back to the model (up to 2 repairs, up to 10 tool rounds each). Answers result.kind proposal with confirm: true, or question. Performs no action |
reads only |
experiment with context.confirmed_proposal |
Re-validates the reviewed proposal (fields, references, validate_study), then calls create_study with that exact study object. No model call. Answers result.kind: running |
reads, then create_study |
experiment_status |
Reads the study (context.study, or context.study_id) and its trials, tails study-events.<study> and run-metrics.<run> of every running trial with bounded receive_events (at most 100 events, 20 s wait, per-topic cursors returned in result.cursors), and calls stop_trial for a trial whose batch has a non-finite value, a gradient norm of 1,000 or more, a tenfold gradient-norm jump to at least 100, or three worsening validation-loss points with a gap over training loss of at least 0.25. Answers result.kind: status |
reads, stop_trial |
experiment_explain (no context.confirmed_promotion) |
Reads get_study and get_study_analysis; reports the best trial, parameter importance, the next search direction, trial-variance and validation-overuse caveats; returns a promotion proposal with confirm: true. Answers result.kind: explanation. Performs no action |
reads only |
experiment_explain with context.confirmed_promotion |
Checks the confirmed study matches the turn's study and the confirmed run is still the best trial, then calls promote_trial. Answers result.kind: promoted |
get_study_analysis, promote_trial |
Plain text such as "yes" never authorizes create_study or promote_trial; only the reviewed object
in context does.
Lake capabilities (exact allow-lists in src/agent_experiments/agent.py)
- Reads (
READ_CAPABILITIES, sent asGET <lake_url>/v1/<capability>):analyze_dataset,
get_project,get_run_metrics,get_solution,get_study,get_study_analysis,
list_methodologies,list_projects,list_studies,list_trials,plan_features,
receive_events,validate_study. The model can call any of them throughlake_read;
get_run_metricsandlist_studiesare not called by the agent's own code. - Actions (
ACTION_CAPABILITIES, sent asPOST):create_study,promote_trial,
stop_trial,stop_study.stop_studyis allowed but never called by this release. - Any other capability is refused inside the agent (
not_allowed) before a request is made. - These are the engine's names, called through Lake's workspace adapter; Lake's own names for them
are theexperimentones (lakedocs/api/experiments.md, lake#561).
The prompt
PROMPT_VERSION = "v1"(inservice.pyandagent.py); the system prompt is
src/agent_experiments/prompts/v1.md, used only for the design turn.- Every turn's trace records
prompt_version, andGET /healthzreports it.
HTTP API (port PORT, default 8080)
| Route | What |
|---|---|
GET /healthz |
{ok, agent: "agent-experiments", version, model, prompt_version}. version is the installed package version (new in this release) |
POST /v1/turn |
One turn. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (otherwise 401). Body: kind, message, lake_token (the person's delegated credential), lake_url (optional when LAKE_URL is set), optional customer_id, conversation_id, context. Answers 200 with {ok: true, conversation_id, trace_id, reply, result}, 422 with {ok: false, error}, 400 when the body is not JSON |
Any other path is 404. Conversation history is kept per conversation_id in the process's memory.
The agent bus: how Lake reaches it
- The operator runs it as a library agent of every Lake: one Deployment and one Service, port 8080,
probed on/healthz, and gives Lake its address inLAKE_AGENT_ENDPOINTS
(agent-experiments=<url>). The operator setsLAKE_URL,AGENT_SERVICE_TOKEN,
AGENT_BUS_TOKENandAGENT_BUS_AGENT=agent-experimentson it. - Lake dispatches each turn as a task on the topic
agent-tasks.agent-experimentsof Lake's own
Zetta. The topic carries task ids only, never content (lakedocs/api/agent-bus.md). - When
LAKE_URL,AGENT_BUS_TOKENandAGENT_BUS_AGENTare set,servestarts one bus thread
(lake_bus.py). It long-pollsreceive_agent_tasks {agent, from_sequence, wait_ms: 10000}
through Lake, starting at sequence 1 on every start and continuing fromnext_sequence. For each
task it callsclaim_agent_task {task_id, agent}; aconflictornot_foundanswer means
another replica or attempt has it and the task is skipped. A claimed task's body (with its task
credential aslake_token) runs on a pool of 4 worker threads through the samerun_turnas
POST /v1/turn. The worker publishes, throughpublish_agent_event, oneprogressevent
("Handling the experiment turn"), then aresultevent with the whole answer or afailedevent
with the error. After a receive failure it waits 2 seconds and polls again. - The bus credential is used only for those three bus capabilities; the person's credential stays
inside the claimed task body. Without the three variables the bus is off and only
POST /v1/turnserves turns.
The model
Every model call goes through Lake's model gateway (lake#248, model.py): an OpenAI client at
<LAKE_URL>/openai/v1 (falling back to the turn's lake_url) with the turn's own lake_token as
the API key, so Lake counts the tokens in the turn's workspace and enforces the account's limit. The
service holds no provider key. Only the design turn calls the model.
Command-line entry points ([project.scripts])
agent-experiments-serve: the service (the image's entry point).experiments-agent-evals: the three acceptance evals; writes dated JSON and Markdown reports under
reports/(git-ignored).
Environment variables
| Name | Meaning |
|---|---|
PORT |
HTTP port; default 8080 |
AGENT_SERVICE_TOKEN |
Bearer token required on POST /v1/turn when set |
LAKE_URL |
Lake's internal root URL: the bus calls and the model gateway |
AGENT_BUS_TOKEN |
This agent's bus credential |
AGENT_BUS_AGENT |
The agent key on the bus; default agent-experiments |
EXPERIMENTS_AGENT_MODEL |
Model name; default gpt-5.5 |
TRACE_DIR |
Where turn traces are written; default /tmp/agent-experiments-traces |
OPENAI_API_KEY (evals only) |
Read by the OpenAI client in experiments-agent-evals; the service never reads it |
The image
oramasearch/agent-experiments on Docker Hub, private. The Dockerfile builds on
python:3.12-slim with uv 0.11.7 (uv sync --frozen --no-dev --no-editable), copies only the
virtual environment into a second python:3.12-slim stage, runs as UID 10001 (agent), exposes
8080 and starts agent-experiments-serve. It needs no named build contexts.
Where it runs
Every Lake runs it, deployed by the Orama operator as a library agent:
- The operator's default library image (
internal/controller/lake_defaults.goondev) is
oramasearch/agent-experiments:62a4d64. The LakeSetslake-nonprod(membersdev,staging
andtestingonorama-nonprod-cluster-01) andlake-prod(memberssharedon
orama-production-cluster-02andshared-usonorama-production-cluster-03) take that default. - Lake
prodon the managing clusterorama-production-cluster-01(namespacelake, the
development environment) pins the sameoramasearch/agent-experiments:62a4d64in its
spec.agents.
v0.1.0 is the same tree as that running commit, 62a4d64, plus these notes, the release test and
the version field on /healthz. Moving the operator default to :v0.1.0 is a separate operator
change.
What it does not include
- Lake's own experiment names (
start_experiment,experiment_id): the agent speaks the engine's
studynames through Lake's workspace adapter, and Lake mapscontext.experiment_idto
context.study. - Calls to
stop_study,list_studiesorget_run_metricsfrom its own code (allowed, unused). - Any model call outside the design turn; any model reached other than through Lake (the evals
excepted). - Persistent convers...