Skip to content

Releases

Release v0.3.0

Choose a tag to compare

@homer6 homer6 released this 16 Nov 07:29
3f83347

Released 2026-11-16. A minor release. Since v0.2.2: conversation history bound to its workspace and bounded, the licence, the security policy, base images pinned by digest and the public-tree cleanup. Image: docker.io/oramasearch/agent-experiments:v0.3.0, not published (see Known issues).

This release is the tree of dev with these notes and the version.

What it includes

Every pull request merged into dev since the v0.2.2 tag:

PR What it does
#41 The public-tree cleanup (see below)
#40 MIT LICENSE and COPYRIGHT (OramaSearch, Inc.), an SPDX-License-Identifier: MIT line in every source file (tools/spdx-headers.py), THIRD_PARTY_LICENSES, CONTRIBUTING.md, CLA.md, CODE_OF_CONDUCT.md, and the SBOM (sbom.spdx.json, sbom.cdx.json, tools/sbom.py)
#37 Security: conversation history is kept per workspace and bounded
#38 The Dockerfile names its base images by digest as well as by tag (python:3.12-slim, ghcr.io/astral-sh/uv:0.11.7), so a later build of the same source gets the same base
#36 SECURITY.md: how to report a vulnerability

Conversation history is bound to its workspace, and bounded — #37

  • The pod serves every workspace and looked history up by conversation id alone, so another workspace that knew the id could read or write into it, and new ids grew memory without limit (#35).
  • New conversations.py: history is keyed by customer, the checked workspace path (lake_url), workspace_id and user_id when the turn has them, and the conversation id. The store holds at most 256 conversations (least recently used dropped first), expires a conversation idle for an hour, and keeps each history to its system prompt plus at most 60 messages and 200,000 characters, dropping the oldest whole exchanges so a tool reply never loses its call.
  • run_turn works on a copy of the history and stores it back whole under the lock.

The public-tree cleanup — #41

Nothing internal is left in the files that will be published: .public-exclude was added and then removed again once the agent files were rewritten to ship publicly; issue links became plain text, the release notes name no environment, and "private" wording about the repository and its image is gone. Documentation and one Dockerfile comment only.

The licence — #40

Everything in this repository that Orama wrote is under the MIT licence. Contributors sign CLA.md once. THIRD_PARTY_LICENSES lists every runtime dependency with its licence text; none is incompatible with MIT distribution.

Before it rolls: the settings it needs

Nothing new. The variables are unchanged from v0.2.2.

What it does not include

Deferred, not in this release:

  • #39: claiming bus tasks by reference and naming the task on every call; it needs the matching Lake change.

Otherwise unchanged from v0.2.2.

Known issues

Security-relevant first. All open at this tag.

  1. #42 (release blocker for publishing): the image is built on python:3.12-slim, which bundles GPL and LGPL userland, so the image is not published. It is built and kept in the project's non-public Docker Hub repository until it moves to a scratch base with static binaries.
  2. #3: the task's workspace is not passed explicitly on data-plane calls. Lake scopes the task credential to the task's workspace.
  3. #33: in an end-to-end run against lake v0.8.1 the final turn did not explain the experiment (empty answer).
  4. #20: the agent reads EXPERIMENTS_AGENT_MODEL and not the operator's AGENT_MODEL.

Where it has run

The v0.3.0 tree had not run on a development environment when it was released. Lakes run the operator's default image of this agent; the changes since v0.2.2 are proven by the test suite only. Moving a Lake to :v0.3.0 is a separate operator change.

The image

Built from the tag v0.3.0 (3f83347) with BuildKit (linux/amd64, the repository's Dockerfile), then tagged :v0.3.0 on that digest without rebuilding:

docker.io/oramasearch/agent-experiments:3f83347@sha256:5d2290ba06fe0a60b04f9e6f28834da73b1ee3474c93e4af6fef45a9f899ec41, also tagged :v0.3.0 (same digest).

It is pushed only to the project's non-public Docker Hub repository, which answers 404 anonymously. Pinning it on a Lake is a separate change.

Verification

Proof Result
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) 67 passed, 0 failed, 0 skipped; 67 tests. Includes tests/test_release.py (the declared version has these notes, indexed)
SPDX headers (python3 tools/spdx-headers.py --check) Every source file carries the header
SBOM (python3 tools/sbom.py --version v0.3.0, then --check) Regenerated on the release commit; --check passes
Batch generation, and a live turn through Lake's model gateway Not run (needs a live model and a Lake)
Linux image build (BuildKit, linux/amd64, from the tag) Built from v0.3.0: oramasearch/agent-experiments:3f83347, also :v0.3.0, sha256:5d2290ba06fe0a60b04f9e6f28834da73b1ee3474c93e4af6fef45a9f899ec41

Release v0.2.2

Choose a tag to compare

@homer6 homer6 released this 27 Oct 07:18
fe60fab

agent-experiments v0.2.2

Released 2026-10-26. A patch release. Since v0.2.1: the Lake URL is never passed to the agent as its Lake client, so Experiments turns run again. Replaces v0.2.1, whose uv.lock did not parse (a release-script error), so it has no image.
Private repository and image, docker.io/oramasearch/agent-experiments:v0.2.2.

What it includes

PR Change
#27 the Lake URL is never passed to the agent as its Lake client, so Experiments turns run again

Found by lake v0.8.0's end-to-end gate on lake-testing.

Known issues

As in v0.2.1.

Verification

  • The repository's test suite on #27, with a regression test; not re-run on the release branch.
  • Image built from the tag on the cluster BuildKit; re-checked by the e2e gate on lake-testing.

Release v0.2.1

Choose a tag to compare

@homer6 homer6 released this 27 Oct 06:57
a3ab120

Superseded by v0.2.2, never deployed. This tag's uv.lock does not parse (an error in the release script), so no image was built. Use v0.2.2, which carries the same change.

Release v0.2.0

Choose a tag to compare

@homer6 homer6 released this 25 Oct 21:07
3df9e05

agent-experiments v0.2.0

Released 2026-10-25. A minor release. Three security fixes since v0.1.0, from the
2026-10-21 security review. Private repository and image,
docker.io/oramasearch/agent-experiments:v0.2.0. Tracking issue:
#21.

This release is the tree of dev at 84fd433 (v0.1.0, the v0.1.0 image digest in its notes
(#14), and the three pull requests
below) with these notes and the version.

What it includes

Every pull request merged into dev since v0.1.0:

PR What
#13 The v0.1.0 release pull request (dev → main), carried back into dev
#14 The v0.1.0 image digest recorded in its notes
#16 No turn trace on the pod's disk
#17 The service token compared in constant time, and every turn refused when it is unset
#19 A turn's credential goes only to Lake at LAKE_URL

No turn trace on the pod's disk — #16

A turn no longer writes <TRACE_DIR>/<trace_id>.json. That file held the person's message and the
whole answer, workspace content on the disk of a pod every tenant shares
(#11, v0.1.0's first known issue).
Lake already keeps the whole answer in the task's sealed record in the turn's workspace Zetta, so
the local copy is dropped, not moved. The answer still carries trace_id. TRACE_DIR is no longer
read; the operator still sets it and the agent ignores it. The Dockerfile drops its default.

The service token: constant time, and required — #17

POST /v1/turn checks the bearer with hmac.compare_digest instead of !=
(#15). The route now fails closed:
with AGENT_SERVICE_TOKEN unset or empty, every turn is refused with 503 (v0.1.0 left it open). A
wrong or missing bearer is 401, as before. Turns from Lake's agent bus are unaffected.

A turn's credential only to Lake — #19

Neither POST /v1/turn nor a bus turn sends the turn's bearer to an address the caller names
(#18):

  • the model gateway is always this service's LAKE_URL; the fallback to the turn's lake_url is
    gone;
  • the turn's lake_url (Lake's workspace path) is accepted only on LAKE_URL's origin and under
    its path, with no user info, query, fragment, backslash or dot segments; without one the turn uses
    LAKE_URL;
  • with LAKE_URL unset, every turn is refused before any request goes out.

Before it rolls: settings

Setting Needed by State
AGENT_SERVICE_TOKEN #17: unset now refuses every turn (503) The operator renders it on every library agent
LAKE_URL #19: unset now refuses every turn The operator renders it
TRACE_DIR #16: no longer read The operator still sets /tmp/traces; harmless
AGENT_MODEL Rendered by the operator, but not read by this agent (#20) The agent reads EXPERIMENTS_AGENT_MODEL (default gpt-5.5), as in v0.1.0

A service run by hand needs AGENT_SERVICE_TOKEN and LAKE_URL set, or it refuses every turn.

What it does not include

Unchanged from v0.1.0, except that turn traces are no longer
kept anywhere by the agent:

  • Lake's own experiment names (start_experiment, experiment_id): the agent speaks the engine's
    study names through Lake's workspace adapter.
  • Calls to stop_study, list_studies or get_run_metrics from its own code (allowed, unused).
  • Persistent conversations: history lives in one process's memory.
  • Reading the operator's AGENT_MODEL (#20).
  • A live-model eval run in this release's verification.

Known issues

Security-relevant first. All open at this tag.

  1. Confirmation gates do not establish a separate human confirmer
    (lake#989): the reviewed object in context
    authorizes create_study and promote_trial, but Lake does not prove a person, rather than the
    caller that asked, confirmed it.
  2. Library agents hold live task credentials for every tenant they serve
    (lake#871), and a delegated turn is not
    attenuated to its parent task (lake#922).
  3. The workspace is never named explicitly
    (#3): no workspace_id on Lake or bus
    calls; the agent relies on the task credential being bound to the task's workspace. Also: no
    backoff on an idle bus.
  4. The agent ignores the operator's AGENT_MODEL
    (#20), found while cutting this
    release.
  5. The design turn asks instead of proposing on the e2e churn table
    (lake#1184): the study validator finds no entity
    column (lake e2e agents.py T12). The e2e script also confirms without proposal_task_id and
    proposal_sha256, so Lake refuses the turn
    (lake#1183).

v0.1.0's trace issue (#11) is fixed by
#16. v0.1.0's experiment_status failure on Lake (lake#797)
is closed.

Where it has run

The v0.2.0 tree has not run on the dev environment. No dev build of this repo was pinned on the
dev member of lake-nonprod; every Lake still runs the operator's default
oramasearch/agent-experiments:v0.1.0. Moving that default to :v0.2.0 is a separate operator
change, rolled through dev → testing → staging → prod.

The image

Built from the tag v0.2.0 (3df9e05) on the cluster BuildKit with build-agent.sh (linux/amd64, the repository's Dockerfile), then tagged :v0.2.0 on that digest without rebuilding:

docker.io/oramasearch/agent-experiments:3df9e05@sha256:51906a21bed3b4a768e82d52545a1fb0607d0bae1ca99579b61b77cea349fc8a, also tagged :v0.2.0 (same digest). The Docker Hub repository answers 404 anonymously (private).

It is not rolled anywhere yet: pinning it on an environment is a separate change to the operator's set files, through dev, testing, staging and prod in order.

Verification

Proof Result
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12) 59 passed, 0 failed, 0 skipped. New since v0.1.0: tests/test_service_auth.py 13 (#17), tests/test_turn_destinations.py 17 (#19); tests/test_service.py asserts a turn writes no file (#16); tests/test_release.py 2 (the installed package is 0.2.0, these notes exist and are indexed)
Eval known-good-search-space (experiments-agent-evals) Not run (needs a live model)
Linux image build (cluster BuildKit, from the tag) Built from v0.2.0 with build-agent.sh: oramasearch/agent-experiments:3df9e05, also :v0.2.0, sha256:51906a21bed3b4a768e82d52545a1fb0607d0bae1ca99579b61b77cea349fc8a; private (404 anonymously)

Release v0.1.0

Choose a tag to compare

@homer6 homer6 released this 17 Oct 22:59
41dcf92

agent-experiments v0.1.0

Released 2026-10-17. The first release. One Python service, the Experiments Agent, and one
private image, docker.io/oramasearch/agent-experiments:v0.1.0. Proprietary to Orama; private
repository and image.

The Experiments Agent designs AutoML experiments ("studies" in the engine's and the agent's own
words) for a dataset, starts one only after a person confirms the exact proposal, watches its trials
and their training metrics, stops trials that diverge, explains the result, and promotes the best
trial only after a second confirmation. It is a Lake library agent: Lake dispatches its turns
(Train → Experiments, and delegations from Ask Orama and the Project Agent), and every read and
action goes through Lake capabilities with the asking person's delegated credential. It has no
direct connection to Zetta, a database, a scheduler or an event broker. Epic:
lake#305.

The contracts: service and bus, designing and running
studies
, evaluation scenarios.

What it includes

Turn kinds

kind What it does Lake capabilities it may call
experiment (no context.confirmed_proposal) The model designs a study from the request and Lake facts, through one tool, lake_read. Every reference is resolved and the complete study is preflighted with validate_study; a defect is fed back to the model (up to 2 repairs, up to 10 tool rounds each). Answers result.kind proposal with confirm: true, or question. Performs no action reads only
experiment with context.confirmed_proposal Re-validates the reviewed proposal (fields, references, validate_study), then calls create_study with that exact study object. No model call. Answers result.kind: running reads, then create_study
experiment_status Reads the study (context.study, or context.study_id) and its trials, tails study-events.<study> and run-metrics.<run> of every running trial with bounded receive_events (at most 100 events, 20 s wait, per-topic cursors returned in result.cursors), and calls stop_trial for a trial whose batch has a non-finite value, a gradient norm of 1,000 or more, a tenfold gradient-norm jump to at least 100, or three worsening validation-loss points with a gap over training loss of at least 0.25. Answers result.kind: status reads, stop_trial
experiment_explain (no context.confirmed_promotion) Reads get_study and get_study_analysis; reports the best trial, parameter importance, the next search direction, trial-variance and validation-overuse caveats; returns a promotion proposal with confirm: true. Answers result.kind: explanation. Performs no action reads only
experiment_explain with context.confirmed_promotion Checks the confirmed study matches the turn's study and the confirmed run is still the best trial, then calls promote_trial. Answers result.kind: promoted get_study_analysis, promote_trial

Plain text such as "yes" never authorizes create_study or promote_trial; only the reviewed object
in context does.

Lake capabilities (exact allow-lists in src/agent_experiments/agent.py)

  • Reads (READ_CAPABILITIES, sent as GET <lake_url>/v1/<capability>): analyze_dataset,
    get_project, get_run_metrics, get_solution, get_study, get_study_analysis,
    list_methodologies, list_projects, list_studies, list_trials, plan_features,
    receive_events, validate_study. The model can call any of them through lake_read;
    get_run_metrics and list_studies are not called by the agent's own code.
  • Actions (ACTION_CAPABILITIES, sent as POST): create_study, promote_trial,
    stop_trial, stop_study. stop_study is allowed but never called by this release.
  • Any other capability is refused inside the agent (not_allowed) before a request is made.
  • These are the engine's names, called through Lake's workspace adapter; Lake's own names for them
    are the experiment ones (lake docs/api/experiments.md, lake#561).

The prompt

  • PROMPT_VERSION = "v1" (in service.py and agent.py); the system prompt is
    src/agent_experiments/prompts/v1.md, used only for the design turn.
  • Every turn's trace records prompt_version, and GET /healthz reports it.

HTTP API (port PORT, default 8080)

Route What
GET /healthz {ok, agent: "agent-experiments", version, model, prompt_version}. version is the installed package version (new in this release)
POST /v1/turn One turn. Requires Authorization: Bearer <AGENT_SERVICE_TOKEN> when that variable is set (otherwise 401). Body: kind, message, lake_token (the person's delegated credential), lake_url (optional when LAKE_URL is set), optional customer_id, conversation_id, context. Answers 200 with {ok: true, conversation_id, trace_id, reply, result}, 422 with {ok: false, error}, 400 when the body is not JSON

Any other path is 404. Conversation history is kept per conversation_id in the process's memory.

The agent bus: how Lake reaches it

  • The operator runs it as a library agent of every Lake: one Deployment and one Service, port 8080,
    probed on /healthz, and gives Lake its address in LAKE_AGENT_ENDPOINTS
    (agent-experiments=<url>). The operator sets LAKE_URL, AGENT_SERVICE_TOKEN,
    AGENT_BUS_TOKEN and AGENT_BUS_AGENT=agent-experiments on it.
  • Lake dispatches each turn as a task on the topic agent-tasks.agent-experiments of Lake's own
    Zetta. The topic carries task ids only, never content (lake docs/api/agent-bus.md).
  • When LAKE_URL, AGENT_BUS_TOKEN and AGENT_BUS_AGENT are set, serve starts one bus thread
    (lake_bus.py). It long-polls receive_agent_tasks {agent, from_sequence, wait_ms: 10000}
    through Lake, starting at sequence 1 on every start and continuing from next_sequence. For each
    task it calls claim_agent_task {task_id, agent}; a conflict or not_found answer means
    another replica or attempt has it and the task is skipped. A claimed task's body (with its task
    credential as lake_token) runs on a pool of 4 worker threads through the same run_turn as
    POST /v1/turn. The worker publishes, through publish_agent_event, one progress event
    ("Handling the experiment turn"), then a result event with the whole answer or a failed event
    with the error. After a receive failure it waits 2 seconds and polls again.
  • The bus credential is used only for those three bus capabilities; the person's credential stays
    inside the claimed task body. Without the three variables the bus is off and only
    POST /v1/turn serves turns.

The model

Every model call goes through Lake's model gateway (lake#248, model.py): an OpenAI client at
<LAKE_URL>/openai/v1 (falling back to the turn's lake_url) with the turn's own lake_token as
the API key, so Lake counts the tokens in the turn's workspace and enforces the account's limit. The
service holds no provider key. Only the design turn calls the model.

Command-line entry points ([project.scripts])

  • agent-experiments-serve: the service (the image's entry point).
  • experiments-agent-evals: the three acceptance evals; writes dated JSON and Markdown reports under
    reports/ (git-ignored).

Environment variables

Name Meaning
PORT HTTP port; default 8080
AGENT_SERVICE_TOKEN Bearer token required on POST /v1/turn when set
LAKE_URL Lake's internal root URL: the bus calls and the model gateway
AGENT_BUS_TOKEN This agent's bus credential
AGENT_BUS_AGENT The agent key on the bus; default agent-experiments
EXPERIMENTS_AGENT_MODEL Model name; default gpt-5.5
TRACE_DIR Where turn traces are written; default /tmp/agent-experiments-traces
OPENAI_API_KEY (evals only) Read by the OpenAI client in experiments-agent-evals; the service never reads it

The image

oramasearch/agent-experiments on Docker Hub, private. The Dockerfile builds on
python:3.12-slim with uv 0.11.7 (uv sync --frozen --no-dev --no-editable), copies only the
virtual environment into a second python:3.12-slim stage, runs as UID 10001 (agent), exposes
8080 and starts agent-experiments-serve. It needs no named build contexts.

Where it runs

Every Lake runs it, deployed by the Orama operator as a library agent:

  • The operator's default library image (internal/controller/lake_defaults.go on dev) is
    oramasearch/agent-experiments:62a4d64. The LakeSets lake-nonprod (members dev, staging
    and testing on orama-nonprod-cluster-01) and lake-prod (members shared on
    orama-production-cluster-02 and shared-us on orama-production-cluster-03) take that default.
  • Lake prod on the managing cluster orama-production-cluster-01 (namespace lake, the
    development environment) pins the same oramasearch/agent-experiments:62a4d64 in its
    spec.agents.

v0.1.0 is the same tree as that running commit, 62a4d64, plus these notes, the release test and
the version field on /healthz. Moving the operator default to :v0.1.0 is a separate operator
change.

What it does not include

  • Lake's own experiment names (start_experiment, experiment_id): the agent speaks the engine's
    study names through Lake's workspace adapter, and Lake maps context.experiment_id to
    context.study.
  • Calls to stop_study, list_studies or get_run_metrics from its own code (allowed, unused).
  • Any model call outside the design turn; any model reached other than through Lake (the evals
    excepted).
  • Persistent convers...
Read more