Repository navigation
Released 2026-11-16. A minor release. Since v0.2.0: model-written profiles are checked before anything is written (safe names, bounded size), the licence, the security policy and base images pinned by digest. Image: docker.io/oramasearch/agent-eval-generator:v0.3.0, not published (see Known issues).
This release is the tree of dev with these notes and the version.
What it includes
Every pull request merged into dev since the v0.2.0 tag:
| PR | What it does |
|---|---|
| #27 | MIT LICENSE and COPYRIGHT (OramaSearch, Inc.), an SPDX-License-Identifier: MIT line in every source file (tools/spdx-headers.py), THIRD_PARTY_LICENSES, CONTRIBUTING.md, CLA.md, CODE_OF_CONDUCT.md, and the SBOM (sbom.spdx.json, sbom.cdx.json, tools/sbom.py) |
| #24 | Security: a model-written profile is checked before anything is written or generated (safe names, bounded size) |
| #25 | The Dockerfile names its base images by digest as well as by tag (python:3.12-slim, ghcr.io/astral-sh/uv:0.11.7), so a later build of the same source gets the same base |
| #23 | SECURITY.md: how to report a vulnerability |
| #19 | Records the v0.2.0 image digest in the v0.2.0 notes (documentation only) |
| #18 | The v0.2.0 release promotion (dev → main), merged back into dev |
A model-written profile is checked before anything is written — #24
- The model's
profile_idand table names were used as file paths, so a turn could write and delete outside its temporary folder in a pod every workspace shares (#21).synthesize()had no limit on tables or columns, so one turn could use up the pod's memory (#22). - New
limits.check_profile()runs right after parsing, inservice.run_turn, in the batch command line and again insynthesize(). Names that become paths must match^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$; a profile has at most 12 tables, 64 columns per table, 128-character column names, no table declared twice, and at most 5,000,000 generated cells. - A refused profile is answered
invalid profile: <reason>and its trace graded failed. records.package()andTable.write_jsonl()resolve every path inside their directory and refuse one outside it; an oldrecords/is removed only when it is a real directory inside the output folder, never through a symbolic link.
The licence — #27
Everything in this repository that Orama wrote is under the MIT licence. Contributors sign CLA.md once. THIRD_PARTY_LICENSES lists every runtime dependency with its licence text; none is incompatible with MIT distribution.
Before it rolls: the settings it needs
Nothing new. The variables are unchanged from v0.2.0.
What it does not include
Deferred, not in this release:
- #26: claiming bus tasks by reference and naming the task on every call; it needs the matching Lake change.
- #28: the public-tree cleanup, which also makes the Hugging Face upload target a setting (
EVAL_PROFILES_DATASET) and so changes the batch command line's behaviour.
Otherwise unchanged from v0.2.0.
Known issues
Security-relevant first. All open at this tag.
- #29 (release blocker for publishing): the image is built on
python:3.12-slim, which bundles GPL and LGPL userland, so the image is not published. It is built and kept in the project's non-public Docker Hub repository until it moves to a scratch base with static binaries. - The public-tree cleanup (#28) is not in this release: the batch command line still uploads to the Hugging Face dataset named in the code, and some files still carry internal references. This tree is not the one to publish.
- #6: the task's workspace is not passed explicitly on data-plane calls. Lake scopes the task credential to the task's workspace.
Where it has run
The v0.3.0 tree had not run on a development environment when it was released. Lakes run the operator's default image of this agent; the changes since v0.2.0 are proven by the test suite only. Moving a Lake to :v0.3.0 is a separate operator change.
The image
Built from the tag v0.3.0 (28eded2) with BuildKit (linux/amd64, the repository's Dockerfile), then tagged :v0.3.0 on that digest without rebuilding:
docker.io/oramasearch/agent-eval-generator:28eded2@sha256:74c5b48f80a6eec2274d3f092e1bd5ac64523dc79b05af111ab1992095c3f440, also tagged :v0.3.0 (same digest).
It is pushed only to the project's non-public Docker Hub repository, which answers 404 anonymously. Pinning it on a Lake is a separate change.
Verification
| Proof | Result |
|---|---|
Test suite (uv run --frozen pytest -q -ra, macOS arm64, Python 3.12, release branch) |
66 passed, 0 failed, 0 skipped; 66 tests. Includes tests/test_release.py (the declared version has these notes, indexed) |
SPDX headers (python3 tools/spdx-headers.py --check) |
Every source file carries the header |
SBOM (python3 tools/sbom.py --version v0.3.0, then --check) |
Regenerated on the release commit; --check passes |
| Batch generation, and a live turn through Lake's model gateway | Not run (needs a live model and a Lake) |
| Linux image build (BuildKit, linux/amd64, from the tag) | Built from v0.3.0: oramasearch/agent-eval-generator:28eded2, also :v0.3.0, sha256:74c5b48f80a6eec2274d3f092e1bd5ac64523dc79b05af111ab1992095c3f440 |