One Python package (omds) that does two things over the same cores:
-
A data-science guardrail toolkit for coding agents. A set of
omds-*CLIs plus portable Agent Skills that let any coding agent check Python for train/test leakage and metric misuse, trace data lineage through a code graph, and keep a durable experiment ledger and episodic memory. Optionally reachable as an MCP server or as a Claude Code plugin. -
E-GDS — an offline evolutionary optimizer for ML pipelines. The
omdsCLI (init/evolve/promote/inspect/models) evolves a pipeline workspace's own code with GEPA (Genetic-Pareto reflective evolution), targets mutations using a code graph, and admits a variant only through five blocking gates ending in a real, sandboxed training run on holdout data.
These are not two codebases. Both halves stand on the same ontological
grounding layer (omds.ontology — probability-simplex bounds, proper scoring
rules, leakage rules, sparsemax caveats) and the same code graph layer
(omds.codegraph — AST dependencies + data lineage). The axiom registry that
blocks a GEPA promotion is literally the registry omds-guardrails check runs
against the file your agent just edited: both call
omds.ontology.static_check.run with omds.ontology.axioms.default_registry.
omds.ontology omds.codegraph
(axiom registry, static AST rules, (AST deps, def-use chains,
runtime assertion injection) data lineage, git blame)
│ │
┌─────────────┴──────────────┬───────────────────┴──────────────┐
│ │ │
omds-* CLIs + skills/ omds-mcp (9 tools) E-GDS: omds evolve
Claude Code plugin → GEPA → 5 gates
PRIMARY surface OPTIONAL adapters → internal PR
(agent-neutral) (same cores, no new logic) (offline optimizer)
Everything is model-agnostic and SLM-first: the LLM layer speaks only
OpenAI-compatible /v1/chat/completions, so it runs against local small models
(Qwen2.5-Coder and friends via Ollama / llama.cpp / vLLM / LM Studio) — the
design floor every prompt must work within — and swapping in a frontier API
model is a config change. The guardrail toolkit needs no model at all.
Both halves are shipped and green. uv run pytest reports 705 passed,
6 skipped (all six skips are opt-in tiers — see Testing — plus 2
pre-existing sklearn UndefinedMetricWarnings). The nine MCP tools, the
omds-* CLIs, the seven skills, the plugin bundle, the GEPA evolution engine,
the omds-slm local-SLM harness, and the benchmarks/tabular_multiclass/
benchmark (which passes all five promotion gates as its own seed) are all
covered. The one thing no test in the default suite exercises is a live
model: that path is smoke-tested behind an opt-in gate (see Local SLM
support (≤7B) under Shared).
One capability was deliberately removed, not deferred. v1 shipped an
omds run command driving a multi-agent runtime loop (AutoGen GraphFlow
step agents, task decomposition, a repair loop). That whole stack was deleted
in the v2 harness rework — there is no omds run today, and the
autogen-* dependencies are gone. What survives of omds.runtime is the
model-agnostic, network-free part the evolution gates reuse: task loading, the
skills catalog, and validate_workspace. Traces are now written by the
evolution evaluator (one per candidate evaluation), not by an agent loop, and
omds inspect traces reads those. Rationale:
docs/adr/2026-07-18-mcp-pivot.md.
- Architecture decision record:
docs/adr/2026-07-18-mcp-pivot.md(read the Decision and both amendments — they reshape it) - v1 design spec:
docs/specs/2026-07-11-egds-design.md - v2 harness design:
docs/specs/2026-07-12-omds-harness-design.md
New here? Prerequisites → Install → Confirm it works → Wire it into your agent. That is the whole setup, and it needs no model and no network.
| Part 1 — agent toolkit | Part 2 — E-GDS optimizer | Shared |
|---|---|---|
| Getting started | Quickstart | Local SLM support (≤7B) |
| Skills | How it works | SLM-first design rules |
| CLIs | The promotion gates | Repository layout |
| MCP server | Config reference | Testing |
| In-kernel leakage guard | Testing without a live model | Verifying a real install |
| Troubleshooting | Non-goals · License · History |
- Python 3.12 or newer (
requires-python = ">=3.12"). Nothing older works. 3.12, 3.13 and 3.14 are each verified by a real install-and-run, not just declared. - git —
omds-codegraph blameshells out to it, and E-GDS clones workspaces withgit worktree. The rest of the toolkit runs fine without a repo. - uv — the recommended installer (see Step 1),
and required for a checkout or for Part 2. Plain
pipworks for the toolkit if you would rather not add another tool. - No model, no API key, no network. Everything in Part 1 is static analysis
and local files. A model is needed only for Part 2 (E-GDS) and
omds-slm.
OMDS is installed from this repository, not from PyPI — pip install omds
will fail with "No matching distribution found". Pick the route that matches
what you want to do; they are not exclusive, and the third is a superset.
(a) Just the agent toolkit — uv tool install. OMDS is mostly ten command
line tools, which is exactly what uv tool is for: one command, all ten on your
PATH, in an isolated environment, with no venv to create or activate.
uv tool install --python 3.12 \
"git+https://github.com/spkc83/omds.git#subdirectory=python/omds"Pass
--python 3.12. It is not optional padding. On uv 0.5.x, omitting it can build the tool environment on Python 3.11 despite the package requiring>=3.12, and then install 3.12-built binaries into it —omds-guardrailsfails with a numpy C-extensionImportErrorthat looks like a broken numpy rather than a wrong interpreter. Newer uv handles this; pinning costs nothing and works on both.
(b) Into your own project environment — when you want the in-kernel half.
Route (a) is deliberately isolated, so omds is not importable from your own
Python: from omds import kernel_guard raises ModuleNotFoundError even though
the CLIs work. The in-kernel leakage
guard and the omds.dstools helpers the
visualization skill uses run inside your kernel, so they need the package
installed alongside your own code:
uv pip install "git+https://github.com/spkc83/omds.git#subdirectory=python/omds"
# plain pip works identically:
pip install "git+https://github.com/spkc83/omds.git#subdirectory=python/omds"(c) From a checkout — required for Part 2. omds init copies benchmark
files that are not shipped inside the wheel, so E-GDS needs the repo itself:
git clone https://github.com/spkc83/omds.git
cd omds
uv sync # installs `omds` editable + the test toolchain
# or, without the uv workspace: uv pip install -e python/omdsEvery route gives you the ten console scripts (omds, omds-guardrails,
omds-codegraph, omds-dstools, omds-memory-store, omds-sessionpad,
omds-skill, omds-mcp, omds-install, omds-slm) and the seven skill files
bundled inside the package. All three wiring routes below assume this step —
in particular a Claude Code plugin cannot install a Python package, so it is
never optional.
Worth doing before wiring anything into an agent — it separates "the package is broken" from "my agent is not calling it". Create a file with a textbook leak in it:
mkdir /tmp/omds-demo && cd /tmp/omds-demo
cat > pipeline.py <<'EOF'
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
def prep(X, y):
scaler = StandardScaler()
scaler.fit(X) # fit on everything...
X_train, X_test, y_train, y_test = train_test_split(X, y) # ...then split
return X_train, X_test, y_train, y_test
EOF
omds-guardrails check --files pipeline.pyThat prints one violation — the scaler saw the test rows before the split:
{"violations":[{"axiom":"ax:fit_on_train_only","severity":"certain","file":"pipeline.py","line":7,
"message":"pipeline.py:7: scaler/encoder fit on full data before split - fit() argument 'X' is also the full dataset passed to train_test_split in this function",
"needs_runtime_proof":false}],"summary":{"certain":1,"warn":0}}If you see {"violations":[],"summary":{"certain":0,"warn":0}}, the package
works but the pattern was not matched — note that the fit-then-split sequence has
to sit inside a function to be detected. A command not found means the
install did not put the scripts on your PATH; see
Troubleshooting.
Two more, to see the rest of the surface:
omds-codegraph lineage --symbol prep --root .
# {"lineage":[{"from":"pipeline.py::<module>","to":"pipeline.py::prep","kind":"defines"}]}
omds-skill lint skills/leakage-audit # only from a checkout
# {"ok":true,"checks":[{"check":"format","passed":true,...}, ...]}All three deliver the same thing.
(a) omds-install — the recommended route; works from the installed package
alone, because the skill files ship inside the wheel:
cd /path/to/your/project
omds-install claude # --project defaults to the current directoryMerged omds MCP server into .mcp.json
Copied skills into .claude/skills
It writes exactly two things, and nothing else:
.claude/skills/{eda,evaluation,feature-engineering,leakage-audit,
model-training,reporting,visualization}/
For any other agent, name the two paths yourself:
omds-install here --config <mcp-config.json> --skills-dir <that agent's skills dir>claude is literally here with .mcp.json and .claude/skills pre-filled;
there is no per-agent behavior, only per-agent paths. The merge into an existing
MCP config is non-destructive and idempotent — rerunning it is safe, and it will
not disturb MCP servers you already have configured.
(b) By hand — nothing above is magic. The skills are just files and the MCP entry is one line, so from a checkout:
cp -r skills/* .claude/skills/ # the primary surface
claude mcp add --scope project omds -- omds-mcp # the optional MCP adapter(c) The Claude Code plugin — this repo is itself a plugin
(.claude-plugin/plugin.json plus a single-plugin marketplace), packaging the
same skills/, the same omds-mcp server, and a guardrail hook:
/plugin marketplace add spkc83/omds
/plugin install omds@omdsTo try a local checkout without going through the marketplace, launch Claude
Code with claude --plugin-dir . from the repo root.
The plugin route additionally installs a PostToolUse hook that runs
omds-guardrails check on every Python file Claude writes or edits, blocking
the turn on an axiom-certain leakage violation. Skills are namespaced under
it as omds:<skill-name> (e.g. omds:leakage-audit). The hook is stdlib-only
and fails open in every abnormal state — including omds-guardrails not
being on PATH — so an uninstalled package makes it silently do nothing rather
than break your editing.
In a Claude Code session, in the project you just wired up:
| Check | Expect |
|---|---|
/mcp |
omds listed and connected. (Route (a)/(b) prompt for approval on first use; the plugin route does not.) |
| ask it to list skills | leakage-audit and six others — namespaced omds:leakage-audit under the plugin route |
"check pipeline.py for data leakage" |
it runs guardrails_check / omds-guardrails check and reports ax:fit_on_train_only |
If /mcp shows omds as failed, run omds-mcp in your shell: it should start
and wait silently for stdio input (Ctrl-C to exit). A traceback instead means
the agent is reporting a real install problem — see
Troubleshooting.
For the whole install checked mechanically in one command, see Verifying a real install.
skills/ holds seven agent-neutral SKILL.md files — eda,
feature-engineering, model-training, evaluation, visualization,
reporting, leakage-audit. Nothing in them is tied to a particular agent;
any tool that reads the SKILL.md format (Claude Code, Codex, Cursor, Gemini
CLI, Copilot, …) can use them. Each teaches one workflow in terms of the real
omds-* CLIs, so installing the package is enough to make every step actionable.
See skills/README.md for the format and for adding one.
omds-skill lint is the contract, executable. Writing a skill by hand is
easy to get subtly wrong, and the expensive mistake is a skill that tells an
agent to run a command that does not exist:
omds-skill lint skills/leakage-audit # a directory or its SKILL.md
# {"ok":true,"checks":[{"check":"format","passed":true,...}, ...]}Six checks run — format, description_shape, no_coupling_tokens,
surface_references, size, distinctness — and all of them are reported
even after one fails, because an author fixing a skill wants the whole list in
one pass. The one that earns its keep is surface_references: it resolves every
omds-* command, MCP tool and omds.* module path a skill mentions against the
live registries (the installed console-script entry points, the Typer apps
behind them, and the MCP tool manager) rather than a hardcoded list — a
hardcoded list being exactly the drift it exists to catch. It found real drift
the first time it ran: eda's own description cited omds_dstools.df_summary,
a module name that has not existed since the monopackage merge.
test_skills_portable.py calls the same lint_skill core, so the command you
run by hand and the assertion the suite makes are the same code, not two
descriptions of it.
Distinctness is checked against the shipped set because skills/README.md's
warning is literal: a vague or duplicated description degrades routing for every
other skill, not just the new one.
Every command exits 0 always: the decision lives in the JSON payload, never in the exit status. Stdout is nothing but JSON; logs go to stderr.
check analyzes the files you name, but builds its code graph over their common
parent directory — so it prunes virtualenvs (by name and by pyvenv.cfg),
caches and node_modules rather than walking them, and a file anywhere in the
tree that cannot be parsed is skipped rather than fatal. A file you asked
about that could not be read is different: it is named in an unreadable
key, present only when non-empty, because a guardrail must never answer "clean"
for a file it never managed to read.
# static leakage / scoring / shape checks over on-disk files…
omds-guardrails check --files pipeline.py
# {"violations":[{"axiom":"ax:fit_on_train_only","severity":"warn",...}],"summary":{"certain":0,"warn":1}}
# …or over content an agent is ABOUT to write (a JSON [{"path","content"},…] manifest)
omds-guardrails check --proposed proposed.json
# code graph: edges incident to a symbol, git blame + enclosing symbol, bounded file summary
omds-codegraph lineage --symbol prep --root .
omds-codegraph blame --file pipeline.py --line 42
omds-codegraph context --files pipeline.py --detail summary --max-items 50
# experiment ledger and episodic memory (both honor $OMDS_STATE_DIR)
omds-dstools ledger show --session demo
omds-memory-store add --session demo --kind note --text "baseline logreg log_loss 0.41"
omds-memory-store recall --session demo --query baseline --k 3
# the session working pad — plan / decisions / problems / log, as one markdown file
omds-sessionpad append --session demo --section plan --text "try target encoding next"
omds-sessionpad show --session demo --max-chars 2000omds-mcp (official mcp SDK, FastMCP, stdio) exposes exactly nine tools.
Each one calls the same core function its CLI calls — guardrails_check and
omds-guardrails check both land in run_static_check — so the tool result is
byte-identical to the CLI's --json stdout and there is no second copy of any
logic. That DRY property is why keeping three surfaces costs almost nothing.
The server is not required. It exists for hosts that sandbox shell access
or prefer typed tool calls. The mapping (identical to skills/README.md):
| CLI (primary) | Equivalent MCP tool |
|---|---|
omds-guardrails check |
guardrails_check |
omds-codegraph lineage |
codegraph_lineage |
omds-codegraph blame |
codegraph_blame |
omds-codegraph context |
codegraph_context |
omds-dstools ledger show |
dstools_ledger_show |
omds-memory-store add |
memory_add |
omds-memory-store recall |
memory_recall |
omds-sessionpad append |
sessionpad_append |
omds-sessionpad show |
sessionpad_show |
Domain outcomes are normal results, never protocol errors: a leakage violation,
an unresolved symbol, a file outside a git repo, an empty ledger, or a session
with no memory DB all return a payload. Only genuine faults (an unreadable
path, unparseable input) surface as tool errors. codegraph_context takes
detail ("summary" default / "full") and max_items so a small-window
model is never blown up by an unbounded dump — counts still reports the true
totals and truncated says whether anything was elided.
The visualization skill additionally uses the in-kernel omds.dstools
helpers (df_summary, capture_plot, ledger), which have no CLI or MCP
equivalent because they run inside your own Jupyter/IPython kernel.
omds.dstools.install() publishes them onto builtins for later cells.
The static checks above read code. The other half is a runtime guard that
fires only on data that actually reaches fit():
from omds import kernel_guard
kernel_guard.install()
X_test = kernel_guard.tag_test(X_test)
StandardScaler().fit(X_test) # raises LeakageViolation
StandardScaler().fit(X_train) # untagged data is forwarded verbatimIt is a tagged-object check, not a taint-tracking system: install() wraps
every sklearn estimator's fit, is idempotent, and fails open to a no-op if
sklearn/numpy cannot be imported. The violation message embeds a delimited JSON
payload (parse_violation_payload recovers it) so an out-of-process reader —
an agent reading a cell's error output, a log scraper — gets structure out of
what a kernel serializes as plain traceback text.
To arm it for every kernel in an environment without a bootstrap cell:
omds-guardrails install-kernel-hook # writes omds_autostart.pth into site-packages
omds-guardrails kernel-hook-status
omds-guardrails uninstall-kernel-hookRead the warning before running that. A .pth executes at interpreter
startup for every Python process in that environment — scripts, CI jobs,
python -c, all of it — not just kernels. It is written to make that nearly
free (it imports only sys at startup, and the guard itself installs only
inside an ipykernel process), but it is still an environment-wide change.
Prefer a venv you own over a shared or system interpreter. Installing the
package deliberately does not arm it: shipping the .pth in the wheel would
make that decision on your behalf, so it is a separate, explicit, reversible
opt-in.
pip install omds → "No matching distribution found"
Expected — OMDS is installed from the repository, not PyPI. Use one of the three
routes in Step 1.
ModuleNotFoundError: No module named 'omds' in your own Python or notebook,
even though the omds-* commands work
You installed with uv tool install, which is isolated by design — it puts the
CLIs on PATH without putting the package on your sys.path. That is correct
for the CLIs and wrong for the in-kernel guard and omds.dstools. Install it
into the environment your kernel runs, per
Step 1(b). Having
it in both places is fine.
omds-guardrails fails with a numpy C-extension ImportError
Almost always a tool environment built on the wrong interpreter — the message
blames numpy, but the cause is a Python 3.11 environment holding 3.12-built
binaries. The traceback names the offending interpreter ("The Python version
is: Python 3.11 from …"). Rebuild the environment pinned:
uv tool install --force --python 3.12 \
"git+https://github.com/spkc83/omds.git#subdirectory=python/omds"omds-guardrails: command not found
The package installed into an environment whose bin/ is not on your PATH —
almost always a virtualenv that is not activated. Check where it went:
python -c "import omds, pathlib; print(pathlib.Path(omds.__file__).parent)"Then either activate that environment, or call the scripts by full path
(/path/to/venv/bin/omds-guardrails). Configure your agent with the absolute
path to omds-mcp if it does not inherit your shell's PATH — a common cause
of an MCP server that works in a terminal but not in the agent.
The install failed on the Python version
omds requires 3.12+. python -V to check; install into a newer
interpreter (uv venv --python 3.12).
check reports no violations on obviously leaky code
The static packs match specific shapes, not intent. ax:fit_on_train_only
needs the fit-then-split sequence inside a function — at module level it is
not matched. This is a deliberate precision-over-recall trade: a guardrail that
cries wolf gets turned off. The runtime
in-kernel guard is the complement, firing on
data that actually reaches fit() regardless of code shape.
The output has an unreadable key
A file you asked about could not be read or parsed — a non-UTF-8 encoding, a
syntax error, a permissions problem; reason says which. Those files were
not analyzed, so treat them as unchecked rather than clean. Files elsewhere
in the directory that cannot be parsed are simply skipped and never appear here.
MCP server shows as failed in the agent
Run omds-mcp directly in a shell. It should start and sit silently waiting for
stdio input (Ctrl-C to quit) — that is a healthy server, since it speaks a
protocol, not a CLI. A traceback or command not found is the real problem;
see the PATH note above.
The plugin's guardrail hook never fires
By design it fails open in every abnormal state, including
omds-guardrails not being on PATH — a missing package makes it do nothing
rather than break your editing, which also means a broken install looks exactly
like a clean file. Confirm omds-guardrails check --files <a leaky file> works
in your shell first. Note the hook only inspects .py files that Claude
writes or edits.
/plugin marketplace add spkc83/omds cannot find the plugin
The marketplace manifest is read from the repository's default branch. Use
claude --plugin-dir . from a checkout to test the plugin without publishing.
omds: command not found / omds init cannot find benchmark files
Part 2 needs the repository, not just the installed package — omds init
copies benchmark files that are not shipped inside the wheel. Clone and
uv sync.
Still stuck? scripts/verify_install.py checks
every mechanical part of an install in one command and names what it cannot
check — see Verifying a real install.
# 1. install (from a checkout — `omds init` copies benchmark files that are not
# shipped in the wheel, so this half needs the repo, not just the package)
uv sync
# 2. a local model behind an OpenAI-compatible endpoint (Ollama shown; llama.cpp
# server / vLLM / LM Studio all work too — anything speaking /v1/chat/completions)
ollama pull qwen2.5-coder:14b
# 3. scaffold a task — copies the tabular_multiclass benchmark's seed pipeline plus
# freshly-generated synthetic data into ./mytask/workspace, and gives that
# workspace its own git history on `main` (evolve/promote clone it via
# `git worktree` and need a repo to clone from)
uv run omds init mytask
# 4. evolve the pipeline offline under a GEPA evaluation budget
uv run omds evolve mytask/task.yaml --max-evals 12
# 5. inspect the Pareto frontier, then promote a gate-passing variant
uv run omds inspect frontier --task mytask/task.yaml
uv run omds promote --dry-run --task mytask/task.yaml
uv run omds promote --approve <variant_id> --task mytask/task.yamlomds inspect also takes traces, violations, and graph;
omds models probe -c mytask/omds.yaml records each configured model's
capabilities into $OMDS_HOME/models.json (default ~/.omds/models.json).
omds.yaml (written by init alongside task.yaml) points at
http://localhost:11434/v1 by default — edit llm.base_url /
llm.roles.*.model to point at any other OpenAI-compatible endpoint or model.
No live endpoint handy? evolve and models probe also work against the
network-free fake:// scheme (see
Testing without a live model).
+---------------------------------------------------------------------------+
| task.yaml + workspace/ (goal, data roles, metrics, bounds, eval budget) |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| ONTOLOGICAL GROUNDING LAYER (omds.ontology) |
| Axiom registry · builtin packs (leakage, scoring, evaluation, shapes, |
| simplex_transforms) · static AST rules · runtime assertion injection |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| CODE GRAPH LAYER (omds.codegraph) |
| AST dependency extraction · def-use chains · data lineage · blame |
+---------------------------------------------------------------------------+
|
+---------------------------------------------------------------------------+
| OFFLINE GEPA EVOLUTION ENGINE (omds.evolution) |
| gepa optimize_anything (single-task search, dataset=None) · |
| code-graph-targeted, ontology-briefed reflective mutation proposer · |
| Pareto objectives/frontier · the five gates · internal-PR promotion |
+---------------------------------------------------------------------------+
| |
each candidate evaluated in an isolated clone traces + frontier.json
(omds.sandbox: git worktree + uv venv, rlimits, (omds.telemetry, read by
network=False) `omds inspect`)
Two distinct git repositories are involved:
- The harness repo (this one) — the framework code. Not mutated by GEPA.
- The pipeline workspace — a per-task git repo under
<task_dir>/workspace/holding the evolving pipeline code, prompts, and configs. GEPA variants aregit worktreeclones of it; promotion is a merge into itsmainvia an internal PR (branch → rebase → gate rerun → merge). This confines genetic mutation to a blast radius the gates fully cover.
gepa==0.1.1's optimize_anything API in single-task search mode
(dataset=None) is the evolution core — it matches E-GDS's
one-candidate/one-evaluation model exactly, with a custom_candidate_proposer
doing code-graph-targeted, ontology-briefed reflective mutation instead of
gepa's default LLM-based reflection. The reflector is the only live LLM call in
the loop; everything else (clones, gates, training runs) is real work with no
model in it.
ontology_static— AST-pattern axiom checks over the workspace; zeroerror-severity violations (warnings need a runtime proof — see gate 5).pytest— the pipeline workspace's own contract test suite.pyright— type check (pyright missing from the clone's venv is a documented SKIP-pass, not a hard failure).ruff— correctness lint rules only (F,E9,B).runtime_validation— a real instrumented pipeline run on holdout data: zero unresolved runtime-assertion warnings, metrics withintask.yamlbounds, no regression beyondevolution.regression_tolerance.
A gate-passing, frontier-selected variant becomes an internal PR:
omds promote --dry-run renders the report (metrics delta, axioms touched,
blame targets, trace ids) without merging; omds promote --approve <id>
rebases onto main, reruns all five gates post-rebase, then merges.
promotion.require_human: true (the config default) always stops at the
report; --dry-run forces the report-only path regardless of that setting.
llm:
base_url: http://localhost:11434/v1 # any OpenAI-compatible endpoint; or fake://... (see below)
api_key: local
roles:
planner: {model: qwen2.5-coder:14b}
coder: {model: qwen2.5-coder:14b}
reflector: {model: qwen2.5-coder:14b, temperature: 0.7} # drives GEPA's mutation proposals
auditor: {model: qwen2.5-coder:7b} # cheap roles can run smaller models
evolution:
max_evals: 12 # gepa's max_metric_calls (budget-driven, not generation-driven)
parallel: 2 # gepa's evaluator thread pool; each call works in its own env clone
max_repair_iterations: 3
regression_tolerance: 0.0
promotion_require_human: true # false enables full promotion autonomy
data_dir: runs # traces/frontier reports, relative to this file unless absolute- Roles → models. Each role binds to a model independently, so cheap roles
can run a smaller/faster model. With the agent runtime removed,
reflectoris the only role the shipped loop actually calls;planner/coder/auditorare still accepted and still validated, but nothing dispatches to them today. EGDS_-prefixed env vars override the file (EGDS_LLM__BASE_URL, …). The prefix predates theegds→omdsrename and was left alone deliberately, so existing configs keep working.llm.base_urlalso acceptsfake://...— see below.task.yaml(separate fromomds.yaml) declares the task itself: goal, task type, dataset paths/roles, metrics, andbounds/eval_budget.agents:andorchestration:no longer exist. They were shaped for the AutoGen orchestration path deleted in the v2 rework and outlived it by a release — validating cleanly while nothing read them, which reads as a working feature. Unknown keys are rejected, so anomds.yamlstill carrying them now fails with a validation error naming them; delete the two blocks.
omds.yaml's llm.base_url accepts the fake: scheme (is_fake_base_url in
omds.cli_support) as a first-class alternative to a real http(s):// URL —
e.g. base_url: fake://local. Under it, evolve and models probe substitute
deterministic, network-free stand-ins for every LLM call (the reflector's
mutation proposals, the capability probe) while every other part of the loop
stays real: actual git worktree env clones, real uv sync, real
ontology/pytest/pyright/ruff/runtime gates, and a real sandboxed training run.
This is what the default (non-opt-in) CLI test suite runs against, and what the
fake-LLM tier of tests/e2e/test_full_loop.py drives end-to-end.
Write it as
fake://local, not the barefake:— a plain YAML scalar ending in a colon is invalid YAML and would need quoting to parse at all.
omds-slm (omds.slm) drives a local small model over the same nine tool
cores the CLIs and the MCP server call. It lives here rather than in Part 1
because it is the one place a model is pointed at the toolkit: Part 1's
guardrails, code graph, ledger and memory need no model at all, and Part 2's
GEPA loop uses a model for mutation proposals, not for tool use.
Start with the caveat, because it is the design. This is a supervised, turn-capped harness, not an autonomous agent — the SLM scope guard (C5) is no autonomous missions on a 7B, and the turn cap is what makes that structurally true rather than merely intended:
--max-turnsis a hard ceiling on model calls, not a hint. The loop makes at most that many calls and then stops, whatever the model asks for.stopped_reason: max_turnsis a normal outcome meaning unfinished, your turn — exit code 0. The loop deliberately does not spend an extra call asking the model to wrap up, because a manufactured conclusion at the cap would hide exactly the fact you need to see. Read the transcript and decide.MAX_TURNS_CEILING = 20bounds what you may even ask for. A cap of 200 is an autonomous mission wearing a cap's clothes; the remedy for a task that genuinely needs more steps is another supervised invocation, not a bigger number.
# a local model behind an OpenAI-compatible endpoint (Ollama shown)
ollama pull qwen2.5-coder
# one supervised run: 4 model calls, maximum
uv run omds-slm run \
--model qwen2.5-coder \
--task "Check python/omds/src/omds/pipeline.py for data leakage." \
--max-turns 4
# the summary format, one line per turn, then the outcome and the answer
# (what a given model actually says is its own business):
#
# turn 1: guardrails_check -> ok
# turn 2: answered
# stopped_reason: final_answer (2/4 turns)
#
# <the model's answer>
#
# …and `stopped_reason: max_turns (4/4 turns)` is the other normal ending.
# --json prints the whole LoopResult: every message sent, the tools offered that
# turn, the estimated token count, and every tool result. That is the auditable
# record of the run — read it, don't trust the summary.
uv run omds-slm run --model qwen2.5-coder --task "…" --max-turns 4 --json--base-url (default http://localhost:11434) is Ollama's native root;
the chat client appends /v1 itself. Any OpenAI-compatible endpoint works —
llama.cpp's server, vLLM, LM Studio — though the context-window probe below is
Ollama-specific and degrades to a static guess elsewhere.
What it does that a plain chat loop does not, all of it in service of a window that may be 4–8k rather than 200k:
- It probes the model's REAL served window (
omds.slm.probe) instead of guessing from the model name. Ollama's/api/showpublishes the trained context length in GGUF metadata — and, crucially, the loop clamps that by the Modelfile'snum_ctx, because the window Ollama actually serves is often much smaller than the one the model was trained for. Budgeting against the larger number is how you overflow a context you thought you had. Every failure here (no server, missing metadata) degrades silently to the static table inomds.llm.registry: a capability probe must never be the reason a run cannot start. - It narrows the tool set per turn (
omds.slm.funnel) — hard triggers → BM25 → synonym-expanded BM25 → optional embeddings → reciprocal-rank fusion → top-5 of the nine tools. The narrowing re-runs every turn against the current request, so the offered subset follows the conversation instead of being frozen at turn 1. - It budget-packs the turn (
omds.slm.context) so tool schemas can never crowd out the task. The system prompt and the current request are priority-0 — never truncated, never dropped. Everything else is shed to fitcontext_length - reserve_output, oldest history first, then skills, then the worst-ranked tools last. If the system prompt and the task alone do not fit, the run stops and says so (stopped_reason: context_too_small, exit 1) rather than quietly truncating your question and confidently answering something you did not ask. - It bounds what comes back from a tool, twice — once at the arguments,
using each core's own knobs (
codegraph_context'smax_items,memory_recall'sk), so the core never builds the oversized payload; and again at the rendered result, with a flat character budget, because seven of the nine tools are bounded only there.
The wire protocol is JSON-in-text, not native tool calling ({"tool": …, "arguments": {…}} or {"answer": …}), parsed leniently through code fences and
surrounding prose — local 7B tool-call support is too uneven to depend on. A
tool the model invented, malformed arguments, or a core that raises are all
normal turn outcomes: the error is fed back and the model gets to correct
itself, within the cap.
Maturity, honestly. The harness itself is well covered by unit tests that inject the transport, the capabilities and the tool caller — no test in the default suite touches a socket. The live path is smoke-tested only behind an opt-in gate (
python/omds/tests/slm/test_ollama_smoke.py), which skips unless an Ollama is reachable and the model is pulled; see Testing. If you are the first person to point this at a model that is notqwen2.5-coder, expect to tune the prompt.
Every prompt in the system is built to work on a 7B coder model with an 8–32k window — SLMs are the design floor, never the ceiling:
- OpenAI-compatible transport only. One
httpx-based async client (omds.llm.client) speaking/v1/chat/completions— no provider SDKs. - Structured output via constrained decoding, not hope. Schema-validated
JSON (
omds.llm.structured.generate), with a bounded repair loop that re-prompts with the bad output plus the validation error; persistent failure is aStructuredOutputError, never coerced or silently dropped. - A model capability registry (
omds.llm.registry) records per-model context length, max output tokens, schema-decoding mode, and native-tool-call support, probed once viaomds models probeand persisted. - Context budgeting (
omds.llm.budget.fit) assembles prompts from prioritized sections and truncates the lowest-priority ones first, so no call ever exceeds the model's window. Priority-0 sections are never dropped. - Micro-prompts. No prompt does two jobs. With the agent runtime removed
the shipped surface is small — the reflector's mutation proposal and the
capability probe are the only live LLM calls — but both go through rules 2
and 4 rather than raw
chat(). - Bounded tool output. The MCP adapter's
codegraph_contextcaps per-file symbols (max_items) and drops per-symbol detail by default (detail), so an SLM-hosted client gets a response that fits its window while still seeing true totals.
The v1 "capability-adaptive dispatch" rule (native tool calls vs. a structured
NextActionfallback) is gone, along with theEgdsChatClientthat implemented it. Nothing in the shipped tree dispatches on a model's probedtool_callscapability; the registry still records the flag.
python/omds/ # the `omds` package — a uv workspace member, the installable artifact
src/omds/
ontology/ # axiom registry + packs (leakage, scoring, evaluation, shapes,
# simplex_transforms), static AST checks, runtime assertions
codegraph/ # AST -> networkx graph, def-use chains, data lineage, blame
dstools/ # in-kernel helpers: df_summary, capture_plot, experiment ledger
memory/ # session pad (markdown) + SQLite FTS5/KV episodic store
llm/ # httpx OpenAI-compatible client, structured-output repair loop,
# capability registry, context budgeter
evolution/ # gepa evaluator, mutation proposer, Pareto objectives,
# the five gates, internal-PR promotion
sandbox/ # git-worktree + uv-venv env clones, rlimit/network-isolated exec
telemetry/ # TraceRecord/RunArtifact models, metric collectors
runtime/ # task spec loading, skills catalog, offline validation gate
slm/ # `omds-slm` — the supervised, turn-capped local-SLM harness:
# Ollama capability probe, tool funnel, budget-aware
# turn assembler, turn-capped loop (see Shared, above)
pipeline.py # @stage / Pipeline contract — the evolvable-code hook
cli.py # `omds` init / evolve / promote / inspect / models
guardrails_cli.py # `omds-guardrails` check + the kernel-hook install/uninstall/status
codegraph_cli.py # `omds-codegraph` lineage / blame / context
dstools_cli.py # `omds-dstools` ledger show
memory_store_cli.py # `omds-memory-store` add / recall
sessionpad_cli.py # `omds-sessionpad` append / show (the markdown session pad)
mcp_server.py, mcp_tools_*.py # `omds-mcp` — the 9-tool MCP adapter
install.py # `omds-install` the optional convenience installer
kernel_guard.py # runtime fit()-on-tagged-test-data guard
kernel_hook.py # the opt-in `.pth` that arms it environment-wide
tests/mcp/ # MCP tools, installer, skill portability, plugin-bundle invariants
tests/slm/ # probe/funnel/assembler/loop, all injected — plus the opt-in
# live-Ollama smoke (skipped unless a model is actually there)
skills/ # the 7 portable, agent-neutral Agent Skills
hooks/ # PostToolUse guardrail hook (stdlib-only, fail-open) + hooks.json
.claude-plugin/ # plugin.json + marketplace.json — the Claude Code plugin manifest
.mcp.json # the project-scoped MCP entry, for this repo itself
benchmarks/tabular_multiclass/ # end-to-end benchmark: make_data.py, task.yaml, workspace_seed/
tests/ # the E-GDS suite: unit + integration, plus opt-in e2e/ and benchmarks/
docs/ # specs, plans, ADRs, research
The tree is Python only — there is no TypeScript, no bun, no node. The
pi-era TS algorithms were kept in an archive/ for one release while the skill
funnel was ported to Python; that port is done and better covered than the
original, so the archive was removed rather than left as a second, unbuilt
definition of the same thing. It is still readable in git history:
git show f3d221f:archive/ts-extension/skill-funnel.ts # the funnel, now omds/slm/funnel.py
git show f3d221f:archive/ts-extension/runner.ts # the workflow runner, never ported
git show f3d221f:archive/ts-extension/README.md # what each file was and why it was keptThere is no CI. This repo runs no GitHub Actions workflow — the three gates below are run locally, and nothing enforces them on push. Run all three before you commit; that is the entire safety net.
uv run pytest # default suite: 705 passed, 6 skipped — no network, no live model
uv run ruff check # correctness lint (F, E9, B); currently clean
uv run pyright # static types over python/omds/src; currently clean
EGDS_SLOW=1 uv run pytest tests/benchmarks/ tests/test_packaging.py
# + the real-clone 5-gate benchmark (uv sync per clone)
# and a real `uv build` of the sdist + wheel
EGDS_E2E=1 uv run pytest tests/e2e/ # + the fake-LLM end-to-end tier (spec §1.3), no network
# best-effort: the real-endpoint E2E tier, against a live local model
EGDS_E2E=1 EGDS_E2E_BASE_URL=http://localhost:11434/v1 EGDS_E2E_MODEL=qwen2.5-coder:14b \
uv run pytest tests/e2e/
# best-effort: the live-Ollama SLM smoke — no env var opts in, the world does.
# `ollama serve` + `ollama pull qwen2.5-coder`, then it simply runs:
uv run pytest python/omds/tests/slm/test_ollama_smoke.py -rsThe six default skips are exactly those opt-in tiers (three in tests/e2e/,
one in tests/benchmarks/, one the SLM smoke, one the real uv build in
tests/test_packaging.py). The env-var names still use the EGDS_ prefix.
tests/test_packaging.py is worth knowing about:
every other test runs against the uv workspace — an editable install resolved by
uv.lock — which is not what a user gets. Two defects lived in that gap (an
uncapped mcp that resolved to a version without mcp.server.fastmcp, and a
source distribution that could not build at all), so these assert on the
declared metadata and on real built artifacts rather than on the imported
package.
The suite proves the code works; it does not prove an installation is wired
up. scripts/verify_install.py does that half, in
one command, against a throwaway temp directory:
uv run python scripts/verify_install.py # add --keep to inspect the temp dirIt checks that every console script resolves, that omds-install claude writes
the MCP entry and copies the skills, that a real omds-mcp subprocess speaks
stdio MCP and returns an ax:fit_on_train_only violation from
guardrails_check, that memory round-trips, that the plugin manifests validate
and the plugin's .mcp.json matches omds.install._OMDS_SERVER, that the
PostToolUse hook both blocks on a certain violation and stays silent with
omds-guardrails off PATH, and that the in-kernel guard raises on tagged
test data. The live SLM run is included when an Ollama is reachable and skipped
with a reason when it is not. It exits non-zero only on a real failure, and
touches nothing outside its temp dir.
It also prints, at the end, the things it cannot verify — chiefly that
Claude Code actually connects in-session, that a skill surfaces as
omds:leakage-audit, and that the hook fires on a real model edit. Those need a
human with a live session; the script is explicit about not covering them.
python/omds/tests/slm/test_ollama_smoke.py is the only test in the tree that
talks to a live model. Its gate is not an env var but reachability: a GET /api/tags with a 2-second timeout at collection time, and a check that the
model is really pulled — so it skips with a specific reason (server down vs.
model missing) instead of failing in transport, and a black-holed port cannot
stall the suite. EGDS_SLM_SMOKE_BASE_URL (Ollama's native root, no /v1)
and EGDS_SLM_SMOKE_MODEL override the defaults. It asserts only what must be
true of a working harness — the run terminated inside its cap, no transport or
budget fault, guardrails_check actually dispatched — never the model's prose,
because a 7B at temperature 0.2 is not deterministic.
tests/e2e/test_full_loop.py is the single source of truth for spec §1.3's
success criteria: the benchmark evolves end-to-end and GEPA's frontier has a
feasible, gate-passing member promotable via internal PR; an injected ontology
violation (unnormalized logits into log_loss) is rejected by the ontology gate
naming the axiom and the offending code-graph node; and every
candidate-evaluation subprocess runs with network=False.
GPU fine-tuning/RL of the underlying models; non-tabular modalities (vision,
audio); a web UI; distributed multi-machine execution; a full OWL/RDF reasoner;
harness self-mutation (GEPA only evolves a pipeline workspace, never omds
itself). Per the SLM scope guard: no autonomous missions on a 7B — anything
model-driven here stays bounded and supervised.
Apache License 2.0 — see NOTICE for the attribution
notices that §4(d) asks redistributors to carry along. Both files ship inside
the wheel and the sdist, so anyone who installs the package has them.
Apache-2.0 rather than a permissive-but-silent licence mainly for the express
patent grant in §3 and its termination clause, which MIT simply does not
address. The practical differences if you build on this: keep the LICENSE and
NOTICE with any redistribution, and state what you changed if you ship
modified files (§4(b)). There is no copyleft — Apache-2.0 does not require you
to open your own code.
No third-party source is vendored anywhere in the tree; everything external is a normal declared dependency. The project holds no contributions from anyone other than the copyright holder, so the relicence from MIT (2026-08-04) needed no third-party consent.
v2 of this project was briefly built as an extension of a pinned, fast-moving
single-maintainer coding-agent fork (@oh-my-pi/pi-coding-agent); that layer
was decommissioned and replaced by the harness-agnostic core described above.
The one algorithm worth keeping — the skill funnel — was ported to Python as
omds.slm.funnel; the rest is in git history (see
Repository layout). Both that decision
and the two amendments that reshaped it — Skills+CLI promoted over MCP
(2026-07-20), and the Claude Code plugin added as a third thin surface
(2026-07-31) — are recorded in
docs/adr/2026-07-18-mcp-pivot.md. The
framework evaluation that chose AutoGen for v1's since-deleted runtime is kept
for the record at
docs/specs/2026-07-12-framework-evaluation.md.