Eval

Evaluation framework for AI-native software — deterministic checks for agents, CLIs, files, snapshots, and workflow outputs.

v1.0.0 stable release: 27 checks, 14 CLI commands, 7 test suites.

Eval answers the question:

Did the agent, workflow, model, generated output, or code change actually do the right thing?

Spec defines what should happen. Eval checks whether the work actually produced the expected evidence.

Eval is part of Kujo’s Control layer: measurable outcomes, repeatable checks, reviewable reports, and machine-readable evidence.

Prioritize copyable examples over tests: examples should model the most token-efficient idioms we want agents to imitate.

Exclude generated/bulk paths from the main sweep unless the task explicitly targets them; document the search exclusions you used.

Why This Exists

Every serious AI-native ecosystem needs evaluations. Without evals, agents are vibes. With evals, agents become software components that can be tested, trusted, compared, and improved.

Eval provides:

27 deterministic checks — command success/failure, file existence/content/line count, JSON shape/value/path, HTTP status/body, regex, timing, snapshots, directory comparison, environment variables, and more.
5 report formats — Markdown, HTML (collapsible details + summary cards), JUnit XML, TAP, and NDJSON streaming.
Snapshot testing — store expected outputs and detect regressions with unified diff.
Run comparison — compare two eval runs to detect improvements or regressions.
Retry support — mark flaky tests with "retry": N to auto-retry on failure.
Dependency ordering — declare "depends_on": [...] to skip tests when dependencies fail.
Enterprise policy presets — use "policy_profile": "strict-ci" | "local-dev" | "release-gate" for fast onboarding.
Timing — every run records duration_ms; command_timing_less_than check for perf regression.
Quiet/verbose modes — --quiet suppresses per-test output (CI mode); --verbose prints full check details.
14 CLI commands — init, run, report, compare, list-checks, snapshots, version, watch, lint, policy-explain, diff, export, verify-manifest, completion.
CI-ready — non-zero exit on failure; JSON/NDJSON output; GitHub Actions step summary; shields.io badge.
Deterministic CI artifacts — each run writes summary.json and artifact-manifest.json for machine consumers.
Machine summary channel — run/report write cli-summary.json (or --summary-channel-path) as a stable automation handoff artifact.
Artifact integrity mode — opt into SHA-256 checksums in artifact-manifest.json and verify with verify-manifest.
Parallel scale mode — opt into --parallel-workers for independent file-check fast paths with deterministic ordering.
Retention controls — bound output growth with history_retention_runs and aux_artifact_retention_files.
Security controls — 5-layer defense: command allowlisting + pattern policy, path boundary enforcement, output redaction, config size/depth limits, max output truncation.
Command guardrails — overly long command payloads are rejected before execution (MAX_COMMAND_LENGTH).
Fully local — no API keys, no network dependencies, completely deterministic.

Kujo's core argument is that AI-native software needs new primitives. Eval is one of those primitives.

Traditional tests answer:

Does this function return the expected value?

Eval answers:

Did this agent or AI-assisted workflow produce the correct outcome under real project constraints?

Requirements

Kujo CLI/runtime available on your machine.
No API keys or external services needed (fully local, deterministic).

Containerized Usage

This repository includes a local/CI-oriented multi-stage Dockerfile that compiles the Kujo runtime from the pinned revision in RUNTIME_VERSION. Treat production deployment of the image as target-environment work: rebuild it in your controlled pipeline, scan it with your registry policy, and validate it against your runner boundaries before using it for production claims.

Build the image:

docker build -t kujo-eval:local .

Optionally override the runtime ref at build time:

docker build --build-arg KUJO_RUNTIME_REF=<tag-or-commit> -t kujo-eval:local .

Run a suite in the container:

docker run --rm -v "$PWD":/workspace -w /workspace kujo-eval:local run

Generate a report in the container:

docker run --rm -v "$PWD":/workspace -w /workspace kujo-eval:local report --format json --out eval_results/report.json

Runtime Binary Selection

kujo-eval expects the Kujo language runtime command to be available as kujo:

kujo --version

If your shell resolves kujo to a different tool, fix your PATH before running these examples.

VM-First Migration Note (Interpreter-Era Guidance)

Older repository docs and shell snippets may still show interpreter-era commands such as:

kujo run main.kujo --interpreter run <suite>

Current recommended default is VM-first execution:

kujo run main.kujo run <suite>

Use --interpreter only for runtime compatibility checks, warning investigations, or parity debugging.

Migration mapping:

Legacy: kujo run main.kujo --interpreter <command>
Preferred: kujo run main.kujo <command>
Optional compatibility check: kujo run main.kujo --interpreter <command>

Interpreter mode can emit KUJORUN001 warning noise while still succeeding; rely on exit code plus generated artifacts (summary.json, artifact-manifest.json) for CI pass/fail decisions.

Operational Watchdog Controls

Release and parity scripts include artifact-aware watchdogs to keep CI deterministic even when interpreter output capture behaves inconsistently.

Watchdog loops use portable sleep polling so behavior stays consistent across macOS and Linux shells.

scripts/release_quality_gates.sh
- KUJO_EVAL_GATE_TIMEOUT_SECONDS (default: 120)
- KUJO_EVAL_BENCH_SUITE_BUDGET_MS (default: 600)
- KUJO_EVAL_BENCH_MEDIUM_SUITE_BUDGET_MS (default: 3000)
- KUJO_EVAL_BENCH_LARGE_SUITE_BUDGET_MS (default: 8000)
- KUJO_EVAL_BENCH_IO_HEAVY_SUITE_BUDGET_MS (default: 12000)
- KUJO_EVAL_REQUIRE_RELEASE_SIGNOFF (default: 0, set to 1 to require human approval)
- KUJO_EVAL_RELEASE_SIGNOFF_FILE (default: docs/release-signoff.md)
scripts/cli_smoke_matrix.sh
- KUJO_EVAL_CLI_SMOKE_INCLUDE_INTERPRETER (default: 0)
scripts/verify_artifact_contract.sh
- deterministic artifact shape/path verification for summary.json, artifact-manifest.json, and cli-summary.json
scripts/verify_docs_command_parity.sh
- KUJO_EVAL_DOCS_WATCHDOG_TIMEOUT_SECONDS (default: 120)

Version note:

Use version for the CLI version/contract output.
--version is not implemented as a dedicated alias and falls back to help output.

Example:

export KUJO_EVAL_GATE_TIMEOUT_SECONDS=180
export KUJO_EVAL_BENCH_SUITE_BUDGET_MS=600
export KUJO_EVAL_BENCH_MEDIUM_SUITE_BUDGET_MS=3000
export KUJO_EVAL_BENCH_LARGE_SUITE_BUDGET_MS=8000
export KUJO_EVAL_BENCH_IO_HEAVY_SUITE_BUDGET_MS=12000
export KUJO_EVAL_DOCS_WATCHDOG_TIMEOUT_SECONDS=180
scripts/cli_smoke_matrix.sh
scripts/verify_artifact_contract.sh
# Optional parity mode:
# KUJO_EVAL_CLI_SMOKE_INCLUDE_INTERPRETER=1 scripts/cli_smoke_matrix.sh --include-interpreter
scripts/release_quality_gates.sh

Known Runtime Limitations

Interpreter mode may print KUJORUN001 type-check warnings even when commands exit successfully.
Command checks support native process timeout termination through timeout_ms and suite-level timeout_seconds.
watch is intentionally long-running and should be interrupted manually in local shells.
Command and file checks run in the host environment and are not a full sandbox boundary.
Some interpreter invocations can stall when stdout/stderr is shell-captured; release/parity gate scripts mitigate this with artifact-driven watchdogs.

Launch Scope Snapshot

Eval is verified for deterministic local and CI evaluation workflows where shell/file access is expected and controlled by repository policy.

It is not yet a universal fit for every enterprise testing context. Before broad rollout, evaluate the following constraints for your environment:

Timeout enforcement depends on runtime support for execute_status timeout handling in your target Kujo build.
No isolated sandbox boundary; commands execute with host-level permissions.
Runtime warning noise in interpreter mode may require log filtering in strict observability pipelines.
Feature roadmap items (LLM-as-judge, model-cost scoring, distributed execution) are intentionally out of current deterministic scope.

If your use case needs strict isolation, guaranteed process preemption, or non-local execution controls, place Eval behind additional platform controls (container boundaries, workload policies, and external supervisors).

Enterprise Deployment Patterns

Eval can be deployed in enterprise CI/CD and platform workflows with a simple layered model:

Runner boundary: execute eval jobs in short-lived containers or isolated runners with least-privilege filesystem mounts.
Command policy: define allowed_commands, allowed_command_patterns, and blocked_arg_patterns at suite level; override only where needed at test level.
Path policy: scope file assertions using allowed_paths to explicitly owned workspace directories.
Environment policy: expose only approved variables and enforce with allowed_env_vars.
Artifact policy: keep report outputs (--output-dir) in per-run isolated folders and publish only required formats (JSON/JUnit/TAP/HTML/NDJSON).

Suggested rollout:

Start with --json --quiet in CI to establish deterministic pass/fail gates.
Add JUnit/TAP report publication for existing enterprise test dashboards.
Enforce suite-level command/path/env policy defaults before allowing per-test overrides.
Add external supervision for hard-kill guarantees on long-running processes until native runtime preemption support is available.

Why Eval vs. Other Tools?

Feature	Eval	Jest	Pytest	Bats
Language	Kujo	JavaScript	Python	Bash
Dependencies	Zero (Kujo runtime only)	npm + 100s packages	pip + plugins	Bash + helpers
Deterministic	✅ Fully	⚠️ JS runtime variance	⚠️ Python env variance	❌ Shell-dependent
Command execution	✅ Native	❌ Needs child_process	❌ Needs subprocess	✅ Native
File checks	✅ 11 types	❌ Needs custom matchers	❌ Needs custom fixtures	❌ Manual
HTTP checks	✅ Built-in	✅ With supertest	✅ With requests	❌ Needs curl
Snapshot testing	✅ Built-in diff	✅ Built-in	✅ With syrupy	❌ Manual
Report formats	5 (md/html/junit/tap/ndjson)	1 (CLI)	2 (CLI + JUnit)	1 (TAP)
Security model	✅ 5-layer defense	❌ None	❌ None	❌ None
AI-native	✅ Purpose-built	❌ Traditional	❌ Traditional	❌ Shell testing
Install size	< 1MB	~50MB+	~30MB+	~2MB

Eval fills a gap: it's a deterministic, zero-dependency, AI-native evaluation framework that tests outcomes (commands, files, JSON, HTTP) rather than code units. It's designed for agents, CLIs, and workflow outputs — not traditional unit tests.

Canonical Examples

Use these as the canonical copy-paste examples before mining tests, generated artifacts, or historical checklists:

Use Case	File	What It Tests
Minimal release gate	`examples/release_gate_suite.json`	Small passing suite for smoke tests, docs parity, and automation examples
Enterprise CLI quality gate	`examples/enterprise_cli_quality_gate.json`	Policy-first CLI behavior checks, output assertions, repository file guards
Enterprise API contract gate	`examples/enterprise_api_contract_gate.json`	Fixture-backed API contract validation, JSON shape/value assertions, parallel file checks
Enterprise agent output gate	`examples/enterprise_agent_output_gate.json`	Policy-first agent output validation for structure/content/confidence and timing
Enterprise policy baseline	`examples/policy_profile_release_gate.json`	Profile-driven command/path/env policy defaults
Strict-enterprise policy example	`examples/strict_enterprise_policy_gate.json`	Minimum-privilege command/path/env allowlists with redaction audit defaults
Sandbox-adjacent policy example	`examples/sandbox_adjacent_policy_gate.json`	Constrained local policy boundaries for fixture-only command and file access
Expected-fail intro suite	`examples/basic_suite.json`	Core checks plus one deliberate failing case for report demos
Bulk accounting fixture	`examples/large_suite_fixture.json`	Tags, dependencies, retries, skip behavior; useful for fixtures, not first-copy examples

Replace placeholders (URLs, file paths, tool names) with your actual values and run immediately.

Generated and bulk paths are intentionally not canonical examples: avoid eval_results/, tests/*.out, examples/fixtures/contracts/, and large stress/performance suites unless you are validating artifact shape, snapshots, or scale behavior.

See architecture and lifecycle diagrams in docs/ARCHITECTURE.md.

Repository Layout

The implementation lives in src/; the root stays intentionally small and runtime-facing:

Path	Purpose
`main.kujo`	Canonical CLI entrypoint for `kujo run main.kujo ...`; keep at root so examples, scripts, package metadata, and CI share one invocation.
`src/`	CLI parsing, config validation, check implementations, suite execution, reports, snapshots, and shared utilities.
`examples/`	Copyable suites and deterministic fixtures.
`tests/`	Contract, security, CLI, quality, stress, benchmark, and coverage suites.
`schema/`	JSON Schema contract for suite authors and editor integrations.
`scripts/`	Release gates, docs parity checks, command inventory generation, and artifact-contract verification.
`docs/`	Deep references, runbooks, architecture, and session handoff checklists.

Root copies of README.md, SECURITY.md, CONTRIBUTING.md, LICENSE, package manifests, Dockerfile, and RUNTIME_VERSION are kept for GitHub/package-manager conventions. Generated outputs such as eval_results/, .eval_*, snapshots/, and tests/*.out are ignored.

Quick Start

Minimal runnable path from a clean checkout:

kujo run main.kujo run examples/release_gate_suite.json --output-dir .eval_quickstart --json

Expected output signal:

{"ok":true}

Common follow-ups:

# Compact human summary
kujo run main.kujo run examples/release_gate_suite.json --output-dir .eval_quickstart --summary-only

# HTML report artifact
kujo run main.kujo report examples/release_gate_suite.json --rerun --output-dir .eval_quickstart --format html

# Manifest checksum verification
kujo run main.kujo run examples/release_gate_suite.json --output-dir .eval_quickstart --artifact-checksums --json
kujo run main.kujo verify-manifest --output-dir .eval_quickstart --json

# Discover command/check surface
kujo run main.kujo list-checks
kujo run main.kujo version

init scaffolds a starter suite; edit the generated file before using it as a gate.

Enterprise Quickstart Matrix (Risk Tiers)

Risk Tier	Best Fit	Command
Tier 1 (Local dev, low risk)	Fast feedback while iterating on checks and fixtures	`kujo run main.kujo run examples/enterprise_cli_quality_gate.json --output-dir .eval_enterprise_cli --json`
Tier 2 (CI gate, medium risk)	Contract + payload validation in pull requests	`kujo run main.kujo run examples/enterprise_api_contract_gate.json --output-dir .eval_enterprise_api --parallel-workers 8 --json`
Tier 3 (Release gate, high risk)	Policy-first release verification with strict controls	`kujo run main.kujo run examples/strict_enterprise_policy_gate.json --output-dir .eval_enterprise_strict --json`
Tier 3 + constrained sandbox-adjacent workflows	Fixture-only policy boundaries for local security testing	`kujo run main.kujo run examples/sandbox_adjacent_policy_gate.json --output-dir .eval_sandbox_adjacent --json`

Policy visibility and command inventory for onboarding:

kujo run main.kujo policy-explain examples/release_gate_suite.json --policy-stage release --json
bash scripts/generate_command_inventory.sh --check

Named path-policy profiles reduce boilerplate for staged hardening:

Profile	Effect
`open`	Leaves path policy open; use only for local exploration or externally sandboxed runs.
`ci-restricted`	Enables `allowlist-required` and permits common repository-owned docs, schema, examples, source, and tests paths.
`release-deny-default`	Enables `allowlist-required` with a narrow release artifact/fixture/schema allowlist; add explicit `allowed_paths` for anything else.

Profiles can be set as top-level path_policy_profile defaults or inside policy_stage_overlays.<stage>.path_policy_profile; explicit path_policy_mode and allowed_paths override profile defaults.

Repository Status (Validated Commands)

Validated on 2026-05-25 with the kujo command on PATH.

Command	Exit	Expected Output Signal
`kujo run main.kujo version`	0	Prints `Eval v2.0.0` and `Contract version: 2.0.0`
`kujo run main.kujo list-checks`	0	Prints available check list and `Total: 27 check types`
`kujo run main.kujo snapshots`	0	Prints snapshot listing output (or empty listing message)
`kujo run main.kujo diff README.md README.md`	0	Reports no differences for identical files
`kujo run main.kujo run examples/release_gate_suite.json --output-dir .eval_status --json`	0	JSON result envelope includes `"ok":true`
`kujo run main.kujo run examples/release_gate_suite.json --output-dir .eval_status --summary-only`	0	Prints compact summary and overwrites prior run artifacts safely
`kujo run main.kujo report examples/release_gate_suite.json --rerun --output-dir .eval_status --json`	0	JSON result envelope includes `"ok":true`
`kujo run main.kujo report examples/release_gate_suite.json --rerun --output-dir .eval_status --format junit`	0	Generates `.eval_status/eval-report.xml`
`kujo run main.kujo report examples/release_gate_suite.json --rerun --output-dir .eval_status --format tap`	0	Generates `.eval_status/eval-report.tap`
`kujo test --update --runtime vm && kujo test`	0	Test suite runner reports all configured test files passing after generating ignored local baselines
`kujo test --update --runtime vm && kujo test --runtime vm`	0	VM runtime reports all 7 test suites passing after generating ignored local baselines

Check Types

Command Checks

Check	Description
`command_succeeds`	Run a shell command, assert exit code 0
`command_fails`	Run a shell command, assert non-zero exit code
`exit_code_equals`	Run a command, assert specific exit code

Output Checks

Check	Description
`output_contains`	Run command, assert stdout contains substring
`output_does_not_contain`	Run command, assert stdout does NOT contain substring
`output_matches_glob`	Run command, assert stdout matches glob/substring pattern

File Checks

Check	Description
`file_exists`	Assert a file path exists
`file_does_not_exist`	Assert a file path does NOT exist
`file_contains`	Assert file content contains a substring
`file_does_not_contain`	Assert file content does NOT contain a substring
`file_matches_glob`	Assert file content matches glob/substring pattern
`file_size_greater_than`	Assert file size exceeds a byte threshold
`file_size_less_than`	Assert file size is under a byte threshold

Data Checks

Check	Description
`json_matches_shape`	Assert JSON has expected keys
`json_value_equals`	Assert a specific JSON path has an expected value
`stdout_json_matches_shape`	Run command, parse stdout as JSON, validate shape
`snapshot_matches`	Compare output against a stored snapshot
`directory_diff`	Compare two directories and report differences

Environment & Network Checks

Check	Description
`env_var_equals`	Assert an environment variable has a specific value
`http_status`	Assert an HTTP endpoint returns expected status code

Eval Suite Format

Create an eval.json file:

{
  "name": "my-agent-eval",
  "description": "Tests for the auth agent workflow",
  "version": "1.0.0",
  "output_dir": "./eval_results",
  "snapshot_dir": "./snapshots",
  "stop_on_failure": false,
  "tests": [
    {
      "name": "auth command succeeds",
      "description": "The auth CLI should exit cleanly",
      "check": "command_succeeds",
      "params": {
        "command": "kujo run auth.kujo login --user test"
      }
    },
    {
      "name": "output file exists",
      "description": "Auth should produce a token file",
      "check": "file_exists",
      "params": {
        "path": "./output/token.json"
      }
    },
    {
      "name": "token JSON has required keys",
      "check": "json_matches_shape",
      "params": {
        "path": "./output/token.json",
        "required_keys": ["access_token", "refresh_token", "expires_in"]
      }
    },
    {
      "name": "output contains success message",
      "check": "output_contains",
      "params": {
        "command": "kujo run auth.kujo login --user test",
        "expected": "Login successful"
      }
    }
  ]
}

Policy Profiles And Retention

Top-level optional controls for enterprise rollout:

Field	Type	Default	Purpose
`policy_profile`	string	`""`	Preset security/governance baseline (`strict-ci`, `local-dev`, `release-gate`)
`policy_stage_overlays`	object	`{}`	Stage-specific policy overrides keyed by `local`, `ci`, `release` (selected with `--policy-stage`)
`history_retention_runs`	int	`200`	Maximum entries retained in `history.json`
`aux_artifact_retention_files`	int	`25`	Maximum retained auxiliary backup artifacts (`*.bak`)
`artifact_checksums`	bool	`false`	Include SHA-256 checksums in `artifact-manifest.json`

Run/report outputs now include:

summary.json: compact deterministic run summary for dashboards/status collectors
artifact-manifest.json: deterministic file index for CI artifact discovery
cli-summary.json: stable machine handoff channel for run/report command outcomes

When checksum mode is enabled (artifact_checksums: true or --artifact-checksums), artifact-manifest.json also includes a checksums block with SHA-256 digests. Validate it with:

kujo run main.kujo verify-manifest --output-dir ./eval_results --json

Snapshot Testing

Snapshots store expected output so you can detect regressions:

{
  "name": "snapshot test",
  "check": "snapshot_matches",
  "params": {
    "name": "my-cli-help",
    "command": "my-tool --help",
    "snapshot_dir": "./snapshots"
  }
}

First run with --update-snapshots to create the baseline:

kujo run main.kujo run --update-snapshots

Subsequent runs will compare against the stored snapshot and fail if output changes.

Comparison

Compare two eval runs to detect regressions or improvements:

kujo run main.kujo compare ./eval_results/run-v1.json ./eval_results/run-v2.json

CI Integration

Eval exits with code 1 on any test failure, making it CI-ready:

- name: Run eval suite
  run: kujo run main.kujo run --json

Use --json for machine-readable output in CI pipelines.

CLI Exit Code Contract

Eval currently uses explicit process exit classes:

Exit Code	Class	Meaning
`0`	Success	Command completed successfully (for `run`, all assertions passed).
`1`	Assertion failure	At least one eval assertion failed during `run`.
`1`	Usage/config/runtime error	Invalid command usage, missing/invalid config, or command execution error.

Practical CI guidance:

Treat any non-zero exit as a hard failure gate.
Parse --json envelopes to distinguish assertion failures from usage/config/runtime failures.
Use JUnit/TAP artifacts for external dashboards where required.

Command Reference

Command	Description
`init`	Create an eval suite file (`--name <name>`)
`run`	Execute all tests in a suite (`--filter`, `--exclude`, `--tags`, `--only-failed`, `--quiet`, `--summary-only`, `--verbose`, `--json`)
`report`	Generate a report from cached results (`--rerun`, `--format <md\|html\|junit\|tap\|ndjson>`, `--json`)
`compare`	Compare two eval run result files
`list-checks`	List all 27 available check types
`snapshots`	List stored snapshots (`--snapshot-dir <dir>`)
`version`	Print eval contract version (currently v2.0.0)
`watch`	Watch config file and re-run on changes
`lint`	Validate eval config without executing tests
`policy-explain`	Print effective merged command/path/env policy for a given stage
`diff`	Compare two files line-by-line
`export`	Export an eval suite as a shell script
`verify-manifest`	Verify artifact-manifest checksum entries against current files
`completion`	Generate shell completion script (`--shell bash\|zsh\|fish`)

Common flags: --json (machine output), --config <path>, --output-dir <dir>, --update-snapshots, --format <md|html|junit|tap|ndjson>, --quiet, --summary-only, --verbose

Generated command inventory: docs/COMMAND_INVENTORY.md (validated via bash scripts/generate_command_inventory.sh --check).

Flaky Test Retry

Add a "retry" field to any test definition to automatically retry on failure:

{
  "name": "flaky network check",
  "check": "http_status",
  "params": { "url": "https://api.example.com/health" },
  "retry": 2
}

The test will run up to 3 times (1 initial + 2 retries) and pass if any attempt succeeds.

Contract Version

Eval uses a describe_module() function for runtime contract discovery. Call it from any Kujo script:

kujo run main.kujo version
# Eval v2.0.0
# Contract version: 2.0.0

The result envelope contract ({ok, error, data}) is stable since v2.0.0 and used consistently across all source modules.

Roadmap

Completed:

Future:

LLM-as-judge evaluations
Agent replay evaluation
Workflow-level evals
Model comparison scoring
Cost and latency scoring
Scout-assisted eval generation
Distributed suite execution across workers/runners

Repository Layout

main.kujo: CLI entry point and command dispatcher for the full command surface
src/common.kujo: Shared utilities (dict_get_or, normalize_*, make_check_*)
src/cli.kujo: CLI argument parsing and help text
src/config.kujo: Configuration loading, validation, and KNOWN_CHECKS
src/checks.kujo: All 27 check implementations + security validators
src/eval_core.kujo: Core eval engine, suite runner, compare_runs
src/report.kujo: Markdown, HTML, JUnit XML, TAP, and NDJSON report generation
src/snapshot.kujo: Snapshot management (save, compare, diff, list, delete)
tests/: 7 test suites — contract, security, CLI integration, coverage, benchmark, quality, stress
examples/: Example eval suites (basic, release-gate, self-check, snapshot)
schema/eval-suite.schema.json: JSON Schema for VSCode autocomplete + validation
docs/: Architecture, agent notes, ecosystem integration, roadmaps
CONTRIBUTING.md: Contributing guide and code conventions
SECURITY.md: Security model, known limitations, reporting
docs/agent-notes.md: Kujo runtime quirks and lessons learned
docs/release-candidate-runbook.md: Human runbook for pre-tag release validation
RUNTIME_VERSION: Pinned Kujo runtime commit hash for reproducible CI

Root Layout Notes

main.kujo intentionally stays at the repository root as the CLI entrypoint referenced by docs, scripts, and CI.
CONTRIBUTING.md and SECURITY.md at root are the canonical governance documents for GitHub and package-manager discoverability.
Functional implementation modules remain under src/; root files are only packaging, entrypoint, policy, and governance artifacts.
tests/*.out files are treated as runtime artifacts and intentionally remain untracked.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
.github		.github
config		config
docs		docs
examples		examples
schema		schema
scripts		scripts
src		src
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
RUNTIME_VERSION		RUNTIME_VERSION
SECURITY.md		SECURITY.md
eval.spec.yml		eval.spec.yml
kennel.toml		kennel.toml
kujo.toml		kujo.toml
main.kujo		main.kujo

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Eval

Why This Exists

Requirements

Containerized Usage

Runtime Binary Selection

VM-First Migration Note (Interpreter-Era Guidance)

Operational Watchdog Controls

Known Runtime Limitations

Launch Scope Snapshot

Enterprise Deployment Patterns

Why Eval vs. Other Tools?

Canonical Examples

Repository Layout

Quick Start

Enterprise Quickstart Matrix (Risk Tiers)

Repository Status (Validated Commands)

Check Types

Command Checks

Output Checks

File Checks

Data Checks

Environment & Network Checks

Eval Suite Format

Policy Profiles And Retention

Snapshot Testing

Comparison

CI Integration

CLI Exit Code Contract

Command Reference

Flaky Test Retry

Contract Version

Roadmap

Repository Layout

Root Layout Notes

License

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages