Evaluar is a local-first evaluation framework for vision and document systems. It gives Python code a repeatable way to call models, normalize their outputs, score them against ground truth, save the result, and inspect failures in a terminal UI.

It is intentionally explicit: evaluation logic lives in Python, thresholds live in reviewable config, and saved runs are plain JSON files.

What It Is

Evaluar is built for structured outputs such as bounding boxes, OCR text, table extraction, and merged document records. The core model is small:

1. Connectors produce raw predictions.
2. Normalizers shape raw responses into task schemas.
3. Evaluators compute metrics.
4. Scorers apply thresholds and produce verdicts.
5. Suites roll up pipeline verdicts into a run-level result.

The same run can be executed from the shell, saved for CI, opened in the TUI, or exported for review.

Capabilities

• Code-first eval files using the eval_*.py convention.
• Pipeline builders for detection, OCR, table extraction, and merged outputs.
• Callable, HTTP, fixture, and custom connector support.
• Mapping, function, and LLM-backed normalizers.
• Metric scorecards with pass, warn, fail, and error verdicts.
• Suite-level rollups across one or more pipelines.
• Local run storage in evaluar/results/<run_id>.json.
• Report commands for show, list, export, compare, prune, archive, delete, and clear.
• A Textual TUI for saved runs, charts, failure inspection, and bbox handoff.
• An OpenCV bbox editor for prediction overlays and small ground-truth edits.

Install

Evaluar is currently used as an internal/private Python package. Until the distribution decision is finalized, install it from an authenticated GitHub URL or from a local checkout you already have access to. Plain SSH clone only works after your GitHub account has repository access and a configured SSH key.

Use the install command that matches the project you are adding Evaluar to.

If the project already has a pyproject.toml and is managed by uv:

uv add "evaluar @ git+https://github.com/Koiiichi/evaluar.git"
uv run evaluar --version

If the project is a legacy requirements.txt / virtualenv repo with no pyproject.toml, install into the existing environment instead:

uv pip install --python .venv/bin/python "evaluar @ git+https://github.com/Koiiichi/evaluar.git"
.venv/bin/evaluar --version

Or, after activating the virtualenv:

source .venv/bin/activate
pip install "evaluar @ git+https://github.com/Koiiichi/evaluar.git"
evaluar --version

uv add modifies a uv project and therefore requires a pyproject.toml. uv pip install / pip install installs into an environment and is the right fit for existing non-uv repos.

For work inside this repository:

git clone https://github.com/Koiiichi/evaluar.git
cd evaluar
uv sync
uv run evaluar --version

If the repository is private, authenticate first with gh auth login, or use a GitHub token with package/repository access in your environment before running the install command.

Quick Start

Scaffold a detection evaluation:

uv run evaluar init detection --name layout_detector

This writes:

.
├── eval_layout_detector.py
├── evaluar.yaml
└── evaluar/
    ├── configs/layout_detector.yaml
    ├── ground_truth/layout_detector_gt.json
    ├── results/
    └── samples/sample_001.png

Run the generated eval file:

uv run evaluar eval_layout_detector.py

Open the TUI:

uv run evaluar

Run every eval file in the current repository:

uv run evaluar test --headless

Eval Files

An eval file is a Python file named eval_*.py. It exposes build_suite(...) and returns an EvaluarSuite.

from evaluar.api import detection, suite

SAMPLE_ID = "sample_001"


def layout_detector(image_url: str) -> dict:
    return {
        "prediction": [
            {
                "label_name": "room",
                "box": [100.0, 100.0, 400.0, 300.0],
                "score": 0.94,
            }
        ]
    }


def build_suite(sample_ids=None, config=None):
    ids = sample_ids or [SAMPLE_ID]
    pipeline = (
        detection("layout_detector")
        .callable(layout_detector)
        .inputs({SAMPLE_ID: {"image_url": "evaluar/samples/sample_001.png"}})
        .ground_truth(
            {
                SAMPLE_ID: {
                    "objects": [
                        {"label": "room", "bbox": [100.0, 100.0, 400.0, 300.0]}
                    ]
                }
            }
        )
        .default_mapping()
        .thresholds(map_50=(0.80, 0.60), precision=(0.85, 0.65))
        .gated_metrics("map_50", "precision")
        .build()
    )

    return (
        suite(sample_ids=ids, suite_name="layout")
        .rollup(pass_threshold=0.80, warn_threshold=0.60)
        .add_pipeline("layout_detector", pipeline)
    )


if __name__ == "__main__":
    result = build_suite().run(save=True)
    print(result.rollup_scorecard.verdict.value)

sample_id is the join key for one evaluation case. It connects model inputs, ground truth, suite selection, per-sample scorecards, and TUI inspection state.

Configuration

Python owns execution. YAML owns reviewable configuration.

evaluar init writes evaluar.yaml and evaluar/configs/<model>.yaml, but the eval file decides how much of that YAML to consume. This keeps callables, branching, normalizers, and non-serializable behavior in Python while allowing thresholds, rollups, config paths, and project metadata to remain diffable.

Scorers do not compute metrics. Evaluators compute metrics; scorers apply thresholds and finalize verdicts.

Task Types

Task	Helper	Evaluator / scorer role
Detection	detection("model_id")	Bounding-box metrics, AP-style scores, precision/recall, bbox artifacts.
OCR	ocr("model_id")	Text recognition metrics such as CER, WER, exact match, and sequence match.
Table	table("model_id")	Header, key-field, schema, and table-structure scoring.
Merged	merged("model_id")	Scoring one model response that already contains a merged record/schema shape.

Suites can contain one or more pipelines. The rollup scorecard is the suite-level verdict for the run.

Reports

Saved runs are single JSON files. The report CLI reads those files directly.

uv run evaluar report list
uv run evaluar report show <run_id>
uv run evaluar report export <run_id> --format summary-json --out summary.json
uv run evaluar report export <run_id> --format csv --out metrics.csv
uv run evaluar report export <run_id> --format markdown --out summary.md
uv run evaluar report compare <run_a> <run_b> --headless
uv run evaluar report archive <run_id> --out run.zip
uv run evaluar report prune --keep 20 --yes

The JSON result is the source of truth for reports, CI parsing, and TUI views.

TUI And Bbox Inspection

The TUI is a local view over saved runs and discovered eval files. It does not host a service or write to a separate database.

For detection runs, Evaluar can hand off to a separate OpenCV bbox editor for read-only prediction overlays or small ground-truth corrections:

Project Layout

The repository is organized around the runtime package, docs app, tests, and visual assets:

src/evaluar/        Python package
docs/               Fumadocs / Next.js documentation site
tests/              Unit, e2e, and TUI tests
misc/               Logo assets

The generated project layout created by evaluar init keeps executable eval files at the project root and managed evaluation assets under evaluar/.

Development

uv sync
uv run pytest
uv run evaluar --version

Useful focused checks:

uv run pytest tests/test_discovery.py
uv run pytest tests/e2e/test_cli_command_shape.py

Docs live in docs/:

cd docs
pnpm install
pnpm run dev

The docs app uses static Fumadocs search so it can build locally and deploy to Cloudflare Pages.

Design Principles

• Local first: runs are files, not hosted state.
• Python first: dynamic evaluation behavior stays in code.
• Explicit composition: connectors, normalizers, scorers, and suites are visible.
• Explicit behavior: no global plugin discovery or hidden registries in user projects.
• Inspectable artifacts: scorecards and metadata are persisted as JSON.
• TUI as a view: interactive inspection reads the same run data used by CI.

License And Status

Evaluar is currently a Latii Inc. proprietary product developed by Muneeb Hassan in R&D. No open-source license is granted at this time.

Discussions about open-sourcing Evaluar are underway. Until a license change is approved and published in this repository, treat the code, documentation, assets, and generated materials as private Latii Inc. property.