v0.2.0

What's Changed

• Redesign the report as a document, and add a demo walkthrough by @rlaope in #1
• Restyle the demo screen as a technical report, and serve both surfaces from one workbench by @rlaope in #2
• Redesign the report, and print money in each currency's own decimals by @rlaope in #3
• Fix the surfaces the redesign left unverified: terminal money, rounding parity, drift labels, print by @rlaope in #4
• Measure yes/no answers on the probability that they are right by @rlaope in #5
• Add the threshold-shift drift gate and a paired head-to-head comparison by @rlaope in #6
• Make the library and the command line one loop: collect.resolve, join on read, jeval status by @rlaope in #7
• Add discrimination and classwise calibration; fix the mislabelled line, the gold count and two edge cases by @rlaope in #8
• Say each finding as a sentence and draw it; fold the tables underneath by @rlaope in #9
• Release 0.2.0 by @rlaope in #10

New Contributors

• @rlaope made their first contribution in #1

Full Changelog: v0.1.8...v0.2.0

v0.1.8

Full Changelog: v0.1.7...v0.1.8

jeval v0.1.7 — segments for instrumented services

Install

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh
jeval demo

Fixed in this release

Segments for instrumented services. collect.track() recorded every answered call but had no
way to say where a request came from, so --by lang answered "no record carries segment 'lang'"
for data jeval had collected itself. track() now takes segment= — a mapping, or a callable over
the call's arguments, exactly like source_key — and the value reaches the record, the segments
section and the per-segment threshold sweep.

A wrapper that hooked nothing is counted. A client whose answering method is not one of the
default names comes back working and unwrapped with no signal, so the log stays empty and nothing
says why. That case is now counted in stats()["no_method_found"], and the docs say to check
stats()["calls"] after the first request.

jeval v0.1.6 — a synthetic classifier that answers more than one class

Install

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh
jeval demo

Fixed in this release

The synthetic classifier only ever answered one class. generate() set prediction = classes[0]
for every record, so the demo, the committed example report and the drift example all described a
classifier that routes exactly one class: per-class thresholds, per-class cost actions and slices by
predicted class were unreachable. The prediction now follows the sampled outcome. Confidence is also
capped below certainty — clipping at 1.0 stacked 45% of records on exactly 1.0 at inflation 1.22 and
handed the threshold sweep a free bucket.

The example report and every number quoted from it were regenerated, including the screenshots.

The drift block in the README is reproducible: examples/make-drift-log.py builds the log it
quotes.

jeval v0.1.5 — one column is not two fields

Install

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh

Fixed in this release

One column is not two fields. A schema field the map did not mention fell back to a raw column of
the same name, so an application logging its prediction in a column called label fed that column to
the schema's label as well — a ground truth equal to the prediction, every record correct, silver
labeled, out of nothing. The report refused to measure it, but the records were poisoned. A column
already assigned to another field is no longer read implicitly; explicit mappings are unaffected.

jeval v0.1.4 — flat columns as segments, readable skip reasons

Install

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh
jeval demo

uvx --from git+https://github.com/rlaope/jeval@v0.1.4 jeval demo
pip install https://github.com/rlaope/jeval/releases/download/v0.1.4/jeval_cli-0.1.4-py3-none-any.whl

Fixed in this release

• A flat column can be a segment. A log with lang: ko as a column could not reach the
  documented --by lang sweep: the mapping renamed fields but could not build the object the schema
  wants, so every row was skipped. A non-object value mapped onto segment now becomes
  {column: value}; an object column is still mapped as an object.
• A skipped row says what to change. The message was a pydantic dump that named neither the field
  nor the file: 2 validation errors for DecisionRecord ... visit https://errors.pydantic.dev/....
  A missing field now says to add it to field_map or defaults, and a text confidence names the
  field and the value.

jeval v0.1.3 — correct version reporting, distribution jeval-cli

Install

uvx --from git+https://github.com/rlaope/jeval@v0.1.3 jeval demo   # nothing installed

pip install https://github.com/rlaope/jeval/releases/download/v0.1.3/jeval_cli-0.1.3-py3-none-any.whl

The PyPI distribution is jeval-cli: pip install jeval installs an unrelated project owned by
another account. The jeval command is unchanged.

Hand it to an agent

Install jeval (uvx --from git+https://github.com/rlaope/jeval jeval --help), find where my
classifier's decisions are logged, describe that shape in .jeval/ingest-map.yaml, run
jeval report, and show me the report file.

• Machine-readable entry point: llms.txt
• The playbook it follows: docs/agent-setup.md

What comes out

Artifact	What it answers	Who reads it
report.html — one self-contained file	Are the confidences trustworthy, where do they break, what does the threshold in use cost?	the human
thresholds.yaml	The threshold to deploy, with a bootstrap interval	your application
labels.csv	Which decisions, labeled next, buy the most certainty	whoever has the answers
calibration-*.yaml	An optional correction map, exported only when the gain is real	your application

Fixed in this release

jeval --version reported 0.1.0 from any build, because the package asked PyPI metadata for a
distribution name it no longer uses. It now reports the installed distribution's real version.

Verified

491 tests, ruff + mypy clean, CI green on 3.10 / 3.11 / 3.12. Installed from this release's wheel
into a clean virtualenv, which reported jeval 0.1.3.

jeval v0.1.2 — the distribution is jeval-cli

Install

uvx --from git+https://github.com/rlaope/jeval@v0.1.2 jeval demo   # nothing installed

pip install https://github.com/rlaope/jeval/releases/download/v0.1.2/jeval_cli-0.1.2-py3-none-any.whl

The PyPI distribution is jeval-cli: pip install jeval installs an unrelated project owned by
another account. The jeval command is unchanged.

Hand it to an agent

Install jeval (uvx --from git+https://github.com/rlaope/jeval jeval --help), find where my
classifier's decisions are logged, describe that shape in .jeval/ingest-map.yaml, run
jeval report, and show me the report file.

• Machine-readable entry point: llms.txt
• The playbook it follows: docs/agent-setup.md

What comes out

Artifact	What it answers	Who reads it
report.html — one self-contained file	Are the confidences trustworthy, where do they break, what does the threshold in use cost?	the human
thresholds.yaml	The threshold to deploy, with a bootstrap interval	your application
labels.csv	Which decisions, labeled next, buy the most certainty	whoever has the answers
calibration-*.yaml	An optional correction map, exported only when the gain is real	your application

Verified

490 tests, ruff + mypy clean, CI green on 3.10 / 3.11 / 3.12. Installed from this release's wheel
into a clean virtualenv, jeval demo produced a 257 KB report with no external references.

jeval v0.1.1 — install today, hand it to an agent

Install

uvx --from git+https://github.com/rlaope/jeval@v0.1.1 jeval demo   # nothing installed

pip install https://github.com/rlaope/jeval/releases/download/v0.1.1/jeval-0.1.1-py3-none-any.whl

pip install jeval needs a PyPI publisher registration that is not done yet. The wheel above is
built by the release workflow for this tag, not uploaded by hand.

Hand it to an agent

Nobody runs a command tour. One sentence, and the artifact comes back:

Install jeval (uvx --from git+https://github.com/rlaope/jeval jeval --help), find where my
classifier's decisions are logged, describe that shape in .jeval/ingest-map.yaml, run
jeval report, and show me the report file.

• Machine-readable entry point: llms.txt
• The playbook it follows: docs/agent-setup.md
  — including the two places every attempt stalls: nothing logged yet, and no labels yet.

What comes out

Artifact	What it answers	Who reads it
report.html — one self-contained file	Are the confidences trustworthy, where do they break, what does the threshold in use cost?	the human
thresholds.yaml	The threshold to deploy, with a bootstrap interval	your application
labels.csv	Which decisions, labeled next, buy the most certainty	whoever has the answers
calibration-*.yaml	An optional correction map, exported only when the gain is real	your application

Verified

489 tests, ruff + mypy clean, CI green on 3.10 / 3.11 / 3.12. Installed from the release URL into a
clean virtualenv, jeval demo produced a 257 KB report.

jeval v0.1.0 — measure what your classifier's confidence is worth

Install

uvx --from git+https://github.com/rlaope/jeval jeval demo

Or with pip, straight from this release:

pip install https://github.com/rlaope/jeval/releases/download/v0.1.0/jeval-0.1.0-py3-none-any.whl
jeval demo

What it does

Your classifier answers with a confidence. jeval measures what that confidence is actually worth
on your own data, and — from what a mistake costs you — derives the confidence line at which a
decision should be automated instead of handed to a human.

Everything is computed from labeled decision records on disk. No server, no database, no network
call, no account.

Using it from an agent

The whole setup is one sentence to an agent:

Install jeval (uvx --from git+https://github.com/rlaope/jeval jeval --help), find where my
classifier's requests and responses are logged, adapt .jeval/ingest-map.yaml to that shape, run
jeval report, and show me the report file.

Machine-readable entry point: https://github.com/rlaope/jeval/blob/main/llms.txt

What you get

Artifact	What it answers
report.html — one self-contained file	Are the confidences trustworthy, where do they break, and what does the current threshold cost?
thresholds.yaml — read by your application	The threshold to deploy, with a bootstrap interval, and whether splitting by segment pays
labels.csv — a sheet to fill in	The decisions whose labels would teach the tool the most
calibration-*.yaml — optional	A correction map your application applies, exported only when the gain is real

Verified

488 tests, ruff + mypy clean, CI green on 3.10 / 3.11 / 3.12.