v0.29.0

Every host app must run bin/rails completion_kit:install:migrations && bin/rails db:migrate for this release. It adds a check_score column to completion_kit_reviews. Every check metric writes that column when it grades a row, and the run pages read it, so a host that skips the migration gets failed checks and broken run pages whether or not it uses list_overlap.

Changed (breaking)

• A regex check with no saved case_sensitive now matches case-insensitively. regex was the one kind that read a missing case_sensitive as case-sensitive, while every other kind read it as case-insensitive. The form's "Case sensitive" box sent nothing when unticked, so the key was never saved, the box misreported what the check did, and the case-insensitive setting it seemed to offer was out of reach. The three check boxes now send an explicit false when unticked, and regex reads a missing key the way every other kind does. Saving a check through the form without changing what it checks does not create a new version, so the explicit false never turns a rename of a check with reviews into a draft. Every regex metric saved before this release without an explicit case_sensitive now matches case-insensitively, so a rerun or regrade can change its pass rate, and a run comparison can show pass/fail flips with no prompt change. Tick "Case sensitive", or send "case_sensitive": true, to keep the old behaviour.

• case_sensitive, multiline and trim are read as booleans however they were written. The checks compared against the literal boolean, but the REST API and MCP store check_config exactly as posted, so a client that sent "case_sensitive": "true" got the opposite of what it asked for. The string forms, and 1 and 0, are now coerced the way Rails coerces them, so a check saved with a string flag can grade differently on its next run or regrade.

• "Trim whitespace" now works on contains and does-not-contain checks. The form offered it for all three text kinds, but only equals read it. A contains or does-not-contain check saved with trim on now strips whitespace from both the value and the response, so its verdicts can change.

• JSON paths that used to fail every row now resolve. The hint under "Path into the JSON" offers data.items.0.name as an example, but path resolution stopped at the first node that was not an object, so any path stepping into a list returned nothing and every row reported "could not resolve target". Numeric segments now index into a list in target_path, expected_path and json_path alike, and a segment that is out of range or not a number still resolves to nothing rather than to the wrong element. A path may also start at the root with $., as in $.optionCodes, and a bare $ reads the whole document, while a key that itself starts with $, such as $schema, is read as written. A target_path saved with stray whitespace is trimmed, as expected_path already was. A check with such a path now grades the value instead of failing every row.

• A length check with a blank maximum no longer fails every response. A blank bound was read as the number zero, so a blank max failed every response that was not empty, and validation let such a config through whenever no minimum was set, because it only looked for bounds that were nil. The web form drops blank fields, so only checks saved through REST or MCP were affected. A blank bound now counts as absent, a decimal bound is no longer cut to a whole number, and a bound that is not a number is refused when the check is saved.

• A check sent through REST or MCP with a target_path and no target now reads that path. Such a check used to grade the response text and ignore the path, so a check on $.vin.confidence sent with no target failed every row. A target_path with no target now sets target to json_path. A target_path sent with any other target is now refused with an error naming "What to check", where it used to be saved and ignored. A check saved before this release keeps its config until its config is next changed, so a rename never trips either rule.

• The check label constants moved out of ApplicationHelper. CHECK_KIND_LABELS, CHECK_TARGET_LABELS, CHECK_COMPARE_TO_LABELS and CHECK_FIELD_LABELS are gone, and the labels now live in CompletionKit::Checks::Labels as KINDS, TARGETS, COMPARE_TO and FIELDS, with the labels that differ by rule, such as "Shortest allowed" for a length check's min, in FIELDS_BY_KIND. CompletionKit::Checks::Registry::VALUE_COMPARISON_KINDS is also gone; call Registry.comparable_kinds or Registry.compares_value? instead. A host that overrides an engine view and reads one of the old constants raises NameError. Read the new constants, or call ck_check_kind_label, ck_check_target_label, ck_check_field_label and ck_check_field_value, which keep their names. ck_check_field_label now takes the rule as an optional second argument, and without it min and max read "Minimum" and "Maximum".

Added

• A JSON field can now be graded against each row's own answer key. compare_to: "expected" was available on contains, not_contains and equals only, so json_path_equals could compare a field against a constant but never against the ground truth for that row. That is fine when a row has one answer and falls apart when the output is a record. An extraction returning a VIN, a trim, a set of option codes and two colours from one photo had to be pushed as one field per dataset, which multiplies rows by fields and loses the fact that they came from the same photo, or graded on a single field with the rest computed outside CompletionKit and given no history. json_path_equals now accepts compare_to: "expected", with expected_path selecting the matching branch of the answer key, so a record is gradeable field by field with one metric per field over one dataset. The answer key keeps its type, so a number or a boolean in the response matches the same value in the answer key, whether expected_path reads it out of a JSON record or the whole answer key is the bare value, such as 82000 or true. A whole answer key that is not JSON is compared as text.

• A new list_overlap check scores a list-valued field as a fraction rather than a verdict. Every existing check answers yes or no about one value, but a large class of extraction output is a list, and the useful measure over a list is partial. A row expecting six option codes that gets four back was previously called a failure, the same verdict given to a run that returned zero and to one that returned five. list_overlap compares two lists and reports recall, precision, f1 or jaccard, chosen with score_by, which the form calls coverage, accuracy, balance and overlap and describes in one line each. The fraction is stored on the review in a new check_score column, which a review's REST and MCP JSON carries as a number from 0 to 1. Each run averages it per metric as avg_check_score, inside that metric's entry in metric_averages, so a pipeline can gate on it. The run page, the response page and the runs list show a list score as a whole percentage beside its verdict, and colour it by that verdict, as they do every other check. Its min is the fraction that counts as a pass, from 0 to 1, such as 0.6 for 60%, and it defaults to 1, a perfect score. The form says so under "Lowest score that passes". A row with an empty expected list scores 1 on coverage, since nothing was missed, and 0 on the other three measures if the answer returned anything. Two empty lists score 1 on every measure, and an empty answer to a list that is not empty scores 0 on every measure. The expected value can be a JSON list or comma-separated text. Only list_overlap takes a list, so REST and MCP refuse a list value on every other rule rather than saving a check that compares against the list's Ruby text and never matches. A number matches however it is written, whether it comes from a JSON list, a string or text, so 2.0 matches 2 and 4.50 matches 4.5, while a code with a leading zero, such as 01, stays text and does not match 1. This shape is not specific to option codes; tags, entities, features, ingredients, symptoms and citations all extract the same way.

• Two new numeric checks, numeric_bounds and numeric_equals. length_bounds takes a min and a max but measures the length of a string, so there was no way to assert that an extracted number falls in a range or matches the row's expected number within a tolerance. That blocked two things. A per-field confidence could not be bounded, so a grade could not separate right and sure from right by luck, and could say nothing about whether the confidence a ranking rests on is calibrated at all. Extracted figures fared no better: equals on "82,000" against "82000" fails on formatting rather than on being wrong, and a model year out by one was graded the same as one out by forty. Both new kinds read a number, ignoring the thousands commas, spaces and leading currency symbol a figure is written with, including the non-breaking and thin spaces some locales group thousands with. A comma counts as a thousands separator only when three digits follow each comma and no decimal point comes before one, so 82,000.50 reads as 82000.5, while a decimal comma, as in 1,5, 0,500 or €1.234,56, makes the value not a number. A comma followed by exactly three digits after a nonzero group, as in German 2,500, is always read as a thousands separator. numeric_equals takes a tolerance that is a plain number for a fixed amount, or a percentage string such as "2%" for a share of the expected value. The tolerance is compared in exact decimals, so 20.00 against 19.99 with a tolerance of 0.01 passes.

• Run comparisons show how far a list check's fraction moved. A list_overlap metric that rose from 0.50 to 0.83 without reaching its min read as fail to fail, with no sign of change. The comparison page now shows each side's verdict with its score as a whole percentage and the change between them, and marks the row Fixed or Broke beside that change when the verdict flips. Each metric in a REST comparison row carries left_check_score, right_check_score and check_score_delta as fractions.

Fixed

• Saving a json_path_equals check whose expected value is a number or true no longer fails. The edit form read the expected value as JSON twice, and the second read failed on the number the first one produced, so the save returned a server error.

• Editing a check saved through REST or MCP no longer creates a draft you did not ask for. The edit form wrote a list value back as Ruby text, ticked a box whose flag was saved as the string "false", and sent numbers back as text, so renaming such a check with reviews saved a draft, and the ticked box changed what the check did. The form also showed a json_path_equals expected value saved as the text "200" as the number 200, so a save turned it into a number and changed what the check matched. The form now writes a list as JSON, ticks a box only when its flag is on, and shows in quotes any expected value saved as text that would read as JSON, such as 200, true, null or [1], so a save keeps it as text. A save compares what the check does rather than how its config is written, so changing an expected value between "200" and 200 saves a draft on a check with reviews.

• Editing a check validates the new values before saving them. A check with reviews saved any change as a draft without validating it, and a check with none showed an error page on an invalid value and lost what you typed. Both now name the problem in the form and keep your values. A rename is saved in the same step as the config it repairs, so a check whose saved config no longer passes validation can be fixed from the form.

• A check form that fails to save shows the fields of the rule you chose, and marks the ones with errors. After a failed save the form showed every check field under its default label, because the script that picks the fields ran only on a full page load. It now also runs when the failed form is rendered. A check error marked no field either, so focus went to the metric's name. Each field a check error names is now outlined in red and marked invalid for screen readers, and focus moves to the first of them.

• A run that uses only checks no longer needs a judge model. The run form kept Create run disabled until a judge was picked whenever any metric was ticked, so a scoring-only run of checks could not start without one, and the run page then warned that the judge's provider was not configured. The form now asks for a judge only when a ticked metric is an LLM judge and says when a picked judge will go unused, and the run page warns about the judge's provider only when the run has an LLM judge metric.

• A check rule with no minimum or maximum no longer leaves a double gap in the form. The empty row that holds those two fields kept its place in the layout after both were hidden. It is now hidden with them.

• A run with both a judge average and a checks score no longer spills into the runs list's When column. The two chips sat on one line wider than the score column. They now wrap inside it.

• A menu showing a long choice no longer widens a form past a phone screen. The selected choice now shortens with an ellipsis, and the open menu wraps each choice.

• Publishing a check version validates it first. Publishing wrote the version's config onto the live metric without checking it, so an invalid draft could make every future row fail and leave the metric impossible to save. A version that does not pass is now refused with the reason over the web, REST and MCP, and the live metric is left as it was.

• The edit page shows a check's pending draft. It showed the live config under a banner saying it showed the draft, and the banner read "Publish to replace the live for future runs". The form now loads the draft, and the banner says publishing replaces the live check.

• A promptfoo import keeps promptfoo's case handling. promptfoo's contains, not-contains, equals and regex asserts are case-sensitive, but they were imported with no case_sensitive, so the text checks matched case-insensitively. They now import with case_sensitive: true, and icontains and not-icontains with false. Regex checks imported before this release have no case_sensitive, so the regex change above applies to them.

Changed

• Saving a check no longer keeps the fields belonging to a kind you moved away from. The form submits every field it has, visible or not, so switching from a JSON check to a regex check left the old json_path and expected in the saved config, where the metric page then listed them as though the check read them. A saved check now keeps only the keys its own kind uses, plus the shared ones. A hidden "Each row's expected value" choice is dropped the same way, so switching to a rule that cannot use the answer key no longer fails to save.

• The metric page lists a check's fields the way the form does. It now shows them in the form's order and under the form's labels, shows a ticked box as "Yes" rather than a true chip, shows a list as JSON, shows in quotes an expected value saved as text that would read as JSON, and keeps each label on one line.

• Check validation errors name fields the way the form does. They used the stored keys, as in "Check config pattern is not a valid regular expression", and now read like "Check needs "Pattern" to be a valid regular expression" or "Check needs "How far off is still right" to be a number that is not negative, or a percentage like 2%".