Pilot100 Results EN

Progress, completion-report and question hints: first100-case results

한국어 · Feature guide · Previously defined pilot scope

The actual published Go weights processed100 separately authored synthetic development messages. Go v0.2 scored69/100 versus31/100for parent v0.1, but completion reports scored10/25. No weights, temperatures or gates changed, and no new model was selected in this round. Actual Riido reads, installed displays and state changes remain zero.

There are25progress,25completion-report and25question cases, plus five cases for each other intent. Korean and English each have50cases. These are AI-authored fictional messages and meaning annotations, not independent human adjudication or product truth.100distinct family IDs do not prove statistical independence. Rows are not renamed/numbered/translated filler variants.

Comparison on the same100cases

Control	Correct/100	Display proposals	Correct proposals	Wrong proposals
Published Go v0.2	69	16	15	1
Parent Go v0.1	31	7	5	2
Original rules	35	22	19	3
Speech-act rules	44	31	28	3

Models retain stored temperatures, confidence0.9and margin0.05. Rule one-hot scores are not calibrated probabilities. Overall accuracy and proposal coverage are separate findings; proposing more rows does not establish greater accuracy.

V0.2 display intent	Correct classification/25true cases	Correct/created proposals
Progress report	21/25	6/6
Completion report	10/25	3/4
Question	18/25	6/6

Overall accuracy69%, proposal precision15/16and correct-display coverage15/75(20%) have different denominators. Korean scored38/50and English31/50. The previous85%on40synthetic cases did not persist in this100-case test.

A confident but incorrect completion report

The original public synthetic case says:

Do completed checks have to be repeated after only the description changes?

Its authored label is question, but v0.2 proposed completion_report at 0.98825(about98.8%). The parent made the same error. Original text, annotation and failure are retained. completion_report classifies a sentence claiming completion; it proves neither successful execution nor completion of the entire assignment.

Eight-intent NLL improved2.9244→0.9607and Brier1.0299→0.4471. Parent/child temperatures0.7/0.9mean this comparison includes calibration. Full eight-intent probabilities and confusion remain intact; summing the five excluded intents into none is a separate loss diagnostic. Predictions are not renormalized over three intents or replaced by a lower-ranked allowed intent.

A smaller executable producing the same results

Two separate macOS processes repeated the same100cases to compare wrapper costs. All probabilities, metrics, proposals and metadata matched exactly. These repetitions add no independent cases.

One four-control process	General riidolaya	Dedicated riido-statehint-pilot
Binary file bytes	11,875,234(about11.88MB)	4,390,322(about4.39MB)
Peak OS RSS bytes	20,463,616	12,566,528
Go heap sampled before JSON encoding	1,892,600	1,650,904

Each process called two models100times each and two rules100times each. This is one-shot execution including file/model loading, JSON, synthetic catalogs and reporting. Live network reads, long-running operation and GPU are excluded; do not add Go heap to RSS. A narrow classification segment's timing does not establish complete-request throughput.

Use riidolaya state-hint-pilot or the dedicated command, which call the same package. Explicit local weights, checksum/input/budget checks and exclusive report files are required. See commands and options. Actual source text and reports remain private by default.

Source, CI and Hub verification

• Source6a4c4f9 passed Linux/macOS/secrets/quality in CI37422306314. Bot merge e8ffd580 preserves the identical474397f3source tree.
• This study's Go CPU step actually ran100cases with published weights on each OS: 200model calls across both OS jobs. It uses a synthetic catalog, not the Laya backbone, GPU or training. Separate native-inference CI checks are not added to this study's count.
• All14files,845,693bytes, were fully reread and checked by SHA-256 at the pinned HF dataset revision. The pilot100-v1 tag is retained.

CI success proves code/weight compatibility and reproducibility, not operational model quality. The repository README's pending-CI sentence predates execution; this page records the subsequently completed run.

Application boundary and next review

The default reader/catalog/revision1are explicitly synthetic fixtures. Actual native-reader calls and installed labels/emoji/states remain zero. Low-confidence rows retain missing/inactive-label diagnostics using annotation_checked as the denominator. Original reader capability is separated from pilot-disabled state planning; status-candidate interpretation remains unchecked.

Next review expands to at least400different work contexts, including questions mentioning completion, partial completion, execution requests and quotations. Actual source text is private by default; publish only authorized aggregates/summaries or new public synthetic reproductions. Group repeated/translated incidents together instead of counting them independently. These100cases are exposed and cannot become a fresh final test for later model selection.

Complete results · Measurement scope · Frozen plan · Issue155