Ingot

Construct the label first. Render the sentence second.

The second badge is red on purpose. tests covers code correctness; the model quality gate checks whether the released checkpoint stays inside its declared accuracy budget, and it does not — 4-bit quantization costs 7.5 exact-match points on held-out surfaces against a 2.0 budget. The budget was not relaxed to make it green. See Reading the table.

Ingot builds strictly-labelled training data by running the annotation pipeline backwards. Nothing in it ever looks at a sentence and guesses what the sentence means.

The task is recurring-schedule extraction: a Japanese or English sentence such as 毎月第2水曜10:30から45分、祝日なら前営業日に goes in, and an RFC 5545 object comes out.

{"dtstart":"2026-04-08T10:30:00","tzid":"Asia/Tokyo","rrule":"FREQ=MONTHLY;BYDAY=2WE","duration_minutes":45,"exdate":[],"holiday":{"calendar":"JP","shift":"before"}}

The thesis

The usual way to build a dataset like this is to collect sentences and label them — by hand, by an LLM judge, or by a heuristic annotator. All three produce labels that are estimates, and every downstream number you report is an estimate of an estimate.

Ingot inverts the direction of information flow:

A Constructor turns an integer seed into a typed ScheduleSpec — frequency, interval, weekday set, timezone, holiday policy, exceptions. This object is the label. It is not derived from anything; it is the input to the pipeline.
A Renderer verbalises that object into a sentence, in one of 31 templates across two languages and five registers, plus 14 label-preserving surface-noise operators. An LLM may be used to widen the surface distribution further — but it is shown the sentence, never the label, and every rewording it returns is re-verified blind and discarded if it drifted (ingot/renderer.py).
A Verifier rebuilds the spec from the renderer's privileged slot record and asserts slots_to_spec(slots).key() == spec.key(). If the sentence and the label ever disagree, the sample is discarded, never repaired.

The label therefore cannot be wrong. It can only be unrepresentative — which is a real limitation, and it is the first thing in Threats to validity.

Four claims follow from that construction, and every public function in the package carries a Proves: line naming which one it substantiates:

claim	mechanism	status
label-strictness	the label is the generator's input, not its output	verified: 55,328/55,328 rows round-trip
zero-contamination	disjoint seed bands, disjoint feature combinations, disjoint renderer templates	verified by construction + tests
hard-slice advantage	the constructor dials difficulty directly	measured — and it is the rule baseline, not the specialist, that holds up on `hard`. See the table.
local-speed	0.6B specialist, quantised, on-device	measured on an Apple M2 (see below)
constrained-decoding effect	prefix automaton over the output grammar	verified: free decoding 98.5% syntax vs grammar-constrained 100% on `test`; 1,200 driven generations, 0 errors

Pipeline

flowchart TD
    SEED(["seed : int"]) --> CON

    subgraph INV["inverted generation"]
        direction TB
        CON["<b>Constructor</b><br/>pure function of the seed<br/>emits a typed ScheduleSpec"]
        REN["<b>Renderer</b><br/>verbalises the spec<br/>emits text + privileged slots"]
        CON -- "the label" --> REN
    end

    REN --> VER{"<b>Verifier</b><br/>rebuild the spec from slots<br/>keys equal?"}
    VER -- "no" --> DROP["discard the sample<br/>never repair"]
    VER -- "yes" --> ROW[("dataset row<br/>text + target_json")]

    ROW --> TRAIN["<b>LoRA fine-tune</b><br/>Qwen3-0.6B"]
    TRAIN --> DEC["<b>ConstrainedDecoder</b><br/>prefix DFA over SCHEDULE_REGEX<br/>forced continuations, GBNF, JSON Schema"]
    DEC --> EVAL["<b>Evaluation</b> conditions A to E<br/>exact / core / occurrence match"]

    EVAL --> CUR["<b>AdversarialCurriculum</b><br/>error profile per feature bin<br/>to ConstructorBias"]
    CUR -. "re-weights the weak bins<br/>fresh seed offset, bands never overlap" .-> CON

    classDef stage fill:#eef4ff,stroke:#3b6ea5,stroke-width:1px,color:#10243b;
    classDef gate fill:#fff4e6,stroke:#c07a1a,stroke-width:1px,color:#3b2a10;
    classDef sink fill:#f3f3f3,stroke:#888,stroke-width:1px,color:#333;
    class CON,REN,TRAIN,DEC,EVAL,CUR stage;
    class VER gate;
    class DROP,ROW,SEED sink;

The dotted edge is the only feedback path, and it moves weights, not data: the curriculum tells the constructor which feature bins to oversample next round, and the next round draws from a fresh seed offset inside its own band. No evaluation item can be pulled back into training by the loop.

Contamination-free by construction

Splits are not de-duplicated after the fact. They are addressed differently.

Each split owns a half-open band of one billion seeds. A construction is a pure function of its seed, so two splits cannot produce the same item unless a band boundary was violated — and SeedPolicy raises SeedBandError at construction time if one ever is.

split	seed band (half-open)	trained on	role
`train`	`[0, 1_000_000_000)`	yes	the only band the LoRA ever sees
`val`	`[1_000_000_000, 2_000_000_000)`	no	curriculum error-profiling and early stopping
`test`	`[2_000_000_000, 3_000_000_000)`	no	in-distribution held-out accuracy
`unseen_combo`	`[3_000_000_000, 4_000_000_000)`	no	only the reserved feature combinations
`unseen_template`	`[6_000_000_000, 7_000_000_000)`	no	only the reserved renderer templates
`paraphrase`	`[4_000_000_000, 5_000_000_000)`	no	blind anchor-based verification, not slot-based
`probe`	`[5_000_000_000, 6_000_000_000)`	no	few-shot demonstrations for condition (C)

unseen_combo is the stronger guarantee. Eight (freq, byday_kind, end_kind, holiday_kind) tuples are reserved and the train/val/test generators refuse to emit them:

('DAILY',   'none',   'until', 'skip')     ('YEARLY',  'nth',    'until', 'after')
('WEEKLY',  'multi',  'until', 'before')   ('YEARLY',  'single', 'count', 'before')
('MONTHLY', 'last',   'until', 'after')    ('WEEKLY',  'single', 'until', 'skip')
('MONTHLY', 'setpos', 'count', 'skip')     ('DAILY',   'none',   'count', 'after')

An unseen_combo item is therefore not merely an unseen string. It is structurally unreachable from the training generator. tests/test_no_contamination.py asserts both directions: train/val/test emit zero held-out combos, and unseen_combo emits only them.

unseen_template holds out the surface rather than the label. Eight of the 31 renderer templates — spanning both languages and four registers — are reserved for it and withheld from every other split, training included:

ja_polite_notice   ja_casual_fullwidth   ja_terse_kv     ja_email_reminder
en_polite_invite   en_casual_dm          en_terse_compact  en_bullet_fields

This split exists because of a measurement, not a hunch. See What the rule-based baseline actually scores.

Note the asymmetry, and do not read past it: the rule-based baseline in condition (A) was written with all 31 templates visible, so it has effectively seen these surfaces while the model has not. The comparison is biased against the specialist, which is the safe direction.

Difficulty rubric

Difficulty is a declared, additive score over the feature axes — not a model's opinion of what was hard. ingot.tasks.chrono.features.rubric_markdown() is the source of truth and prints:

axis	value	points
`freq`	`DAILY`	0
`freq`	`WEEKLY`	0
`freq`	`MONTHLY`	1
`freq`	`YEARLY`	2
`interval_kind`	`1`	0
`interval_kind`	`2`	1
`interval_kind`	`3plus`	1
`byday_kind`	`none`	0
`byday_kind`	`single`	0
`byday_kind`	`multi`	1
`byday_kind`	`nth`	2
`byday_kind`	`last`	2
`byday_kind`	`setpos`	3
`monthday_kind`	`none`	0
`monthday_kind`	`fixed`	1
`monthday_kind`	`last`	2
`end_kind`	`never`	0
`end_kind`	`count`	0
`end_kind`	`until`	2
`dst_kind`	`none`	0
`dst_kind`	`crosses`	4
`holiday_kind`	`none`	0
`holiday_kind`	`before`	3
`holiday_kind`	`after`	3
`holiday_kind`	`skip`	3
`exdate_kind`	`none`	0
`exdate_kind`	`some`	2
`style_kind`	`polite`	0
`style_kind`	`casual`	0
`style_kind`	`terse`	0
`style_kind`	`bullet`	1
`style_kind`	`email`	1
`relative_ref`	`no`	0
`relative_ref`	`yes`	1

easy = score ≤ 1 · medium = 2–3 · hard = score ≥ 4

Because the rubric is fixed before any model is trained, "the specialist wins on the hard slice" is a falsifiable statement about a pre-registered subset, not a slice chosen after seeing results.

Measured difficulty distribution

At default constructor settings (uniform bias, lang_ratio=0.6, year_range=(2026, 2028)), over the first 2000 seeds of each band:

split	easy	medium	hard	ja	en
`train`	21.2%	28.3%	50.5%	60.2%	39.8%
`val`	21.7%	28.5%	49.9%	60.9%	39.2%
`test`	20.5%	29.2%	50.4%	58.2%	41.9%
`unseen_combo`	0.0%	5.7%	94.3%	62.1%	38.0%

unseen_combo is nearly all hard by construction — the reserved combos are built from the high-scoring axes. That is a design property, not a result, and it means an unseen_combo accuracy number is meaningless unless this distribution is printed next to it. See Threats to validity.

Worked examples

Real output of construct -> render -> verify, not illustrations:

seed 7 · split=train · lang=en · template=en_casual_chat · hard · verified
  text   ok so from Oct 17, the backup job is every 3 weeks on Tue, Thu, Fri, Sat and Sun
         at 9am, about 30 minutes
  ref    2026-10-17 · default_tz America/Los_Angeles
  label  {"dtstart":"2026-10-17T09:00:00","tzid":"America/Los_Angeles",
          "rrule":"FREQ=WEEKLY;INTERVAL=3;BYDAY=TU,TH,FR,SA,SU","duration_minutes":30,
          "exdate":[],"holiday":{"calendar":null,"shift":"none"}}
  next   2026-10-17T09:00:00-07:00 · 2026-10-18T09:00:00-07:00 · 2026-11-03T09:00:00-08:00

Note the third occurrence: the series crosses the US DST transition and the UTC offset changes from -07:00 to -08:00 while the wall-clock time stays at 09:00. The label is correct there because expand computed it from the spec, not because anyone checked.

seed 3 · split=train · lang=ja · template=ja_polite_wareki · easy · verified
  text   令和９年１０月８日から毎日午後６時３０分より１時間、読書会を執り行います。
  ref    2027-10-08 · default_tz Asia/Tokyo
  label  {"dtstart":"2027-10-08T18:30:00","tzid":"Asia/Tokyo","rrule":"FREQ=DAILY",
          "duration_minutes":60,"exdate":[],"holiday":{"calendar":null,"shift":"none"}}

That sample carries a Japanese-era date (令和９年 = 2027) and full-width digits, both applied by label-preserving noise operators. The label was never re-read off the text.

seed 6000000004 · split=unseen_template · lang=en · template=en_terse_compact · hard · verified
  text   freq=the third weekday of each month; time=14:30; len=15 minutes;
         start=2027-06-03; end=3 times; tz=Paris time
  ref    2027-05-15 · default_tz America/New_York
  label  {"dtstart":"2027-06-03T14:30:00","tzid":"Europe/Paris",
          "rrule":"FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=3;COUNT=3",
          "duration_minutes":15,"exdate":[],"holiday":{"calendar":null,"shift":"none"}}
  next   2027-06-03T14:30:00+02:00 · 2027-07-05T14:30:00+02:00 · 2027-08-04T14:30:00+02:00

Three things are happening at once in that last one, and all three are label content rather than wording: BYSETPOS=3 over the weekday set is "the third business day", the text overrides the prompt's default timezone (America/New_York) with Europe/Paris, and it comes from a template the model never saw in training. Reproduce any of them with ingot render --seed <n> --split <split>.

What the rule-based baseline actually scores

This is the most important number in the repository, so it goes above the results table rather than in a footnote.

A blind, hand-written dateutil parser reaches ~90% exact match on Ingot's data. Measured, on the full splits, with no model involved:

split	n	exact match	ja	en
`test`	1,998	90.09%	88.5%	92.4%
`unseen_template`	2,000	91.40%	89.6%	94.1%
`unseen_combo`	2,000	81.95%	73.2%	96.2%

ingot/tasks/chrono/baseline_rule.py imports only re, dateutil and zoneinfo. It has no access to the renderer, its templates, or its slots. It is 1,627 lines of honest pattern matching, and on template-rendered surfaces it is very hard to beat.

Three things follow, and all three shape how the rest of this README should be read.

"The specialist beats hand-written rules" is not a claim test can support, and this project does not make it there. When the surfaces are themselves rule-generated, a sufficiently engineered parser inverts them. That is a property of template rendering, not a flaw in the parser or in the model.
Holding surfaces out does not fix it. unseen_template was added specifically to test this, and the baseline scored higher there (91.4%) than on test. A pattern-based parser does not care which template emitted a weekday token. That is a negative result, and it is reported as one.
Where rules genuinely break is Japanese with unseen feature compositions: 73.2% on unseen_combo against 96.2% for English on the same split — a 23-point language gap that does not appear anywhere else. That slice, and local latency, are where a specialist has something to prove.

The claims this repository does support without qualification are the structural ones: labels are strict by construction, splits are contamination-free by construction, and the model runs locally at measured speed.

Evaluation

Conditions (A), (D) and (E) were run. Conditions (B) and (C) cannot be run on this machine — there is no Anthropic credential — so they read n/a (not run) with the reason inline. They are not scored zero and not estimated.

Every cell below comes from benchmarks/results/summary.json, regenerated by python3 scripts/update_readme.py. Raw model outputs are kept in benchmarks/results/raw/ so any metric can be recomputed from the evidence rather than trusted.

The five conditions, all given identical prompts from ingot.prompts (same system prompt, same reference date, same default timezone — so a difference between conditions is a difference in capability, not in prompt engineering):

cond	system	where it is defined
A	hand-written `dateutil` rule parser	`ingot/tasks/chrono/baseline_rule.py`
B	frontier API, zero-shot, JSON-Schema constrained	`grammar.SCHEDULE_JSON_SCHEMA` via `output_config.format`
C	frontier API, few-shot (demos drawn from the `probe` band)	`prompts.build_fewshot_messages`
D	Ingot LoRA on Qwen3-0.6B, local	`scripts/train_lora.py`
E	the same adapter quantised to GGUF, GBNF-constrained	`decode.llama_cpp_grammar()`

Condition (A) is written in good faith on purpose. A straw-man baseline would make the headline result worthless; see the module docstring, which documents every ambiguity it guesses at and why it guesses that way.

test (n=200)

cond	system	n	exact % (95% CI)	core %	occ-exact %	syntax %	strict %	hard exact %	ja %	en %	median s
A	Rule-based (dateutil)	200	90.5 (85.6–93.8)	94.5	92.0	100.0	100.0	89.3	89.2	92.1	0.001
B	Frontier API (prompt) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
C	Frontier API (few-shot) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
D	Ingot SLM (bf16)	200	67.5 (60.7–73.6)	69.5	83.5	98.5	99.0	62.1	71.2	62.9	1.363
E	Ingot SLM (Q4)	200	65.5 (58.7–71.7)	70.0	84.5	100.0	100.0	57.3	71.2	58.4	1.842

Quantization cost (D to E): +2.00 exact-match points.

McNemar on the hard slice: A_vs_D_hard p=2.54e-05, A_vs_E_hard p=1.07e-06, D_vs_E_hard p=0.267.

unseen_template (n=200)

cond	system	n	exact % (95% CI)	core %	occ-exact %	syntax %	strict %	hard exact %	ja %	en %	median s
A	Rule-based (dateutil)	200	90.0 (85.1–93.4)	95.0	91.0	100.0	100.0	85.6	87.3	95.5	0.000
B	Frontier API (prompt) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
C	Frontier API (few-shot) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
D	Ingot SLM (bf16)	200	67.5 (60.7–73.6)	70.5	85.5	100.0	100.0	55.6	73.1	56.1	1.791
E	Ingot SLM (Q4)	200	60.0 (53.1–66.5)	63.0	79.0	100.0	100.0	51.1	64.2	51.5	2.083

Quantization cost (D to E): +7.50 exact-match points.

McNemar on the hard slice: A_vs_D_hard p=2.53e-05, A_vs_E_hard p=3.12e-06, D_vs_E_hard p=0.344.

unseen_combo (n=200)

cond	system	n	exact % (95% CI)	core %	occ-exact %	syntax %	strict %	hard exact %	ja %	en %	median s
A	Rule-based (dateutil)	200	77.0 (70.7–82.3)	93.5	77.5	100.0	100.0	75.5	64.0	98.7	0.001
B	Frontier API (prompt) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
C	Frontier API (few-shot) — not run	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)	n/a (not run)
D	Ingot SLM (bf16)	200	47.5 (40.7–54.4)	49.0	79.5	100.0	100.0	44.7	52.8	38.7	1.653
E	Ingot SLM (Q4)	200	44.0 (37.3–50.9)	45.5	77.0	100.0	100.0	41.5	48.8	36.0	2.405

Quantization cost (D to E): +3.50 exact-match points.

McNemar on the hard slice: A_vs_D_hard p=2.87e-09, A_vs_E_hard p=1.64e-10, D_vs_E_hard p=0.392.

Reading the table

The rule-based baseline wins on every split. That is the result, and it is not a bug in the harness. Three things explain it, in descending order of importance:

The surfaces are rule-generated, so a rule parser can invert them — the point argued at length above. This ceiling does not move with more training.
The checkpoint is undertrained by design: 1,000 steps, ~8% of one epoch, stopped so the GPU could run the evaluation inside one session. Validation loss was still falling.
n = 200 per split, so the 95% intervals are wide (±7 points). Differences under ~10 points between (D) and (E) are not resolvable at this sample size, and are not claimed to be.

Where the model does hold up is the semantic metric. On test, condition (D) scores 67.5% exact but 83.5% occurrence-exact — that is, in a sixth of cases it writes a rule that is not byte-identical to the gold RRULE yet produces the same next ten meetings. Exact match is the strict metric; occurrence match is the one a calendar user would feel. Both are reported because they disagree, and the disagreement is informative rather than embarrassing.

Quantization is not free, and it is least free out of distribution. Measured cost of 4-bit:

split	(D) bf16	(E) Q4_K_M	drop
`test`	67.5%	65.5%	2.0
`unseen_combo`	47.5%	44.0%	3.5
`unseen_template`	67.5%	60.0%	7.5

That ordering is suggestive — 4-bit appears to cost three times more on held-out surfaces than in-distribution — but it does not survive significance testing. McNemar on the paired hard items gives D_vs_E p = 0.267 on test and p = 0.344 on unseen_template: at n=200 the quantization gap is indistinguishable from sampling noise. The budget test still fires, because a budget is an engineering ceiling rather than a significance claim, but do not read the 7.5-point figure as an established effect. It is a reason to re-measure at larger n, nothing more.

By contrast, the gap that is significant is the one that argues against the specialist: A_vs_D on the hard slice gives p = 2.5e-05 on test and p = 2.5e-05 on unseen_template. The rule baseline is genuinely, not accidentally, ahead.

One inversion worth noticing. On unseen_combo, the two systems fail on opposite languages:

system	ja	en
(A) rule-based	64.0%	98.7%
(D) Ingot SLM	52.8%	38.7%

The hand-written parser is near-perfect on English novel compositions and loses 35 points on Japanese; the model is the other way round. That is the one place in these results where the two approaches look complementary rather than ranked, and it is the slice a larger run should target first.

tests/test_quantization.py currently FAILS, and that is intentional. The declared budget is 2.0 absolute exact-match points (INGOT_QUANT_MAX_DROP); unseen_template costs 7.5 and unseen_combo 3.5, so the suite goes red with an explicit message naming both. The budget was not relaxed to make it pass — moving a threshold to accommodate a measurement is the exact failure mode the test exists to prevent. Fix the model, or change the budget deliberately and say so.

Constrained decoding does what it claims — and measuring it found a real bug. Condition (D) decodes freely and reaches 98.5% syntax validity on test; condition (E) decodes under the GBNF grammar. Driving the prefix automaton directly for 1,200 adversarial generations produces zero syntax errors.

On the first run, (E) scored 99.5% rather than 100%, which should be impossible under a grammar. Rather than round it off, the single failing output was pulled out of benchmarks/results/raw/:

FREQ=WEEKLY;INTERVAL=2;BYDAY=1TU

An ordinal weekday ("the first Tuesday") is meaningful only under MONTHLY/YEARLY; RFC 5545 forbids it under WEEKLY, and ScheduleSpec.validate rejects it. The decoder's automaton already enforced that coupling — but SCHEDULE_REGEX, SCHEDULE_GBNF and SCHEDULE_JSON_SCHEMA did not, so the GGUF path was constrained by a grammar strictly looser than the validator. Two further mismatches surfaced from the same audit: the holiday calendar/shift pair was uncoupled, and duration_minutes allowed five digits in the regex against four in the schema.

All four dialects now describe one language, and tests/test_constrained_decoding.py pins the invariant that matters: anything the grammar admits must survive full spec validation. A grammar that admits strings its own validator rejects is not a constraint, it is a suggestion.

This is what the harness is for. The bug was invisible to inspection and visible to measurement, and it was found because raw outputs are kept rather than summarised away.

One limit survives the fix and is documented rather than papered over. A regular grammar constrains shape, and "the day is in range for this month" is not a shape property: 2026-02-30 is a well-formed ISO string and is not a date. No regex, GBNF or JSON Schema pattern can exclude it, and condition (D) emitted one. So the two syntax columns measure genuinely different things — strict % asks "does it match the canonical grammar?", syntax % asks "is it a real schedule?" — and strict can exceed syntax by a hair. Both are reported. What must never happen is the reverse, and ScheduleSpec.from_json is now total: every malformed input raises SpecError rather than leaking a ValueError into a scoring loop, where it would silently truncate a run.

scripts/run_eval.py regenerates the block between those two markers from metrics.results_table_markdown(...). If you are reading a copy of this file where the cells are still n/a (not run), that is the honest state of the repository, not a rendering failure.

Conditions (B) and (C) cannot be run on the development machine at all. There is no ANTHROPIC_API_KEY and no API client configured here. scripts/run_eval.py therefore reports them as unavailable and falls back to a cache file rather than crashing — and an unavailable condition is never silently replaced by a different one. Anyone with a key can fill those two rows in; until then they are blank, and the "specialist beats the frontier model on the hard slice" claim is unsubstantiated in this repository.

What has been measured

All measured on an Apple M2, 16 GB, no discrete GPU. Throughput and footprint only — these say nothing about whether the specialist is correct, which is what the table above is for.

End-to-end, on the real evaluation prompt (~250 prompt tokens, ~90 generated), 25 requests after 3 warmups, greedy decoding, batch size 1 (scripts/bench_local.py):

artifact	size on disk	generation	time to first token	mean request	peak RSS
MLX 4-bit	335 MB	104.9 tok/s	279 ms	0.76 s	946 MiB
GGUF Q4_K_M (GBNF-constrained)	378 MB	96.4 tok/s	57 ms	0.81 s	1632 MiB
GGUF Q8_0	610 MB	not benchmarked	—	—	—
fused bf16	1.1 GB	58 tok/s¹	—	—	—

¹ The bf16 figure is from a short-prompt microbenchmark and is not directly comparable to the two rows above. Tokens/sec counts generated tokens over generation wall time only; counting prompt tokens would flatter a local model with a long prompt, which is the dishonest direction.

An earlier draft of this file quoted 149 tok/s for MLX 4-bit. That number came from a 16-token toy prompt and does not survive contact with the real ~250-token prompt. The honest figure is 104.9 tok/s. It is left visible here rather than quietly edited out.

activity	measurement
`mlx_lm` LoRA training	0.44 it/s at batch 4 / 16 layers / seq 448 with gradient checkpointing, 3.6 GB peak; 1,000 steps ≈ 38 min
round-trip verification	15,000/15,000 (100%) in the test sweep (3,000 per split); 55,328/55,328 (100%) in the published corpus
constrained decoding	1,200 adversarially-driven generations, 0 syntax errors; ~65% of output characters are forced by the grammar and never sampled

The point of the local-speed claim is the pairing: a 335 MB artifact answering in under a second, on a laptop, with no network. A 57 ms time-to-first-token is the number that matters for interactive use, and it belongs to the quantised, grammar-constrained configuration.

Figures

scripts/make_figures.py reads only benchmarks/results/*.json and skips any figure whose inputs are missing, printing which one it skipped and why:

figure	shows	needs
`figures/hard_slice.png`	exact match by difficulty, conditions A–E, Wilson 95% CI	`summary.json`
`figures/split_generalisation.png`	`test` vs `unseen_template` vs `unseen_combo`	`summary.json` with ≥2 splits
`figures/throughput.png`	tok/s and time-to-first-token, local vs API	`latency.json` (`scripts/bench_local.py`)
`figures/syntax_validity.png`	constrained vs unconstrained syntax-error rate	`summary.json` or `constrained_decoding.json`
`figures/ja_en.png`	exact match by language	`summary.json`
`figures/curriculum.png`	per-round error rate on the targeted weak bins	≥2 `loop_round_*.json`

Placeholder data is never plotted. PNGs are not committed — they are a view of benchmarks/results/, and a committed PNG can silently outlive the numbers it was drawn from. Regenerate with ingot figures.

Published artifacts

what	where
Dataset	`NagaYu/ingot-chrono` — 55,328 rows, six splits, 100% round-trip verified
Model	`NagaYu/ingot-chrono-qwen3-0.6b` — MLX 4-bit + GGUF Q4_K_M + the raw LoRA
Code	this repository

from datasets import load_dataset
ds = load_dataset("NagaYu/ingot-chrono")          # train / val / test / unseen_combo / unseen_template / probe

from mlx_lm import load, generate
model, tokenizer = load("NagaYu/ingot-chrono-qwen3-0.6b")   # remember enable_thinking=False

Use the model card's warning, not just the model. This checkpoint is a reference point for the pipeline, not a recommended parser: it is undertrained by design (1,000 steps, ~8% of one epoch) and the hand-written dateutil baseline beats it on every split. If your inputs look like this generator's output, use baseline_rule.py instead.

Only the quantised builds are uploaded. The bf16 fused checkpoint (1.1 GB) and the Q8_0 GGUF are reconstructible from the shipped adapter in about a minute, so publishing them would be 1.7 GB of duplication:

python -m mlx_lm fuse --model Qwen/Qwen3-0.6B --adapter-path adapter --save-path fused
python3 scripts/train_lora.py --base Qwen/Qwen3-0.6B --iters 6000   # train it properly: ~2.5 h on an M2

The dataset remains the artifact worth having: its labels are strict by construction and its splits are contamination-free by construction, and neither claim depends on the model.

Quickstart

pip install ingot            # core: python-dateutil only
pip install "ingot[all]"     # + data, train, mlx, local, constrain, api, app, figures, dev

Generate a contamination-free dataset. assert_disjoint runs before anything is written, so a band collision fails the build instead of poisoning it:

# One command builds every split. Sizes are per-split flags, and `assert_disjoint` runs first.
ingot generate --task chrono --n 24000 --variants 2 --out data/chrono
ingot verify   --n 5000      # re-run the round-trip audit yourself, on every split

Optionally widen the surface distribution with an LLM. The constructed label is never sent to the model, and every rewording is re-verified blind against it and discarded if it drifted:

ingot generate --task chrono --n 24000 --llm-diversify 5000   # needs ANTHROPIC_API_KEY

Inspect a single item before you trust 50 000 of them:

ingot render --seed 7 --split train --variants 5

Fine-tune. --backend auto picks mlx on Apple Silicon and trl elsewhere:

ingot train --base Qwen/Qwen3-0.6B --backend auto --data data/chrono \
            --iters 2000 --batch-size 2 --num-layers 8 --out runs/chrono-v1

Evaluate and benchmark:

ingot eval  --conditions A,D,E --split test,unseen_combo,unseen_template --n 500 \
            --model runs/chrono-v1/fused --quantized runs/chrono-v1/mlx-q4
python3 scripts/update_readme.py     # rewrites the results table from measured data only
ingot bench --model runs/chrono-v1/mlx-q4 --backend mlx --n 50

One-off inference:

ingot parse --text "隔週水曜9時から30分、5回まで" --model runs/chrono-v1/mlx-q4 \
            --ref 2026-04-01 --tz Asia/Tokyo

Run the adversarial curriculum loop (eval on val, re-weight the constructor, regenerate from a fresh seed offset, retrain, re-eval):

ingot loop --rounds 3 --n-per-round 10000

Note the generation step is the only one that can touch the network, and only with --push-to-hub, which is off by default.

Extending Ingot to a new task

The full recipe is in docs/EXTENDING.md. The short version:

Ingot's spine — seed bands, feature tagging, difficulty scoring, constrained decoding, metrics, curriculum — is task-agnostic. A new task is three functions and a type. Chrono is the reference implementation; read it as a worked example, not as a framework you must subclass.

Create ingot/tasks/<yourtask>/ and supply:

1. The label type. A frozen dataclass with a canonical serialisation and a key() used for equality. key() is what makes verification and exact-match scoring the same predicate.

@dataclass(frozen=True)
class YourSpec:
    ...
    def to_json(self) -> str: ...
    @classmethod
    def from_json(cls, raw: str) -> "YourSpec": ...
    def key(self) -> tuple: ...      # canonical identity; two specs are equal iff keys are

2. The Constructor. A pure function of an integer seed — random.Random(seed) only, never the module-level random, never datetime.now(). Determinism is what makes seed bands mean anything; a constructor that reads the clock silently destroys the contamination guarantee.

class YourConstructor:
    def __init__(self, split: str, bias: ConstructorBias | None = None): ...
    def construct(self, seed: int) -> Construction: ...   # -> your spec + context

Refuse to emit your reserved feature combinations unless split == "unseen_combo". That single if is the entire zero-contamination mechanism.

3. The Renderer. Spec to sentence, returning (text, slots) where slots is your privileged record of every semantic value you verbalised. The slots dict is a contract: the verifier must be able to reconstruct the spec from it without reading the text.

class YourRenderer:
    def render(self, c: Construction, seed: int | None = None) -> Rendering: ...

4. The Verifier. Rebuild the spec from the slots and compare keys.

def slots_to_spec(slots: dict) -> YourSpec: ...

def verify_rendering(rendering, construction) -> VerifyResult:
    return VerifyResult(ok=slots_to_spec(rendering.slots).key() == construction.spec.key(),
                        mode="slots")

If that assertion can pass while the sentence is wrong, your slots dict is too weak — it is recording what you meant rather than what you wrote. The chrono renderer's until_local field is the canonical example: it stores local wall time, so a bug in the renderer's UTC conversion is caught by the verifier instead of being absorbed into a self-consistent lie.

Everything else you get for free: ingot.seeds for bands, features.tag/rubric_markdown for difficulty (supply your own WEIGHTS), ingot.decode for grammar-constrained generation, ingot.metrics for scoring, ingot.curriculum for the adversarial loop.

Threats to validity

Written to be usable against the project, not to reassure.

The renderer is rule-based, so the task is in principle invertible. Ingot's sentences are generated by 31 templates plus 14 surface-noise operators. A sufficiently determined engineer could write a parser that inverts that generator and scores very well — the mapping is, after all, a program. This bounds what a good score on Ingot means. It does not mean the data is worthless, because the templates and noise operators are held constant across all five conditions, and because condition (A) is a genuine attempt at exactly that inversion, written before any model was trained. But a reader should treat "system X scores well on Ingot" as evidence about this generator's surface distribution, not as a measurement of open-world Japanese and English scheduling language. Real-world transfer is untested and is not claimed.

The round-trip verifier is privileged, and is never used to score a model. verify_rendering reads Rendering.slots — the renderer's internal record — which no model ever sees. It is a data-quality gate at generation time, and using it to grade a prediction would be circular. Model predictions are scored only by ingot.metrics.score_prediction, which compares the predicted JSON against the gold spec and against the gold occurrence list. The paraphrase split exists precisely because slot-based verification is unavailable there; it uses blind anchor-based verification (verify_paraphrase) instead, which is weaker and is labelled as such in the data (verify_mode).

unseen_combo skews hard by construction, so its accuracy is uninterpretable alone. Measured above: 94.3% of unseen_combo items are hard and none are easy, against roughly 50% hard on test. A model that scores lower on unseen_combo than on test may be failing at generalisation, or may simply be meeting harder items. Any unseen_combo accuracy figure must be reported alongside its difficulty distribution, and compared against the hard slice of test rather than against test overall. metrics.aggregate returns by_difficulty for exactly this reason.

Difficulty is declared, not discovered. The rubric above was written from a human's intuition about what is hard in RFC 5545, then frozen. It is pre-registered, which prevents post-hoc slice selection, but "hard" means "scores high on this rubric", not "empirically hard for models". If a model finds dst_kind=crosses (4 points) easy and interval_kind=2 (1 point) hard, the rubric is wrong and the hard-slice claim is measuring the wrong thing.

Round-trip verification proves consistency, not coverage. 15,000/15,000 says every rendered sentence maps back to its label. It says nothing about whether those 8000 sentences resemble anything a human would write, or whether the constructor's distribution over specs resembles the distribution of real schedules. Both are open.

The published checkpoint is undertrained, and its numbers are a floor rather than a ceiling. Training was stopped at 1,000 optimiser steps — about 4,000 samples of the 48,000 available, on the order of 8% of one epoch — because the run was proceeding at 0.44 it/s on an M2 and the GPU was needed for evaluation. Validation loss was still falling monotonically at the stop point (0.042 at step 200, 0.024 at 400, 0.013 at 800). Read condition (D)/(E) accuracy as "what a 0.6B model reaches after twenty minutes of laptop training", not as what the method converges to. Re-run scripts/train_lora.py --iters 6000 to push it; the harness is unchanged.

Nothing about accuracy has been measured yet. Claims of a hard-slice advantage and of a constrained-decoding effect are, in this repository, hypotheses with a harness attached. The frontier-API conditions cannot be run here at all. Until benchmarks/results/ is populated the only demonstrated claims are label-strictness, zero-contamination, and local speed.

Repository layout

ingot/
  constructor.py      Constructor/Verifier protocols, Task registry   <- core #1 (generic)
  renderer.py         Renderer protocol + LLMSurfaceDiversifier       <- core #2 (generic)
  curriculum.py       ErrorProfile -> ConstructorBias                 <- core #3
  decode.py           TemplateConstrainedSampler, GBNF, logits masks  <- core #4
  seeds.py            seed bands, SeedPolicy, assert_disjoint
  prompts.py          the one prompt definition, shared by every condition
  dataset.py          verified corpus assembly, collision checks, manifest
  evaluate.py         the five-condition harness, honest not_run handling
  metrics.py          exact / core / occurrence match, Wilson CIs, McNemar
  cli.py              the `ingot` command
  tasks/chrono/
    spec.py           ScheduleSpec (the label type), RRULE canonicalisation
    constructor.py    seed -> spec
    renderer.py       spec -> (text, slots); 31 templates, 8 held out
    surface.py        lexicons, wareki, full-width, label-preserving noise ops
    verify.py         slots -> spec (privileged) and blind anchor verification
    anchors.py        blind surface-evidence extraction (ja + en)
    features.py       feature tagging, difficulty rubric, held-out combos
    expand.py         spec -> real datetimes (DST gap/fold policy, holiday shifting)
    holidays.py       JP / US calendars, 振替休日 / 国民の休日 / observed
    grammar.py        SCHEDULE_REGEX, SCHEDULE_JSON_SCHEMA, SCHEDULE_GBNF
    runner.py         MLX / GGUF / transformers inference, one timing convention
    baseline_rule.py  condition (A)
    baseline_api.py   conditions (B) and (C)
scripts/              build_dataset, train_lora, merge_and_quantize, run_eval,
                      adversarial_loop, bench_local, make_figures, update_readme
benchmarks/results/   evaluation JSON + raw/ model outputs (the evidence)
docs/                 EXTENDING.md, dataset_card.md, model_card.md
.github/workflows/    CI: import purity, round-trip sweep, end-to-end corpus build
tests/                round-trip, contamination, constrained decoding, quantization budget

Cards

docs/model_card.md — the model card, published verbatim as the README of NagaYu/ingot-chrono-qwen3-0.6b. It leads with the fact that a hand-written parser beats the model, rather than burying it.
docs/dataset_card.md — the dataset card template.
data/chrono/dataset_card.md — the generated card for the build actually on disk, written by scripts/build_dataset.py. It carries that build's real seed bands, verification rates and difficulty distribution, so it describes the data you have rather than the data in general.
data/chrono/manifest.json — the machine-readable version of the same provenance.

Both cards carry Hugging Face card frontmatter. This README deliberately does not: GitHub renders YAML frontmatter as literal text, so carrying it here would put a broken-looking metadata dump above the title for every visitor. The Hub-facing copy lives in docs/_hf_readme_frontmatter.yml if this file is ever used as a model card.

Both the dataset and the quantised model are published (see Published artifacts). --push-to-hub is off by default on every script, so nothing leaves your machine unless you ask for it.

License

Apache-2.0. See LICENSE.

The chrono task generates entirely synthetic text from templates; it contains no scraped, user-contributed, or personal data.

Navigation Menu

Repository files navigation

Ingot

The thesis

Pipeline

Contamination-free by construction

Difficulty rubric

Measured difficulty distribution

Worked examples

What the rule-based baseline actually scores

Evaluation

Reading the table

What has been measured

Figures

Published artifacts

Quickstart

Extending Ingot to a new task

Threats to validity

Repository layout

Cards

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
benchmarks		benchmarks
data		data
docs		docs
figures		figures
ingot		ingot
scripts		scripts
tests		tests
.gitignore		.gitignore
CITATION.cff		CITATION.cff
LICENSE		LICENSE
README.md		README.md
app.py		app.py
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation

Ingot

The thesis

Pipeline

Contamination-free by construction

Difficulty rubric

Measured difficulty distribution

Worked examples

What the rule-based baseline actually scores

Evaluation

Reading the table

What has been measured

Figures

Published artifacts

Quickstart

Extending Ingot to a new task

Threats to validity

Repository layout

Cards

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages