Skip to content

v1.1.0 — corrected findings, full reproducibility

Latest

Choose a tag to compare

@harukicoder harukicoder released this 17 Sep 22:59
· 1 commit to main since this release

This release corrects claims in v1.0.0 that the data does not support, and makes every published figure reproducible from the script.

Corrections to v1.0.0

  • "No mixed granularities, no silent scheme drift" was false. The collection holds 17 distinct cefr_level strings: the six bands, A1+ through C2+ in three corpora, a bare B, and four separate null markers (EMPTY, NA, Unassessable, unrated). This is now Finding 4.
  • "Three corpora repeat their own texts under conflicting levels" — it is ten. Only cople2_pt and elg_cefr_nl are structural; the rest are a handful of rows. caes_es was wrongly listed among them.
  • "No cross-corpus overlap anywhere in thirteen languages" rested on an approximate sweep that compares only the six languages holding more than one corpus, skips 16.7% of rows under 80 characters, and has about 64% recall at its own threshold. Replaced with an exhaustive inversion over every text.
  • "lang is correct on every row" overstated a metadata-consistency check. No language identifier was run.
  • "673 rows under 20 characters" was 310.
  • 13,611 duplicate rows were presented as a leakage figure; 77% are caes_es short-utterance coincidences that are not a defect.

Reproducibility

Figures cited in v1.0.0 but never emitted — the 130 exact matches, none ×214, the elg_cefr_nl multiplicity, the parquet file count, and "every other field identical" — are now printed by the script. FINDINGS.md states that any figure not in run_output.txt is a defect in the document, and that now holds.

New checks

cefr_level inventory · exhaustive cross-corpus inversion · field identity across repeated rows · short and placeholder rows · parquet file count per dataset · row license field against each dataset card.

New findings

  • The label space is not one scheme — 17 distinct level strings, four different ways of recording no label.
  • The row license field disagrees with the dataset card in 13 of 24 corpora. Six differ substantively, and in three the rows state weaker terms than the card: icle500_en rows say CC0 1.0 (Public Domain) against cc-by-nc-4.0, learn_welsh_cy says public, and zaebuc_ar drops the NC term.

Finding 1 strengthened

Containment now reports injectivity and the similarity distribution, answering the obvious objection that a 0.5 threshold might be matching shared exam boilerplate. It is not: the match is one-to-one onto 324 distinct texts with no collisions, median similarity 0.97, and only 16 of 324 below 0.7. 130 are identical. If only the exact matches are admitted, the finding stands on those alone.

Fixed

The dataset-card comparison reported "0 disagree" when it had reached no cards at all. It now reports unreachable cards and draws no conclusion for them.