UnicodeTestSuite v3.0

A correctness release. v2.0's three central promises — cryptographic integrity, a machine-parseable filename contract, and usable detection ground truth — each had a hole in it. This release closes them, and adds the long-form and line-ending coverage the corpus needed to discriminate between detectors rather than defeat all of them equally.

Not backward compatible. Manifest.csv gains columns, two files are renamed, and several documents change content, so hashes move. Read ManifestVersion.txt (or the meta table in Manifest.sqlite) to tell a v2 corpus from a v3 one.

Downloads

Asset	Contents
UnicodeTestSuite-v3.0.zip	The generated corpus — 1,359 files plus manifests, hashes and certificate
UnicodeTestSuiteGenerator-v3.0.zip	The generator, its tests, and the vendored long-form source text

Both archives are reproducible: rebuilding from the same tree yields byte-identical zips.

Verify what you downloaded

python GenerateCorpus.py verify --corpus /path/to/UnicodeTestSuite

This re-checks the metadata files against CorpusCertificate.txt, then every file against Manifest.csv, re-decoding each under its declared encoding, and fails if any file is present on disk but absent from the manifest.

Highlights

verify no longer passes on a tampered corpus. In v2.0, MasterHashes.sha256 covered 1,212 of the 1,222 files a corpus contained — the manifests, the index, the certificate and the reference files were all outside it. Editing a reference file, corrupting Manifest.sqlite, or planting an unlisted file were each reported as All 1,212 files verified OK. All three now fail, and CI replays them on every push.

Ground truth is a set. 300 of v2.0's text files were byte-identical to a file carrying a different declared encoding, and 27 byte sequences appeared twice under contradictory labels. A detector answering ascii for pure-ASCII content — the more precise answer — scored 65 false negatives for being right. The new AlsoValidAs column lists every encoding that decodes a file's bytes to the same characters. Score set membership, not string equality.

Samples a detector can actually classify. v2.0's legacy files had a median length of 31 bytes. 15_LongForm/ adds multi-kilobyte natural-language text in 29 encodings, each document emitted only into encodings that historically carried its language. Measured against a real detector, legacy accuracy on long-form samples is 92.2%, against 72.2% on the short ones.

CR and CRLF beyond UTF-8. v2.0 had no CRLF file in UTF-16, UTF-32 or any legacy code page, leaving 0D 00 0A 00 — the most common byte pattern in real Windows text — unrepresented. Coverage goes from 1 encoding to 12.

generate no longer deletes without asking, and --corpus DIR lets both subcommands target any directory.

Statistics

Item	v2.0	v3.0
Total generated files	1,212	1,359
Canonical documents	94	94
Long-form documents	0	24
Root folders	15	16
Distinct encodings	35	35
Legacy sample maximum	125 B	27,819 B
Encodings with CR/CRLF	1	12
Files covered by MasterHashes	1,212 of 1,222	all

Attribution

Long-form documents derive from the UDHR in Unicode project — the Universal Declaration of Human Rights, published by the United Nations, in a plain-text preparation by that project. Unicode discontinued it in January 2024, so the source artifact is pinned by SHA-256 to stay reproducible against a dead upstream. See data/udhr/PROVENANCE.md for the pin, the attribution, the two documented character substitutions, and the sources considered and rejected.

The generator is MIT licensed; the generated corpus is CC BY 4.0 (LICENSE-CORPUS), excluding the UDHR-derived text, which remains subject to its own notice.

See CHANGELOG.md for the full list.