UnicodeTestSuite v3.0

A correctness release. v2.0's three central promises — cryptographic integrity, a machine-parseable filename contract, and usable detection ground truth — each had a hole in it. This release closes them, and adds the long-form and line-ending coverage the corpus needed to discriminate between detectors rather than defeat all of them equally.

Not backward compatible. Manifest.csv gains columns, two files are renamed, and several documents change content, so hashes move. Read ManifestVersion.txt (or the meta table in Manifest.sqlite) to tell a v2 corpus from a v3 one.

Downloads

Asset	Contents
UnicodeTestSuite-v3.0.zip	The generated corpus — 1,359 files plus manifests, hashes and certificate
UnicodeTestSuiteGenerator-v3.0.zip	The generator, its tests, and the vendored long-form source text

Both archives are reproducible: rebuilding from the same tree yields byte-identical zips.

Verify what you downloaded

python GenerateCorpus.py verify --corpus /path/to/UnicodeTestSuite

This re-checks the metadata files against CorpusCertificate.txt, then every file against Manifest.csv, re-decoding each under its declared encoding, and fails if any file is present on disk but absent from the manifest.

Highlights

verify no longer passes on a tampered corpus. In v2.0, MasterHashes.sha256 covered 1,212 of the 1,222 files a corpus contained — the manifests, the index, the certificate and the reference files were all outside it. Editing a reference file, corrupting Manifest.sqlite, or planting an unlisted file were each reported as All 1,212 files verified OK. All three now fail, and CI replays them on every push.

Ground truth is a set. 300 of v2.0's text files were byte-identical to a file carrying a different declared encoding, and 27 byte sequences appeared twice under contradictory labels. A detector answering ascii for pure-ASCII content — the more precise answer — scored 65 false negatives for being right. The new AlsoValidAs column lists every encoding that decodes a file's bytes to the same characters. Score set membership, not string equality.

Samples a detector can actually classify. v2.0's legacy files had a median length of 31 bytes. 15_LongForm/ adds multi-kilobyte natural-language text in 29 encodings, each document emitted only into encodings that historically carried its language. Measured against a real detector, legacy accuracy on long-form samples is 92.2%, against 72.2% on the short ones.

CR and CRLF beyond UTF-8. v2.0 had no CRLF file in UTF-16, UTF-32 or any legacy code page, leaving 0D 00 0A 00 — the most common byte pattern in real Windows text — unrepresented. Coverage goes from 1 encoding to 12.

generate no longer deletes without asking, and --corpus DIR lets both subcommands target any directory.

Statistics

Item	v2.0	v3.0
Total generated files	1,212	1,359
Canonical documents	94	94
Long-form documents	0	24
Root folders	15	16
Distinct encodings	35	35
Legacy sample maximum	125 B	27,819 B
Encodings with CR/CRLF	1	12
Files covered by MasterHashes	1,212 of 1,222	all

Attribution

Long-form documents derive from the UDHR in Unicode project — the Universal Declaration of Human Rights, published by the United Nations, in a plain-text preparation by that project. Unicode discontinued it in January 2024, so the source artifact is pinned by SHA-256 to stay reproducible against a dead upstream. See data/udhr/PROVENANCE.md for the pin, the attribution, the two documented character substitutions, and the sources considered and rejected.

The generator is MIT licensed; the generated corpus is CC BY 4.0 (LICENSE-CORPUS), excluding the UDHR-derived text, which remains subject to its own notice.

See CHANGELOG.md for the full list.

UnicodeTestSuite v2.0

UnicodeTestSuite v2.0

Highlights

This release significantly improves the correctness, consistency, and interoperability of the generated corpus.

• Canonical Python/.NET encoding names
• Machine-readable filename format
• Fixed encoding token position (always token #5)
• Exhaustive compatibility validation
• Deterministic corpus generation and verification
• Improved category numbering and organization
• Simplified, cross-platform encoding support

---

Added

• Added deterministic generate and verify command-line modes.
• Added exhaustive post-generation verification.
• Added synthetic binary format signature generation.
• Added globally unique numeric category codes.
• Added a deterministic filename format suitable for automated parsing.
• Added verification that every generated filename places the encoding token at a fixed position.

Changed

Encoding Names

• Replaced cosmetic encoding labels with canonical Python/.NET codec identifiers.
• Standardized Unicode encoding names (for example utf-8, utf-16LE, utf-32BE).
• Standardized Windows code pages (windows-1250 through windows-1258).
• Standardized ISO-8859 codec names.
• Standardized KOI8 codec names.
• Standardized East Asian codec names.

Filename Format

• Redesigned filenames to be fully machine-readable.
• Preserved hyphens inside encoding names (for example iso-8859-1, shift-jis).
• Converted underscores inside encoding identifiers to hyphens for filenames only, preventing ambiguous parsing.
• Guaranteed that the encoding identifier is always located at index 4 (the fifth token).

Filename format:

DocumentID_CategoryCode_CategoryName_Title_Encoding_[BOM_]LineEnding.ext

Category Organization

• Unified ASCII and shared document categories into a single category model.
• Added numeric identifiers to ASCII categories.
• Renumbered all categories to follow directory order.
• Category codes are now globally unique.

Corpus

• Reduced duplicate encodings.
• Removed unsupported or ambiguous codec aliases.
• Updated the corpus to contain only verified encoding identifiers.

Removed

Removed encodings that could not be reliably supported by both Python and .NET or that represented duplicate implementations.

Removed ISO-8859 variants

• iso-8859-10
• iso-8859-11
• iso-8859-14
• iso-8859-16

Removed East Asian aliases

• cp932
• gbk
• cp874
• cp949
• ISO-2022-JP
• Big5-HKSCS
• HZ
• TIS-620

Validation

Every remaining encoding has been verified through research and/or live .NET testing.

Validation includes:

• Python codec compatibility
• .NET Encoding.GetEncoding()
• deterministic corpus generation
• deterministic verification
• filename parsing verification
• post-generation hash verification

Statistics

Item	Value
Total generated files	1,212
Canonical documents	94
Root folders	15
Supported encodings	40
Windows code pages	9
ISO-8859 encodings	11
East Asian encodings	7
KOI8 encodings	2

Downloads

UnicodeTestSuite-v2.0.zip

Includes the complete UnicodeTestSuite project:

• Generator source code
• Pre-generated Unicode test corpus
• 1,212 verified test files
• Manifest.csv
• Manifest.sqlite
• MasterHashes.sha256
• CorpusCertificate.txt
• Statistics.txt
• Documentation
• Changes
• MIT License

The corpus is ready for immediate use. No generation step is required.

---

UnicodeTestSuiteGenerator-v2.0.zip

Includes only the source code required to generate the corpus:

• GenerateCorpus.py
• Generator source code
• Documentation
• Changes
• MIT License

Running the generator reproduces the corpus deterministically. The generated files, manifests, and SHA-256 hashes are expected to be bit-identical to those included in UnicodeTestSuite-v2.0.zip.

Unicode Test Suite v1.0

Unicode Test Suite v1.0

Initial public release.

Highlights

• Deterministic, reproducible benchmark corpus
• ~1,300 verified test files
• 51 Unicode and legacy encodings
• Automatic post-generation verification
• SHA-256 integrity verification
• Stable document identifiers
• Machine-readable manifest
• Cross-platform corpus
• MIT licensed generator

See the README for complete documentation.

---

Downloads

UnicodeTestSuite-v1.0.zip

Pre-generated benchmark corpus, ready for immediate use.

Contents:

• Complete Unicode Test Suite corpus
• ~1,300 verified test files
• Manifest.csv
• Manifest.sqlite
• MasterHashes.sha256
• CorpusCertificate.txt
• Statistics.txt

No generation is required.

---

UnicodeTestSuiteGenerator-v1.0.zip

Generator source code used to build the corpus.

Contents:

• GenerateCorpus.py
• Generator source
• Built-in canonical documents
• Documentation
• MIT License

Running the generator reproduces the corpus deterministically. The generated files and SHA-256 hashes are expected to be bit-identical to those included in UnicodeTestSuite-v1.0.zip.