Repository navigation
Releases: cdcailf/cjkfold
Release list
0.4.0 — every name means one thing
No new features. One structural fix, and it breaks an import path.
Changed
-
The fold family moved from
cjkfold.normalizetocjkfold.folding; the two
writes moved tocjkfold.rewrite. Nothing about the functions changed —
from cjkfold import fold, normalize, strip_invisiblebehaves exactly as it did.Why: a function named
normalizecannot live in a module namednormalizewithout
one of the two losing its name, andimport a.b as cmeansc = a.b. So
import cjkfold.normalize as mhanded you the function, andm.foldraised
AttributeError. Every name now means one thing:cjkfold.normalizeis the
function,cjkfold.foldingis the module. (af57ad7)What breaks:
from cjkfold.normalize import fold— a path nothing ever
documented, and it now fails with a plainModuleNotFoundErrorinstead of an
AttributeErrorfrom something that was never a module. Every documented import
is untouched.
Tests
scripts/acceptance.pygained import-path checks, including one that the old
path is gone. (af57ad7)
0.3.0 — the broken class, directory walking
Added
- The
brokenclass.scannow reportsU+FFFDand the private use areas
alongside look-alikes, compat characters and invisible ones — damage that cannot
be repaired, only reported. A private use code point earns its place in the
report precisely because nothing defines what it looks like: the font doing the
rendering decides, and it can come out as a box, as nothing at all, or as a
glyph that means something else. (b17902d) scanwalks directories. A corpus is a tree, not an argument list.
Dot-entries stay out, so.gitdoes not end up in the report; a non-text file
found by walking is skipped and counted, while a non-text file you named is
still an error — pointing at a PDF is a mistake, walking past one is not.
(b17902d)- The
scanfooter prints once per run instead of once per file, which starts to
matter the moment you point it at a directory. (b17902d)
Tests
- The four compat characters the roadmap named are pinned with measured values:
full-widthA→A, ideographic space → space,U+F900→ 豈, NBSP → space.
(b17902d)
No change to fold(), to check, or to the equivalence table. Exit codes and
check output are unchanged.
0.2.0 — scan, normalize, strip_invisible
Added
cjkfold scan— every character that breaks a comparison, in one read-only
pass: the look-alikes, the NFKC familyfold()refuses to touch, and the
invisible family. Each finding says what to do with it —fold,nfkc,
strip, ormanual, wheremanualmeans do not touch this without looking.
It exits0by default, because it is a report and not a gate: a gate that fails
on a load-bearing joiner fails on ordinary emoji.--strictgates anyway,
--jsonfeeds a pipeline,--classesnarrows the report. (c5be88a,5835dbf)normalize()andstrip_invisible()— the two explicit writes. The
first applies the NFKC family with full-width punctuation spared by default; the
second deletes invisible characters,safeby default, because half of that
family carries meaning — a family of three is five code points, and deleting the
joiners leaves three separate people. Neither is exposed as a subcommand:fold -w
is already an in-place rewrite, and two more destructive commands would be a gun
pointed at a corpus. (a733ef5)Finding,scan_stats()andis_punctuation()in the public API.
(2bd05e6,5835dbf)
Changed
scan's whitelist lookup is now a single set test instead of a dict lookup plus
four range comparisons: 0.42s → 0.29s over 640,000 characters. (f6c5f4f)
Docs
- README: "The characters that never show up" and "The other two rewrites",
both carrying output from real runs rather than hand-copied examples; the 中文简述
follows. Thecheckexample also gained the summary line it had been missing —
the reason it was missing is that it had been copied by hand, which is exactly
what the new section refuses to do. (e9c4d68, and this release) tests/fixtures/— the three files the roadmap asked for, withcheck's output
frozen byte for byte. (1a468e5)
No change to fold(), to check, or to the equivalence table. Text that folded
correctly under 0.1.2 folds identically under 0.2.0.