Repository navigation
CodecCat v0.1.1 — Opt-in decoding ambiguity diagnostics
This is a backward-compatible patch to the native codeccat Runtime. The released default inference model remains byte-for-byte identical to v0.1.0 (SHA256 16073306b5027bf93acc2883f57fbda0cb0cf2d77b9a7bb62cf1b07fcbb9748b). The established detect(), decode(), Detection fields, CLI and BOM/L3 policies are unchanged.
New: Optional diagnostic for potentially ambiguous C1 controls
from codeccat import detect
from codeccat.diagnostics import inspect_ambiguity
raw = b"Price: \x80; total."
result = detect(raw)
diagnostic = inspect_ambiguity(raw, result)
if diagnostic.warning:
print(diagnostic.reason, diagnostic.alternative_encodings)inspect_ambiguity(raw, result) returns immutable metadata (warning, reason, alternative_encodings). It checks whether the already-selected decoding includes Unicode C1 controls (U+0080–U+009F), and if one of the already-ranked codecs can strictly decode the original bytes into different, C1-free text. It does not change or substitute the selected codec, decoded string, candidates, Detection.ambiguous, BOM decision or opt-in L3 state. It has no new ML dependencies.
Interpretation: a warning is a request to consult a trusted encoding or file metadata, not proof of wrong decoding and not a model confidence probability. Legal Latin-1 data may intentionally contain C1 controls. No warning does not imply correctness.
Validation and caveats
- Clean-installed Wheel/API/model identity checks passed on Linux, macOS and Windows for x86-64 and ARM64; Python 3.10–3.14 supported.
- Examined 1,886 historical original HTML files after provenance and training-overlap screening. Of the 572 files whose Unicode text differed from the archive-directory codec's strict decoding, 17 were flagged; none of 1,314 directory-label-agreeing results were flagged. This is only source-label agreement, not manually verified ground truth or an estimated population false-positive rate.
- Replaced per-character Python C1 scan with compiled regex, reducing incremental diagnostic call p50 approximately 10–12× on this original HTML test set. Times are Linux CI sample measurements, not universal latency guarantees.
- See research protocol and validation record. No original HTML or copyrighted training corpus is bundled.
Distribution
This release adds the tested source distribution (sdist) to the six platform-specific Wheels: manylinux_2_28 x86-64/ARM64, macOS x86-64/ARM64, and Windows x86-64/ARM64. The frozen default CCM model and optional L3 artifact are verified in every asset. Official PyPI Trusted Publishing is a separate action after GitHub release and asset audits.
codeccat-train==0.1.0 is a separate, already published training package and is not changed by this Runtime patch.
The project's historical training-data rights caveat remains described in RIGHTS_AND_PROVENANCE.md.