You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A policy identity can be resolved back into its policy.PolicyResolver in :common turns a stored (id, version) pair into the frozen policy that produced it, so an application adding rows to an existing store normalizes them under the policy those rows were derived with, and a policy can be named in configuration rather than hardcoded. Each module exposes one resolver — QuodlibetPolicies today — and callers combine the ones they depend on with +. There is no global registry and nothing registers at startup.
The policy id grammar is published API.PolicyId renders and parses a chain of links, and every policy in the suite renders its own id through it, so the written and parsed forms cannot drift apart. Parsing is strict: a chain out of canonical order, an unknown link, or a version this build does not carry is a typed PolicyIdentityError, never a near match.
Policy in :common — the base type every policy now implements, carrying id and version.
The Unicode table generator. A JVM-only build module, never published, turns a pinned copy of the Unicode Character Database into the suite's frozen tables. The data files are checked in with their checksums and verified before parsing, so a build needs no network and a regeneration is reproducible. Tables are emitted as Kotlin source that compiles on every target, with no resource loading.
A material-change check, wired into check. Regenerating classifies every difference from the previous baseline as an addition, a modification or a removal; a modification in a normalization table fails the build, because the suite's byte-stability promise and its delta packaging both assume additions only. IDNA and confusable data, whose upstream guarantees are weaker, report changes without failing. verifyUnicodeTables also fails when a generated file is edited by hand or the data moves without a regeneration.
:unicode: NFC, NFD, NFKC and NFKD, via normalizeText(value, policy) under four frozen policies — TextPolicy.NfcU17, NfdU17, NfkcU17 and NfkdU17, with ids nfc.u17, nfd.u17, nfkc.u17 and nfkd.u17. The tables are frozen against Unicode 17.0.0 and ship with the library: nothing reads the platform's Unicode data, which is what would otherwise make the same input normalize differently on an old Android build and a current iOS one. A later Unicode release mints a new policy rather than changing one.
The Unicode Consortium's conformance suite runs on every target. All ~19,000 cases of NormalizationTest.txt for Unicode 17.0.0, on jvm, js, wasmJs and iOS, because a normalizer verified only on JVM has not been tested for the one property this suite sells.
:phone: phone numbers to E.164, via normalizePhone(value, policy) with PhonePolicy.E164, E164Lenient, e164ForRegion(region) and e164ForRegionLenient(region) — ids phone.e164, phone.e164+lenient, phone.e164+region-ca and phone.e164+region-ca+lenient. The region is part of the policy identity and is never inferred (ADR-0001). Digits from any script are converted through the frozen table io.github.aughtone:phonenumber publishes, so this module ships no table of its own; letters are refused rather than dialled.
A URL whose host is an address is accepted only in canonical dotted-quad form, and every other spelling — leading zeros, hexadecimal parts, fewer than four parts, a bare integer — is refused rather than rewritten. Rewriting would mean choosing between readings that disagree and hiding that choice inside a URL; normalizeIpv4 is where that choice belongs, and its policy identity records it.
:ubilibet: URL normalization, via normalizeUrl(value, policy) with UrlPolicy.Rfc3986U17 and Rfc3986U17Lenient — ids url.rfc3986+domain.ascii.u17 and url.rfc3986+domain.ascii.u17+lenient, which name the host policy they use. Limited to transforms that cannot change which resource is addressed: the query and fragment survive byte for byte, and sorting query parameters is deliberately not done.
:confusables: UTS-39 skeletons, via normalizeSkeleton(value, policy) with ConfusablePolicy.SkeletonU17 (id skeleton.u17). The full skeleton is implemented, not the simpler internalSkeleton: the standard defines it as bidiSkeleton(LTR, X), so the text is laid out by the bidirectional algorithm before it is reduced. A skeleton is documented throughout as a check rather than an identity — it is many-to-one by design, and storing one as an account key merges different users.
The Unicode Bidirectional Algorithm (UAX #9) through rule L2, with the paired-bracket rule, isolates and overrides. Its conformance suite runs in full on JVM — all 94,000 cases of BidiCharacterTest.txt — and a deterministic sample of it on every other target, because the whole corpus is several megabytes once compiled into a test binary.
:quodlibet gains five more normalizers, all table-free and all in the same bundle: credit-card/PAN (pan.digits, pan.digits+lenient), IBAN (iban.compact, iban.compact+lenient), IPv4 (ipv4.dotted-quad, ipv4.inet-aton), IPv6 (ipv6.rfc5952) and usernames (username.basic).
IPv4 ships two policies rather than one interpretation. Stacks disagree about the shorthand spellings — 192.168.0.010 is 8 under inet_aton, 10 under a plain decimal reading, and refused outright by Go and Python — so ipv4.dotted-quad accepts only the unambiguous form while ipv4.inet-aton applies the classic rules deliberately and records in its id that it did. Values under the two never match, which is correct: they are different claims. The financial pair gate on their check digits by default and relax that under a lenient policy, matching how :phone treats an implausible number; IPv6 follows RFC 5952 and has no lenient variant, because every candidate relaxation changes which address is meant rather than how it is spelled; the username base mirrors email's ASCII-only rules and encodes no platform behaviour.
:ubilibet: hostname and domain normalization under UTS-46, via normalizeDomain(value, policy) with DomainPolicy.AsciiU17 (id domain.ascii.u17) and DomainPolicy.AsciiU17Lenient (id domain.ascii.u17+lenient). Output is the A-label form. Every hostname is normalized here, ASCII included, so an ASCII name can never carry two policy identities for the same bytes. Nontransitional processing only; transitional processing is deprecated upstream and is not implemented.
The UTS-46 conformance suite runs on every target. All of IdnaTestV2.txt for Unicode 17.0.0, checked against both policies, with the file's own mapping from status codes to flags deciding what each policy must refuse.
Punycode (RFC 3492), including the bootstring overflow checks, which are part of the specification rather than defensive padding.
NormalizationStep in :common, the interface a table-free module accepts so a caller can compose a transform from another module without either module depending on the other. Each TextPolicy is one, so email.byte-stable+nfc.u17 names its transform in the identity.
Changed
Breaking: the email normalizer moved to a new coordinate. It is published as io.github.aughtone.normalize:quodlibet instead of io.github.aughtone.normalize:email. Change the dependency coordinate and nothing else: the package io.github.aughtone.normalize.email, every type and function name, the canonical output and the policy versions are all unchanged, so no stored value or derived token is affected. io.github.aughtone.normalize:email:0.0.1 remains on Maven Central and is not republished. :quodlibet is the bundle for normalizers that need no lookup table and no external dependency, so PAN, IBAN, IPv6 and the username base will join it rather than arriving as separate artifacts.
Breaking: the relaxed email policy is renamed.EmailPolicy.Lenient becomes EmailPolicy.ByteStableV1Lenient, and its id moves from email.lenient to email.byte-stable+lenient. Canonical output, policy version and error types are unchanged — an address normalized under it produces exactly the bytes it did in 0.0.1.
Policy ids are now chains. An id is an ordered list of links joined by +: the base rule set, then anything qualifying it, then any steps from other modules. The id therefore describes the policy rather than merely labelling it, which is what lets a stored id be resolved back to the policy that produced it.
Taken deliberately in alpha, before any consumer had derived a token under email.lenient. A published id does not move once anyone holds a value derived under it; email.byte-stable is unchanged and never will move.