Fix crash on supplementary-plane characters (emoji, astral CJK) in release/catalog titles

[jbob06]

What

Fixes #116. UnicodeComparisonNormalizer.NormalizeWordsWithSourceSpans iterated a string one UTF-16 code unit at a time and called .Normalize(NormalizationForm.FormD) on each unit alone. Any character outside the Basic Multilingual Plane (an emoji, or a character like a CJK Extension B ideograph) is a surrogate pair in .NET strings — normalizing one half in isolation throws ArgumentException: String contains invalid Unicode code points, since a lone surrogate isn't well-formed UTF-16 on its own. This is a real, valid character being torn in half by the per-code-unit loop, not malformed input.

This crashes ReleaseTitleMatchScorer.TokenizeWithSpans, which DownloadDecisionMaker calls per report while matching RSS releases (BuildContradictoryVariants tokenizes both the release title and the matched author's full catalog titles). The per-report catch in GetBookDecisions turns the crash into a permanent, non-bypassable rejection ("Unexpected error processing release"), so any release whose title — or a title in the author's catalog it gets compared against — contains such a character can never be grabbed, silently. (I initially thought this could abort matching for the whole RSS batch; that was wrong, see the correction comment on #116 — it's scoped to the one report.)

Fix

• Walk the input by Unicode scalar value (consuming full surrogate pairs via char.IsHighSurrogate/char.IsLowSurrogate) instead of by UTF-16 code unit, so a supplementary-plane character normalizes as a whole instead of being split. Source-span tracking (used to map normalized tokens back to original release-title offsets) merges the two spans of a pair into one.
• Switched character classification from char/CharUnicodeInfo to System.Text.Rune (Rune.IsLetterOrDigit, Rune.GetUnicodeCategory, Rune.ToLowerInvariant) so a supplementary-plane letter is recognized as a letter instead of being silently dropped (a lone surrogate half is never IsLetterOrDigit). One side effect worth calling out: a supplementary-plane symbol (e.g. an emoji) sitting directly between two letters with no surrounding whitespace is now correctly treated as a word boundary — previously it vanished without inserting one ("ab🔥cd" → "abcd" before, "ab cd" now). Covered by a new test.
• A genuinely unpaired/lone surrogate (actual ill-formed input, not a valid character) is sanitized to U+FFFD via a new SanitizeUnpairedSurrogates helper — reusing the method's existing replacement-character handling — instead of crashing. NormalizeWordsWithSourceSpans sanitizes up front (it's length-preserving, so the source-span indices stay valid); NormalizeInternal (backing NormalizeKey/NormalizeWords) had the same latent whole-string-Normalize() bug and gets a try/sanitize-and-retry fallback instead, since that path isn't on the RSS-matching hot loop and the common case shouldn't pay for a sanitize scan it doesn't need.

Known limitation, not fixed here

ReleaseTitleMatchScorer's TokenRegex ([\p{L}\p{Nd}]+) matches per UTF-16 code unit, and .NET regex doesn't recognize a surrogate pair as \p{L} even when it's a real letter — so a supplementary-plane letter embedded in a release title still won't survive tokenization as part of a token (e.g. Tokenize("abc𠀀def") still splits into two tokens rather than one). It no longer crashes, which is the actual reported bug, but full round-trip fidelity for supplementary-plane letters through the regex tokenizer would need a separate change to the tokenizer itself. Flagging so it isn't mistaken for solved by this PR.

Testing

Added regression tests in UnicodeComparisonNormalizerFixture and ReleaseTitleMatchScorerFixture covering: an emoji no longer crashing (NormalizeWords, NormalizeWordsWithSourceSpans, and end-to-end ReleaseTitleMatchScorer.Tokenize), a genuinely unpaired surrogate no longer crashing, a real supplementary-plane letter (U+20000) being preserved through NormalizeWords, the word-boundary behavior change for an adjacent symbol called out above, and the source-span merging for a surrogate pair (asserted by value, not just "doesn't throw").

Full suite: dotnet test src/Chaptarr.Core.Test/Chaptarr.Core.Test.csproj → 2859 passed, 0 failed, 1 skipped (pre-existing, unrelated).

Independently reviewed (fresh-context Opus pass) before opening this PR — findings incorporated: removed the per-scalar try/catch on the RSS-matching hot path in favor of sanitizing once up front (same fix, one code path, no exception-as-control-flow); added value-asserting tests in place of DoesNotThrow-only ones; corrected the issue's understanding of blast radius (see comment on #116).

🤖 Generated with Claude Code

https://claude.ai/code/session_01XNX9Y1DFRRTsVFXG6aYoHH