Investigate Unicode normalization in command routing

[iriswolf]

Problem

A user reports that a Cyrillic command containing ё is matched for some Telegram users and not for others. The report is not reproducible yet.

The current dispatcher compares command tokens with StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase; it does not apply Unicode normalization first. That can distinguish a composed ё (U+0451) from a canonically equivalent decomposed sequence (е + U+0308). It must not be assumed that affected users actually send this sequence: they may instead be entering the distinct letter е.

Why This Needs Investigation Before Design

ё versus decomposed е + diaeresis is a Unicode representation issue. е versus ё is a product-level aliasing decision. A global replacement or normalization of incoming message text would be unsafe: it could alter raw handler input, template route values, and regex captures.

Investigation Plan

1. Capture the exact Message.Text code points from one matching and one non-matching update. Record the Telegram update type and the route declaration, but do not publish user text or personal data.
2. Add a deterministic regression test for the observed representation through the generated registration path and direct handler registration path.
3. Confirm whether the mismatch is:
   • composed versus decomposed canonical Unicode;
   • plain е versus ё spelling;
   • a routing-boundary issue unrelated to Unicode normalization.
4. Verify the behavior for Command, CommandTemplate, and CommandRegex separately. Regex capture values must be treated as a distinct contract.
5. Only after the observed case is classified, write and review the implementation design.

Design Constraints After Reproduction

• Preserve the raw ctx.Message.Text exposed to handlers.
• Do not globally replace е with ё or the reverse.
• If canonical normalization is justified, scope it to the smallest safe command-matching boundary and use NormalizationForm.FormC consistently for both route literals and compared command input.
• Do not silently normalize regex captures or arbitrary text routes without an explicit, tested contract.
• Keep user-facing spelling aliases explicit, for example separate [Command("работенка")] and [Command("работёнка")] routes when a bot chooses to support both.

Code Pointers

• src/TeleFlow.Framework/Internal/Handlers/TelegramHandlerSelector.cs:321-339 chooses the command route matching path.
• src/TeleFlow.Framework/Internal/Handlers/TelegramHandlerSelector.cs:404-474 extracts the prefixed or prefix-less command body.
• src/TeleFlow.Framework/Internal/Handlers/TelegramHandlerSelector.cs:500-524 compares exact command tokens with ordinal comparers.
• src/TeleFlow.Framework/Internal/Handlers/TelegramHandlerSelector.cs:543-575 passes command templates and regex routes to the pattern matcher.
• tests/TeleFlow.ArchitectureTests/TelegramHandlerDispatcherTests.cs:823-870 covers command-template dispatch and prefix-less regressions; extend it with the verified Unicode case.

Acceptance Criteria

• A deterministic test demonstrates the actual reported representation mismatch, or the report is classified as a bot-level aliasing/configuration issue.
• The issue records the verified Unicode sequences and the affected route kinds without retaining user message content.
• A reviewed design is added only after reproduction and explicitly states the behavior for exact commands, templates, regex, raw handler text, and aliases.
• No implementation is merged under this issue before those conditions are met.

[iriswolf]

Verified reproduction

The framework defect is reproducible without depending on a particular Telegram client.

The following visually equivalent command characters are different UTF-16 sequences:

• NFC/composed: ё = U+0451;
• decomposed: е + combining diaeresis = U+0435 U+0308.

Current behavior:

Operation	Result
OrdinalIgnoreCase before normalization	no match
current template Regex before normalization	no match
OrdinalIgnoreCase after NFC normalization	match
template Regex after NFC normalization	match

This proves that the current command matcher is not canonically Unicode-equivalent. It does not prove that iOS is the source of the reported input. A captured update is still useful for confirming the device-specific report, but it is no longer required to demonstrate the framework gap.

Both generated registration and direct registration converge on TelegramRouteDescriptor and TelegramHandlerSelector, so the behavior should be implemented once in the shared runtime matcher rather than duplicated in the generator or reflection registration.

Proposed contract

Exact commands

• Normalize the configured command pattern to NFC once when the route descriptor is created.
• Normalize the extracted command token to NFC before comparison.
• Keep the existing ordinal/ordinal-ignore-case comparison semantics after normalization.
• Do not modify the original Message.Text.

Command templates

• Normalize the configured template to NFC once before its matcher is compiled.
• Normalize the extracted command body to NFC before template matching.
• Bind template route values from the normalized command body.
• String route values therefore have an explicit NFC contract. Numeric route values are unaffected.
• Keep the original Message.Text available through the context unchanged.

Normalizing template route values is intentional. Maintaining a second normalized string plus a UTF-16 index map solely to reconstruct original capture spans would add substantial complexity to a hot routing path for no meaningful command-processing benefit. Callers that require the exact original representation already have Message.Text.

Command regex routes

• Do not normalize CommandRegex patterns or input.
• Regex is the low-level route contract: its author controls canonical-equivalence behavior explicitly.
• Regex captures continue to expose the exact matched representation.

Out of scope

• No global е to ё aliasing. Those are distinct letters and aliases remain explicit route declarations.
• No normalization of Text, TextTemplate, or TextRegex routes in this issue.
• No mutation of Telegram schema objects or handler context text.
• No device-specific workaround.
• No compile-time generator branch: generated and direct registrations must use the same runtime contract.

Implementation shape

Introduce a small internal NFC normalization helper with an allocation-free return for already normalized strings:

internal static string NormalizeForCommandMatching(string value)
{
    return value.IsNormalized(NormalizationForm.FormC)
        ? value
        : value.Normalize(NormalizationForm.FormC);
}

Apply it at two ownership boundaries:

1. TelegramRouteDescriptor / matcher construction for exact and template route patterns;
2. TelegramHandlerSelector after prefix and bot-mention extraction, immediately before exact/template matching.

Do not place normalization in the source generator. The generator describes routes; the shared runtime selector owns matching semantics.

Test plan

Add executable specifications for both direct and generated registration:

1. composed exact command route matches decomposed input;
2. decomposed exact command route matches composed input;
3. composed command template literal matches decomposed input;
4. template string route values are NFC;
5. raw Message.Text remains byte-for-byte unchanged;
6. plain е does not alias to ё;
7. CommandRegex remains unnormalized;
8. text route kinds remain unchanged;
9. already normalized ASCII and Cyrillic commands keep existing behavior.

The implementation should include a focused benchmark or allocation assertion only if the normalization helper causes a measurable routing regression. Command strings are short, and correctness is the primary concern, but the normal NFC path must not allocate a replacement string.

Acceptance decision

This issue is now reproducible and ready for design review. Implementation should start only after the proposed exact/template/regex contract is accepted.