Mojibake v0.6.0

Codename: [GRAVE ACCENT]

Added

• Rust crates in src/rust/: mojibake-sys compiles the library with Cargo and declares the raw
  FFI, and mojibake-rs (imported as mojibake) adds a safe API for normalization, locale
  casing, caseless matching, segmentation, terminal width, character properties, collation, IDNA,
  and confusables. Run their tests with make test-rust
• mojibake-sys binds every public function except the variadic and va_list formatting ones.
  The declarations in src/rust/mojibake-sys/src/functions.rs are generated from
  utils/generate/functions.ts by make generate
• The mojibake-rs crate adds encoding, which validates, decodes, encodes, counts, and converts
  UTF-8, UTF-16LE/BE, and UTF-32LE/BE byte slices with a Malformed policy (stop with the byte
  offset, replace with U+FFFD, or skip), and bidi, whose Paragraph resolves embedding levels
  and reorders lines, splits them into runs, and renders them in visual order with mirroring
• The mojibake-rs crate adds filter with combinable Filter flags, is_identifier with an
  IdentifierProfile, emoji::{sequence, is_sequence, is_rgi} for UTS #51 sequences,
  security::resolved_scripts for UTS #39 mixed-script detection, and a Property type that
  looks up any of the library's properties by name for CharExt::has_property and
  CharExt::property_value. CharExt also gains script_extensions, is_id_start,
  is_id_continue, is_pattern_syntax, and is_pattern_white_space. mojibake-sys adds the
  filter, emoji sequence, emoji qualification, and script set constants, checked against C
• The mojibake-rs crate exposes the whole mjb_character record: CharExt::info returns a
  CharInfo with the general category, canonical combining class, BidiClass, Bidi_Mirrored,
  DecompositionType, decimal, digit, and numeric values, and simple case mappings, and CharExt
  gains canonical_combining_class, bidi_class, is_bidi_mirrored, decomposition_type, and
  simple_uppercase, simple_lowercase, and simple_titlecase. bidi::BidiChar gains a
  class field with the resolved bidi class. mojibake-sys adds the bidi class, decomposition
  type, and MJB_NUMBER_NOT_VALID constants
• The mojibake-rs crate adds LanguageTag, which parses a BCP 47 tag strictly with
  mjb_locale_parse and exposes its case-normalized language, extended language, script, region,
  variant, extension, private-use, and grandfathered subtags, and the casing Locale it selects.
  It implements FromStr and Display, which writes the tag in canonical case
• The mojibake-rs crate adds hangul::{compose, decompose, syllable_name} and the Hangul
  syllable and jamo predicates, CharExt::{is_cjk_ideograph, is_cjk_extension_ideograph}, and
  grapheme_prefix and word_prefix, which return the first N graphemes or word-break segments
  in one pass. idna::IdnaError::errors is now a typed IdnaErrors flag set instead of a u32
• The mojibake-rs crate adds Plane and CharExt::plane, with the plane number, name,
  abbreviation, and codepoint range. Planes 4 to 13 have no name. mojibake-sys adds the
  MJB_PLANE_* constants, checked against C
• make package-rust vendors the C sources into mojibake-sys and runs cargo package and
  cargo publish --dry-run for both crates, also in CI. make update-version sets the crates'
  version. src/rust/README.md describes the release steps
• mjb_nul_terminated_length returns the byte length of a string that ends at its first NUL code
  unit, in any input encoding

Changed

• The Rust example in examples/rust/ now uses the safe mojibake crate through Cargo instead of
  hand-written FFI declarations, and make -C examples example-rust maps the FEATURE_* variables
  to the crate's Cargo features. README.md documents the Rust crates
• mjb_text_make and the mjb_text_* helpers resolve a MJB_NUL_TERMINATED length when the text
  is created, so it is scanned once. The sentinel is kept when the buffer is NULL or the encoding is
  invalid, and the function that takes the text reports the error
• mjb_decode_next and mjb_decode_previous return MJB_STATUS_INVALID_ARGUMENT for a text whose
  byte_length is MJB_NUL_TERMINATED, like the break iterators. They rescanned the whole string
  for its terminator on every call, so iterating took quadratic time
• All internal decoding uses the whole-scalar decoder of mjb_decode_next, inlined. The streaming
  DFA decoder is removed. Decoding valid text is faster: about 2.4x for UTF-8, 2x for UTF-16 and
  1.4x for UTF-32 in a 2 MB mixed-script loop
• Break iterators on malformed input: each maximal subpart of a malformed subsequence is one
  U+FFFD with a position of its own, as in mjb_decode_next. A run of malformed bytes used to be
  merged into one U+FFFD, and an incomplete sequence at the end of the text was dropped
• Break iterators return MJB_BT_NOT_SET for an invalid encoding and for generic UTF-16 or UTF-32
  text without a byte order mark, instead of iterating over replacement characters
• The state and in_error members of mjb_grapheme_break_state, mjb_word_break_state,
  mjb_sentence_break_state, and mjb_line_break_state are removed. Callers only set index
• make test-wasm and nmake /F Makefile.nmake test-wasm rebuild and sync the WASM artifacts
  before testing, so they no longer test a stale src/api/mojibake.wasm

Fixed

• mjb_plane gave the Supplementary Special-purpose Plane and Supplementary Private Use Area-A
  the values 4 and 5. They are now MJB_PLANE_SSP = 14 and MJB_PLANE_PUA_A = 15, matching
  mjb_codepoint_plane, so mjb_plane_name(mjb_codepoint_plane(0xE0001), false) returns
  "Supplementary Special-purpose Plane" instead of "Unassigned", and planes 4 and 5 are now
  "Unassigned". This changes the values of public enum constants (an ABI change); the TypeScript
  Plane enum changes the same way
• mjb_bidi_reorder_line skipped UAX #9 rule L1 for lines that end inside the paragraph:
  whitespace and isolate formatting characters at the end of such a line kept their resolved level
  and were reordered with the text before them. In an RTL paragraph, the space of the line "abc "
  of "abc def" was placed after c instead of at the line end. The line levels are now reset on
  a private copy, and mjb_bidi_line_runs reports the same levels. mjb_bidi_line_runs also
  returns MJB_STATUS_INVALID_ARGUMENT for a visual-order index outside the paragraph. The
  BidiTest.txt conformance test now checks every two-line split of each case
• Grapheme segmentation split an emoji flag that followed another flag and a control, CR, or LF:
  "🇮🇹\n🇮🇹" gave four clusters instead of three. The regional indicator count now restarts at
  every codepoint that is not a regional indicator (GB12, GB13)
• Grapheme segmentation joined an Extended_Pictographic to a preceding ZWJ that was followed by
  Extend characters or another ZWJ, as in U+1F468 U+200D U+0308 U+1F468. GB11 requires the ZWJ to
  immediately precede the Extended_Pictographic
• The Windows CLI interactive mode stopped accepting input after an unpaired surrogate key event.
  It decodes key events with mjb_decode_next and skips malformed surrogates. The internal
  mjb_utf16_decode_step helper and src/utf16.h are removed
• mjb_grapheme_prefix_length, mjb_word_prefix_length, and their _for_width variants
  estimated boundary offsets from the replacement codepoint and could cut at the wrong byte in
  malformed text. They use the exact offsets of the iterator
• The line break iterator now reports the LB3 mandatory break at the end of a text that holds only
  a byte order mark
• mjb_bidi_resolve byte offsets of multi-byte UTF-8 characters, and of the first character after a
  byte order mark, are the offset of the first byte
• Out-of-range and surrogate UTF-32 code units were silently dropped instead of being reported as
  malformed input, so functions such as mjb_normalize accepted them or passed them through
• A UTF-16 code unit following an unpaired high surrogate was swallowed together with the
  surrogate instead of being decoded on its own