Skip to content

Mojibake v0.8.0

Latest

Choose a tag to compare

@zaerl zaerl released this 17 Nov 15:59
· 3 commits to main since this release
e91994c

Codename: [<control>]

Added

  • Elixir: a mojibake Mix package in src/elixir that compiles the C library into a NIF, with
    normalization (including NFKC_Casefold) and quick checks, locale-aware upper, lower, and
    titlecase, full and simple case folding, caseless matching, grapheme, word, and sentence
    segmentation, line-break opportunities, counts and prefixes, and terminal width and truncation.
    It passes the Unicode normalization, segmentation, and line-break conformance tests
  • make test-elixir, make vendor-elixir, and make package-elixir, and an Elixir CI job
  • Elixir: character data. Mojibake.Char returns names, categories, combining and bidi classes,
    decomposition types, numeric values, simple case mappings, blocks, scripts and
    Script_Extensions, East Asian width, emoji properties, and every stored binary and enumerated
    property, as atoms. Mojibake.Script, Mojibake.Block, Mojibake.Plane, and
    Mojibake.Property describe their values, Mojibake.Emoji classifies emoji sequences, and
    Mojibake.Hangul composes and decomposes syllables. The atom tables are generated by
    make generate into src/elixir/lib/mojibake/unicode_data.ex
  • Elixir: Mojibake.Collation (UCA comparison, sorting, and sort keys), Mojibake.IDNA (UTS #46
    ToASCII and ToUnicode), Mojibake.Security (confusable skeletons and matching, resolved script
    sets), Mojibake.Identifier (UAX #31), Mojibake.Encoding (UTF-8, UTF-16, and UTF-32
    detection, validation, and conversion), and Mojibake.LanguageTag (strict BCP 47 parsing and
    formatting). They pass the Unicode collation and IDNA conformance tests
  • Elixir: Mojibake.Bidi, the Unicode Bidirectional Algorithm. resolve/2 returns a paragraph
    whose levels the NIF keeps, and levels/1, chars/1, reorder/2, runs/2, and visual/2
    lay out its lines. It passes BidiCharacterTest.txt
  • An Elixir example in examples/elixir, run with make -C examples example-elixir
  • Elixir: precompiled NIFs for Linux (x86_64 and aarch64, glibc), macOS (x86_64 and arm64), and
    Windows (x86_64), downloaded from the GitHub release and checked against checksum.exs, so
    that installing the package needs no compiler there. Other platforms, and packages built with
    fewer features or MOJIBAKE_BUILD=1, compile from source. The Elixir precompiled NIFs
    workflow builds, checks, and uploads them
  • Elixir: Windows support with MSVC, through Makefile.win and nmake, tested in CI
  • Makefile.nmake gains test-elixir, vendor-elixir, and package-elixir, which the Windows
    Elixir CI job runs

Changed

  • Updated Attractor to 1.1.0
  • Moved src/api to src/wasm

Fixed

  • mjb_codepoint_info returns a record for surrogates (U+D800 to U+DFFF, category Cs) and
    private-use characters (U+E000 to U+F8FF, U+F0000 to U+FFFFD, and U+100000 to U+10FFFD,
    category Co) instead of MJB_STATUS_NOT_FOUND, with no name and default properties as in
    UnicodeData.txt. As a result mjb_for_each_codepoint, mjb_filter, and the CLI char and
    codepoint commands no longer drop private-use characters
  • mjb_codepoint_info returns a record (category Lo) for every Unicode 18 ideograph range listed
    in UnicodeData.txt: Tangut Ideograph now ends at U+187FF and Tangut Ideograph Supplement at
    U+18D20 (MJB_TANGUT_IDEOGRAPH_END and MJB_TANGUT_IDEOGRAPH_SUPPLEMENT_END), and the new
    Jurchen Character (U+18E00 to U+19191, "JURCHEN CHARACTER-") and Seal Character (U+3D000 to
    U+3FC3F, "SMALL SEAL CHARACTER-
    ") ranges are handled, with the new
    MJB_JURCHEN_CHARACTER_START/_END and MJB_SEAL_CHARACTER_START/_END limits
  • mjb_codepoint_info returns MJB_STATUS_NOT_FOUND for the unassigned U+18CDB to U+18CFE
    instead of a "KHITAN SMALL SCRIPT CHARACTER-*" record
  • MJB_FILTER_LIMIT_COMBINING no longer restarts its count at a Default_Ignorable_Code_Point
    character, so inserting U+200D ZERO WIDTH JOINER between runs of marks cannot stack more than
    MJB_FILTER_MAX_COMBINING_MARKS marks on one letter
  • Documented that MJB_FILTER_LIMIT_COMBINING counts every combining mark (Mn, Mc, and Me) in
    any block, so a mark outside U+0300 to U+036F, such as U+0489, does not restart the count