Mojibake v0.5.0

Codename: [LATIN CAPITAL LETTER P]

Added

• TypeScript: graphemeSegments, wordSegments, sentenceSegments, and lineSegments split the
  input at its boundaries
• C++: _into transforms that write into a caller's string and reuse its capacity, for the
  std::string and the UTF-16 and UTF-32 overloads: normalize_into, nfc_into, nfd_into,
  nfkc_into, nfkd_into, filter_into, nfkc_casefold_into, case_map_into,
  uppercase_into, lowercase_into, titlecase_into, casefold_into, casefold_simple_into,
  and confusable_skeleton_into, plus convert_encoding_into, collation_key_into,
  idna_to_ascii_into, and idna_to_unicode_into, which return the IDNA error flags
• mjb_text input structure with the mjb_text_make, mjb_text_utf8, mjb_text_utf16le,
  mjb_text_utf16be, mjb_text_utf32le, and mjb_text_utf32be helpers
• mjb_buffer output structure, holding caller-provided storage, its capacity or written size, and
  its encoding, with the mjb_buffer_make helper and mjb_buffer_text to pass written output to
  another call as input
• C++ support for UTF-16 and UTF-32 text in the platform's byte order (mjb::native_utf16_encoding
  and mjb::native_utf32_encoding
• position in mjb_grapheme_break_state, mjb_word_break_state, mjb_sentence_break_state,
  and mjb_line_break_state: the break iterators store the byte offset of each position they
  report, so a segment runs from one break position to the next

Changed

• Breaking: mjb_result.transformed is now mjb_result.owned. It is true when the library
  allocated output; when false, output points into the caller's input. The C++
  TextResult::transformed() is now owned()
• Breaking: mjb_string_validate(buffer, byte_length, encoding, diagnostic) is now
  mjb_validate_encoding(const mjb_text *text, diagnostic)
• Breaking: all the functions that took a buffer, byte length, and encoding now takes a
  const mjb_text*
• Breaking: all the functions that took the output encoding, output buffer, and output size now
  takes a mjb_buffer*
• Breaking: the TypeScript graphemeBreakNext, wordBreakNext, sentenceBreakNext, and
  lineBreakNext return BreakPosition[], a { position, type } entry for each position, instead
  of the break types alone. position is the byte offset in the input's encoding, read from the
  iterator's state.position. The site places its break symbols by position
• Breaking: the TypeScript convertEncoding(input, outputEncoding, options) is now
  convertEncoding(input, options) and reads options.outputEncoding like the other
  transformations. The output encoding still defaults to UTF-8
• Breaking: mjb_caseless_match, mjb_collation_compare, and mjb_confusable_match take two
  const mjb_text * instead of two buffer, byte length, and encoding triples
• Breaking: mjb_collation_compare, mjb_caseless_match and mjb_confusable_match now takes a
  malformed_policy and a diagnostic
• Breaking: mjb_map_case and mjb_map_case_into take the casing locale as an argument after
  type
• Breaking: MJB_LOCALE_NOT_SET selects the default mappings, and an invalid locale returns
  MJB_STATUS_INVALID_ARGUMENT. Concurrent calls with different locales no longer interfere
• Breaking: removed mjb_set_locale, mjb_get_locale, and mjb_reset_locale; the library no
  longer has a process-global locale. The C++ set_locale, get_locale, and reset_locale and the
  TypeScript setLocale and getLocale are removed too. The C++ casing functions take the locale
  as the argument after the input (after type for case_map), the TypeScript mapCase reads
  options.locale, and the CLI --locale option is passed to each casing call
• Breaking: allocator callbacks receive block sizes, so arena, pool, and accounting allocators
  no longer need to record them: mjb_realloc_fn is now (context, ptr, old_size, new_size) and
  mjb_free_fn is now (context, ptr, size). The size passed is the one most recently requested
  for the block; requested sizes are never 0, realloc receives NULL with an old_size of 0 for a
  new block, and free is never called with NULL
• Breaking: mjb_result has a new capacity field that records the allocation size of an owned
  output for mjb_result_free
• Breaking: the segmentation functions are named after their unit, so each unit's functions
  sort together: mjb_next_grapheme_break, mjb_next_word_break, mjb_next_sentence_break, and
  mjb_next_line_break are now mjb_grapheme_break_next, mjb_word_break_next,
  mjb_sentence_break_next, and mjb_line_break_next; their states mjb_next_state,
  mjb_next_word_state, mjb_next_sentence_state, and mjb_next_line_state are now
  mjb_grapheme_break_state, mjb_word_break_state, mjb_sentence_break_state, and
  mjb_line_break_state. The TypeScript API follows: graphemeBreakNext and so on
• Breaking: the truncation functions are named after what they return, the byte length of a
  prefix, since they do not modify their input: mjb_truncate_grapheme,
  mjb_truncate_grapheme_width, mjb_truncate_word, and mjb_truncate_word_width are now
  mjb_grapheme_prefix_length, mjb_grapheme_prefix_length_for_width, mjb_word_prefix_length,
  and mjb_word_prefix_length_for_width, and the TypeScript methods are now
  graphemePrefixLength and so on. The C++ wrappers return the truncated std::string_view, so
  they are named mjb::grapheme_truncate, mjb::grapheme_truncate_width, mjb::word_truncate,
  and mjb::word_truncate_width
• Breaking: mjb_for_each_codepoint takes a void *context after the callback and passes it
  to every call, so callbacks can keep their own state without global variables:
  mjb_for_each_codepoint(text, malformed_policy, callback, context, diagnostic). The callback
  type is now bool (*)(const mjb_character *character, mjb_character_position position, void *context); the character is const because callbacks only read it
• Breaking: C++ wrapper changes:
  • mjb::for_each_codepoint accepts any callable, including a lambda with captures, taking
    (const mjb_character &, mjb_character_position); an exception it throws stops the iteration
    and is rethrown. The C-callback overload takes the context pointer after the callback
  • normalize_result, filter_result, and convert_encoding_result reject a temporary
    std::string at compile time, because their TextResult can borrow the input
  • LibraryError::what() appends the library's message for the status, and
    mjb::status_message exposes mjb_status_message
  • mjb::BreakResult::index is now position, the byte offset of the reported position. It
    held the iterator's internal index, which is not a boundary offset. BreakResult::codepoint
    is decoded at position, U+FFFD for a malformed subsequence and MJB_CODEPOINT_NOT_VALID at
    the end of the text; it was the iterator's internal current codepoint, which is the previous
    character after a rule absorbs a format or extend character, such as U+200B (SB5) or U+0301
    (WB4), and at the end of the text. The breaker classes take an mjb::Text
  • analysis functions take an mjb::Text instead of a std::string_view and an encoding
    parameter: validate_encoding, codepoint_count, for_each_codepoint, grapheme_count,
    word_count, sentence_count, terminal_width, is_identifier, normalization_quick_check,
    resolved_script_set, emoji_sequence, is_emoji_sequence, is_rgi_emoji, locale_parse,
    caseless_match, confusable_match, collation_compare, collation_key, and
    collation_key_result. UTF-8 callers drop the encoding argument; for another encoding, pass
    mjb::Text(bytes, encoding)
  • renamed to match the C API: validate_string to validate_encoding; is_hangul_l,
    is_hangul_v, and is_hangul_t to is_hangul_leading_jamo, is_hangul_vowel_jamo, and
    is_hangul_trailing_jamo; is_cjk_extension to is_cjk_extension_ideograph (also on
    Character); codepoint_numeric_value to numeric_value; parse_locale to
    locale_parse; and compare to collation_compare
• Breaking: mjb_hangul_syllable_composition is no longer public, and mjb_buffer_character
  moved out of mojibake.h
• Breaking: The new mjb_hangul_syllable_compose(leading, vowel, trailing, syllable)
  composes a syllable from its jamo, and mjb_hangul_syllable_decomposition is now
  mjb_hangul_syllable_decompose. The C++ wrapper follows with hangul_syllable_compose and
  hangul_syllable_decompose
• Breaking: the _into functions now write in a single pass instead of measuring first
• make tidy now runs clang-tidy with .clang-tidy over the library, CLI, and tests, replacing
  cppcheck

Fixed

• mojibake.h accepts C++17 on MSVC without /Zc:__cplusplus. MSVC reports __cplusplus as
  199711L unless that flag is set, so projects including the header from C++ hit the "C++17 or a
  later version is required" error and lost [[nodiscard]]; the check now also reads
  _MSVC_LANG, which holds the selected standard
• mjb_map_case, mjb_filter, and mjb_normalize now return MJB_STATUS_UNSUPPORTED instead of
  MJB_STATUS_NO_MEMORY when the output encoding cannot represent a result codepoint, matching
  their _into variants
• Documented MJB_STATUS_UNSUPPORTED for every function whose output encoding can reject a
  codepoint, and MJB_STATUS_INVALID_ENCODING for mjb_confusable_skeleton
• CLI: the interactive screen signal handler now restores the cursor and terminal using only
  async-signal-safe calls
• CLI: break now prints the input bytes between the break positions the iterators report, so a
  format or extend character that a rule absorbs, such as U+200B in break sentence "a<U+200B>b",
  no longer replaces itself with a copy of the previous character (÷aab÷)
• Tests: the CLI tests in tests/shell/ now run under CTest; their assert-command.cmake helper
  was missing, so break-json.cmake never ran
• Fixed all clang-tidy findings in the library, CLI, and tests
• Site: the malformed policy is now passed through the API options, so the arguments that follow
  it, such as the normalization form, are no longer shifted
• Site: mjb_convert_encoding now uses the selected output encoding
• Tests: the CollationTest_SHIFTED.txt conformance check compared each string with itself since
  the mjb_collation_key conversion, so it could not fail; it now compares the two strings again