Skip to content

reject out-of-range code points in UTF-32 wide-string input - #5348

Open
Angadi56 wants to merge 1 commit into
nlohmann:developfrom
Angadi56:utf32-eof-sentinel
Open

reject out-of-range code points in UTF-32 wide-string input#5348
Angadi56 wants to merge 1 commit into
nlohmann:developfrom
Angadi56:utf32-eof-sentinel

Conversation

@Angadi56

@Angadi56 Angadi56 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

wide_string_input_helper<BaseInputAdapter, 4>::fill_buffer converts UTF-32 input into the UTF-8 the lexer reads, and any code point above U+10FFFF falls into an "unknown character" branch that passes the code unit straight through. That pass-through narrows a 32-bit unit to int, so the single unit 0xFFFFFFFF arrives as -1, which is exactly std::char_traits<char>::eof(). The lexer reads that as end of input, so the parse stops there and the strict trailing check in parser::parse still succeeds: json::parse(std::u32string(U"[1]") + char32_t(0xFFFFFFFF) + U"anything") returns [1] and json::accept returns true, with every unit after the sentinel silently discarded. I found it while lining this branch up against the sibling UTF-16 helper, whose pass-through is safe only because its units cannot exceed 0xFFFF. The neighboring out-of-range values already behave correctly, since 0x110000 and a lone surrogate both stay positive after the cast and raise parse_error.101, so only the one value that collides with the sentinel is mis-decoded. The fix has that branch emit a byte that is never valid UTF-8 instead of a narrowed unit, which is the same collision input_stream_adapter::get_character already documents guarding against, and it leaves the two in-range casts in the file alone because they are bounded to 0..0x7F. Valid input is unchanged and the regression test goes in the existing invalid std::u32string section.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running make amalgamate.

Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! The change looks good, but please fix the self-cast warning from GCC.

@nlohmann

nlohmann commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Clang was updated and the latest develop branch fixes a CI issue. Please rebase to the latest develop branch.

@nlohmann nlohmann added the please rebase Please rebase your branch to origin/develop label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

M please rebase Please rebase your branch to origin/develop tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants