Skip to content

fix(variant): sort object keys by UTF-8 bytes like Java - #789

Merged
JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/variant-object-key-utf8-order
Sep 4, 2026
Merged

fix(variant): sort object keys by UTF-8 bytes like Java#789
JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/variant-object-key-utf8-order

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Variant object keys are ordered with encode_utf16(), but Java orders them by
unsigned UTF-8 bytes: GenericVariantBuilder.FieldEntry.compareTo delegates to
BinaryString.compareTo, which compares (b & 0xFF) per byte.

The two disagree when a non-BMP key meets one in U+E000..=U+FFFF — UTF-16 sorts
the surrogate 0xD8xx first, UTF-8 sorts the 4-byte sequence last. A Java-written
object with, say, an emoji key beside a fullwidth key then fails to read with
Malformed Variant object key order, and the writer emits an order Java rejects.

Rust str is already UTF-8, so comparing the bytes is Java's comparison.

@JingsongLi
JingsongLi merged commit ced0c86 into apache:main Sep 4, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants