fix(javascript): align TypeMeta preamble constants with python/java/rust/go xlang bindings#3603
Merged
chaokunyang merged 1 commit intoApr 22, 2026
Conversation
…ng bindings
@apache-fory/core's NAMED_COMPATIBLE_STRUCT output was not byte-
compatible with pyfory, fory-java, fory-rust, or fory-go because three
constants in the 8-byte TypeMeta preamble diverged from every other
binding:
constant | js (before) | py/java/rust/go | location
NUM_HASH_BITS | 41 | 50 | TypeMeta.ts:43
COMPRESS_META_FLAG | 1 << 63 | 1 << 9 | TypeMeta.ts:40
HAS_FIELDS_META_FLAG | 1 << 62 | 1 << 8 | TypeMeta.ts:41
And the hash value in prependHeader was read from the 128-bit
MurmurHash3 output as an UNSIGNED BigInt (constructed from two uint32
halves), while pyfory/java/rust all treat the same bytes as a SIGNED
int64 (`hash_buffer()[0]` unpacks `int64_t[0]` in python,
`murmurhash3_x64_128(...)[0]` returns `long` in java, `.0 as i64` in
rust). Since unsigned BigInt is never negative, the subsequent
`abs()` was effectively a no-op and the hash bits that ended up in
the int64 header differed from every other binding whenever the hash
result had its high bit set.
After this fix, the same logical struct produces byte-for-byte
identical 8-byte TypeMeta preambles across all five xlang bindings.
Verified with a minimal Point(x, y) round-trip at 0.17:
pyfory.Fory(xlang=True, ref=True, compatible=True)
+ register_type(Point, namespace='demo', typename='Point')
Fory.builder().withLanguage(Language.XLANG).withRefTracking(true)
.withCompatibleMode(CompatibleMode.COMPATIBLE).build()
new Fory({ ref: true, compatible: true })
all produce:
02 00 1e 00 2a 81 a9 bc 9f 33 15 20 23 15 31 12 ... (identical)
and `pyfory.deserialize(javaBytes)`, `pyfory.deserialize(jsBytes)`,
`fory.deserialize(pyBytes)` round-trip cleanly.
No public API change; only the wire bytes change (fixing them).
chaokunyang
approved these changes
Apr 22, 2026
Collaborator
chaokunyang
left a comment
There was a problem hiding this comment.
LGTM, thanks for fixing this bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
@apache-fory/core'sNAMED_COMPATIBLE_STRUCTTypeMeta preamble is not byte-compatible with pyfory, fory-java, fory-rust, or fory-go. For the same logical struct, the JavaScript binding emits an 8-byte int64 header that no other binding can read. I noticed this as part of issue #3602.With this patch my cross-language tests pass, but I don't know if this is entirely correct — I'd appreciate a deeper review from someone who knows the TypeMeta spec better than I do (especially around the signed-vs-unsigned hash interpretation in
prependHeader).What does this PR do?
Aligns four constants / behaviours in
javascript/packages/core/lib/meta/TypeMeta.tswith what every other xlang binding does at 0.17:NUM_HASH_BITS4150python/pyfory/meta/typedef.py:37,java/.../meta/TypeDef.java:77,rust/fory-core/src/meta/type_meta.rs:76,go/fory/type_def.go:35COMPRESS_META_FLAG1n << 63n1 << 9HAS_FIELDS_META_FLAG1n << 62n1 << 8prependHeaderBigIntbuilt from twouint32halves viagetUint32(0, false) << 32n | getUint32(4, false)int64int64_t[0], fory-javamurmurhash3_x64_128(...)[0]returnslong, rust.0 as i64On the hash read specifically: reading the same 8 bytes as unsigned
BigIntnever produces a negative value, so the subsequentabs()is effectively a no-op. Whenever the hash's high bit is set, the resulting header diverges from what the other bindings emit for the same struct. The patch useshash.getBigInt64(0, false)(signed read) followed by explicit arbitrary-precisionabs()+ 63-bit mask, mirroring pyfory'sabs(hash) & 0x7FFFFFFFFFFFFFFF.Empirical reproduction (fory-core 0.17.0 on every binding, matching config
xlang=true, ref=true, compatible=true, NAMED_COMPATIBLE_STRUCT via(namespace, typename)registration):Before this PR:
02 00 1e 00 10 01 d2 92 ce 5f 2b 73 22 0d 0c 8c 70 13 bd c8 6c c0 40 05 5c 40 05 60 14 28(30 bytes)02 ff 1e 00 10 00 00 ad 86 c0 98 d5 23 15 31 12 92 1c d0 2d f6 53 04 e9 2e c4 92 7b 9b 22 00 58 07 …The field-descriptor and value bytes align once you get past the preamble, but the 8-byte int64 header and the byte-1 reference flag diverge.
pyfory.deserialize(jsBytes)silently returnsPoint(x=0, y=0)(every field unmatched, falls through to defaults);fory.deserialize(jsBytes)in Java throwsDeserializationException: read objects are: [null].After this PR: javascript produces byte-identical output to python / java / rust / go, and each binding can decode every other binding's bytes. Ran manual round-trip against both pyfory 0.17 and fory-java 0.17 with a Point struct and with a richer struct containing strings, a
list<string>, and int/float fields — both succeed.Related issues
AI Contribution Checklist
yes, I included a completed AI Contribution Checklist in this PR description and the requiredAI Usage Disclosure.yes, my PR description includes the requiredai_reviewsummary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.Does this PR introduce any user-facing change?
@apache-fory/coreclients communicating only with each other will continue to work (same-binding output still round-trips). Any persisted JS-produced bytes, or in-flight messages relying on JS-specific preamble, will no longer be readable. Given cross-binding interop was broken on 0.17 anyway, practical impact should be small.Benchmark
Not applicable — constant alignment with no hot-path change.