[Proposal] Java compatibility fixtures and PoC selection matrix for the Rust modernization groundwork

[dpol1]

Follow-up to the discussion in #3110. @imbajin confirmed the testing and validation groundwork can start before any Rust decision, so this issue tracks the Java-only part: a selection matrix for the first PoC candidate and golden compatibility fixtures for the codec classes.

Scope

Three deliverables, all plain Java against the current build:

1. A selection matrix for PoC candidates (first cut below, to be refined in this thread).
2. A golden fixture corpus for LongEncoding plus a replay test that runs it against the current implementation.
3. Short written notes on the behavioral contract the fixtures lock down (edge cases, error behavior, invariants).

Non-goals

No Rust, no JNI, no Cargo, no packaging changes, no new Maven modules, no runtime wiring. Those wait for the component decision, a mentor, and a design review with contributors who have deeper Rust experience, as discussed in #3110.

Selection matrix

Criteria: an existing test oracle, enough call sites to matter, work per call big enough to survive a JNI crossing, and, following the maintainer feedback in #3110, keeping the Java implementation alive has to stay cheap: graceful fallback is the maintainer-preferred direction, which this matrix adopts as a selection criterion.

Candidate	Oracle	Fallback cost	Notes
LongEncoding	LongEncodingTest (684 lines)	low, pure functions	output feeds storage keys, so the contract needs locking either way
NumericUtil	NumericUtilTest (475 lines)	low	coupled to LongEncoding.encodeNumber, candidate for the same corpus later
BytesBuffer	BytesBufferTest (1351 lines)	high	persisted format, 30+ callers, a fallback would have to stay byte-identical; the store's read path structurally parses server-encoded values via the struct copy, so the shared format surface spans ids, vints and property encoding (thread)
IntSet / IntMap	IntSetTest, IntMapTest	high	concurrent structures on Unsafe, a live double implementation risks divergence

With the fallback criterion the concurrent structures and the serializer drop out of round one. LongEncoding is the first fixture target. That is a choice about where to write fixtures first, not a decision that it becomes the Rust PoC.

Fixture shape

Plain ASCII, line-based (TSV with \uXXXX escapes for non-ASCII), checked into test resources so it needs no extra dependency to read. It covers: round trips across the full value range including Long.MIN_VALUE/MAX_VALUE (the encoding does intentional wrapping arithmetic), arbitrary UTF-16 input on the decode side (valid inputs are ASCII, invalid ones are not), the error cases (IllegalArgumentException vs NumberFormatException vs NPE on null), and the length limits.

The replay test is JUnit 4 and compiles at the Java 8 level, since that is the current hugegraph-commons baseline. Root is Java 11. Java 17 / commons Java 11 from #3117 are treated as declared targets, not assumptions.

Acceptance

• matrix reviewed in this thread
• corpus + replay test merged and running in the normal test suite
• contract notes written up here or in the PR

[dpol1]

Contract notes for the LongEncoding corpus (PR #3146), the invariants worth knowing before porting or refactoring:

Alphabet and ordering. B64_SYMBOLS = 0-9 A-Z _ a-z ~, 64 chars in ascending ASCII order. For canonical encodeSortable outputs, plain string comparison gives numeric order: negatives (prefix 0) sort before positives because positive outputs always start with a length char >= 1, the length character orders different magnitudes within one sign, and the digits decide between encodings of equal length.

Length symbols. LENGTH_SYMBOLS (0-9A-F) is exactly the first 16 characters of B64_SYMBOLS, and decodeSortable reads the declared length through B64_SYMBOLS.indexOf, so the prefix relationship is load-bearing for decoding: declared lengths from 0 to 63 are structurally accepted. Declared length 0 is reachable only in the negative form "00", since the initial str.length() >= 2 check rules out a bare positive length char.

Wrapping is intentional. encodeSortable maps negatives by num += Long.MIN_VALUE (overflow on purpose) and decodeSortable reverses it with value -= Long.MIN_VALUE; the digit loop accumulates with multiply/add that wrap silently. A port must use wrapping arithmetic (e.g. wrapping_add/wrapping_sub/wrapping_mul in Rust), not checked arithmetic.

Decode accepts more than encode produces. Structurally valid input goes up to 64 chars in the positive form and 65 with the negative prefix, and the accumulation wraps rather than failing. Concrete case: decodeSortable("00") returns Long.MIN_VALUE, while encodeSortable(Long.MIN_VALUE) produces "010". Round-trip equality holds encode->decode, not decode->encode.

Empty-string edge. decodeB64("") = 0, decodeSignedB64("") = 0, decodeSignedB64("-") = 0. No exception.

Specials. encodeSignedB64(Long.MIN_VALUE) returns the literal "-80000000000" (the b64 of 2^63). encodeB64 rejects any negative input with IllegalArgumentException.

Error taxonomy. IllegalArgumentException for argument/structural precondition failures (input shorter than 2 chars, declared length not matching the actual length, negative input to encodeB64); NumberFormatException for any character outside the symbol table during digit decoding, including non-ASCII BMP characters and surrogate code units; NullPointerException on null. Exception classes are contract; messages deliberately are not.

Domains. Encode outputs are always pure ASCII. Decode input is arbitrary UTF-16; the corpus exercises representative inputs on that side (non-ASCII BMP chars, surrogate pairs, lone surrogates), not an exhaustive sweep.

[dpol1]

Compatibility inventory, first pass. Analysis snapshot: master c10779dc6. Counting rule: production sources only — test code, generated sources, and the bundled dist copy are excluded. File:line evidence exists for every cell, happy to expand any row.

Candidate (module)	Wired production surface	Persisted or wire boundary	Existing oracle	Java fallback / PoC risk
LongEncoding (commons)	encodeNumber, encodeSignedB64, decodeSignedB64, validB64Char in 7 files; decodeSortable/decodeNumber have no production callers today	encoded strings embedded in vertex ids (primary-key strategy), edge sort keys, index ids and values, stored-string ids; the same contract reaches the store side via the struct copies of IdGenerator/ConditionQuery/PropertyKey	LongEncodingTest, golden corpus (PR #3146)	fallback cheap: final class, static pure functions, no state
NumericUtil (commons)	sortable conversions and byte helpers in 20 files	range-index row keys, LongId bytes, HBase counter cells; algorithm derived from Lucene NumericUtils, effectively frozen	NumericUtilTest; the corpus pins it only transitively via encodeNumber	fallback cheap: static, stateless; the byte-array surface has no golden corpus yet
BytesBuffer (server/core)	serializer stack, 26 files	the binary row format itself for every backend, raft log entries, client-visible page tokens; non-primitive property values embed Kryo bytes	BytesBufferTest, BinarySerializer round-trip tests	high: byte layout is the on-disk format, process-wide config state, Kryo coupling
BytesBuffer (hugegraph-struct, consumed by hstore modules)	store read/query paths, 16 files	gRPC ByteString id fields (wire) and RocksDB keys and values on the store side (disk)	none — no test imports this copy	high, and currently uncharacterized
IntSet/IntMap (server/core)	traversal records only, 4 files	none: per-request, in-heap	IntMapTest, IntSetTest, JMH benchmark	state cheap, but the concurrency contract is real (concurrent=true in K-neighbor/K-out) and the implementation sits on sun.misc.Unsafe, which is on the JEP 471 removal path

Findings that actually move the selection:

1. LongEncoding's wired surface is narrower than its API: production never calls decodeSortable/decodeNumber. The corpus still locks them (they are public contract), but a PoC can be judged on the four wired operations.
2. NumericUtil's byte-array surface (numberToSortableBytes and friends, used in range-index keys) is the natural next corpus once the shape of test(commons): add golden fixture corpus for LongEncoding #3146 settles.
3. Facts about the two BytesBuffer copies, recorded separately: the struct copy declares ID_LEN_MAX = 32768 but its writeId enforces a literal 16384 and the declared constant is referenced only in a comment; the core copy declares ID_LEN_MAX = 16384; BLOB_LEN is 5 in core and 4 in struct; git history shows the struct file was extracted from the core one in 2025-09 while the core copy kept evolving (latest 2026-06, feat(server) : make buffer max capacity configurable #3049); no test currently locks the struct copy. Whether the two copies must interoperate on the same byte format is one of the open points below. Until that contract is stated, the safe deliverable is a characterization fixture per copy, not a cross-copy equality test.
4. IntSet/IntMap: only 4 production call sites and transient state, so porting value looks low; the direct sun.misc.Unsafe usage matters for JDK upgrades independently of any Rust work. Side note: IntMapByDynamicHash has tests and a JMH benchmark but no production wiring.

Open points I could not settle from the tree — input welcome, and @dosu feel free to take a first pass:

• whether hugegraph-server's core BytesBuffer and the hstore struct copy ever need to read each other's bytes, or the two formats live behind separate boundaries;
• whether the store ingest path writes struct-BytesBuffer-encoded bytes, or only re-encodes on read and query;
• whether external tools (loader, hubble, tools) parse or generate the binary row format;
• whether stored Kryo-encoded property values are guaranteed readable across a Kryo upgrade, and whether page tokens are expected to survive a server upgrade.

I'll fold the answers into the matrix as they come.

Update (2026-08-09): the reply below answers the four points, and the mechanical claims check out against the tree (HstoreTable.insert forwards byte arrays unchanged; encoding is server-side; loader/hubble/tools stay on the client/REST layer; new Kryo() with no version header, pinned via TinkerPop 3.5.1; page tokens are ephemeral cursors). One design-level correction after reading the store code directly: the shared surface between the two BytesBuffer copies is wider than string ids. On the store's filtered-read path the struct copy structurally parses and re-encodes server-written column values — SelectIterator.select() does readId/readVInt/readProperty(dataType) and rewrites with writeId/writeVInt/writeProperty (hg-store-core), and BusinessHandlerImpl.scanIndexToElementId re-encodes element ids parsed from server-written index columns. So the struct copy is effectively a decoder of the core copy's output for ids, vints, the cardinality/type byte and per-type property values on that path. Whether the two copies are intended to track one format remains a maintainer question; until stated, per-copy characterization fixtures are still the deliverable, and where the read path overlaps they double as cross-copy compatibility evidence.

[dosubot]

Nice inventory. Here's a first pass on your four open points, traced through the current tree:

1. Core BytesBuffer vs struct BytesBuffer — do they need to read each other's bytes?

No, not at the gRPC boundary. The server's BinarySerializer encodes mutations using the core BytesBuffer into BackendColumn.name/BackendColumn.value byte arrays [1]. HstoreTable.insert() passes those byte arrays unchanged to the session/store [2]. The gRPC payload itself is an opaque bytes data field on StoreCommandRequest — no field-level protobuf structure for IDs or properties [3].

The struct BytesBuffer is used on the store side for its own key/query construction (prefix building, ID reading on the query path), not for re-encoding mutation payloads that crossed the wire [4] [5]. So the two copies live behind separate boundaries. The encoding logic for string IDs is compatible (same 0x80/0xc0 prefix-bit scheme, same effective 16384-byte limit enforced in writeId) [6] [7], but they don't cross-decode each other's output at the gRPC layer.

That said, the store does read bytes that the server encoded — so format compatibility matters for the string-ID encoding path even though the two BytesBuffer classes don't call each other. A characterization fixture per copy (as you proposed) is the right deliverable; a cross-copy equality test would only be meaningful for the shared string-ID encoding subset.

2. Store ingest path — does struct BytesBuffer encode on write, or only on read/query?

Encoding happens on the server side. The core BinarySerializer produces fully encoded BackendEntry columns , HstoreStore.mutate() forwards BackendAction objects without re-serializing [8], and the session passes byte arrays straight to graph.put() [9]. The struct BytesBuffer is used in the store for read-side and query-side key construction [10], not for encoding ingest payloads.

3. External tools — do loader/hubble/tools parse or generate the binary row format?

No. All three go through the client/REST API layer:

• The loader reads source records, converts to graph elements, and inserts via context.client().graph() [11] [12].
• Hubble communicates through HTTP/REST [13].
• Tools constructs a HugeClient from URL + graph name [14].

None import or use HugeGraph's BytesBuffer. The binary row format is internal to the server ↔ store boundary.

4. Kryo property values and page tokens across upgrades

Kryo: There's no explicit cross-version compatibility mechanism. The Kryo instance is a bare new Kryo() plus a UUID serializer, pinned transitively through TinkerPop 3.5.1 [15] [16]. Generic properties use writeClassAndObject()/readClassAndObject() with no application-level version header [17], and the read path has no fallback for incompatible data [18]. A Kryo or TinkerPop upgrade should not be assumed safe for stored generic property values without testing representative persisted data.

Page tokens: These are ephemeral, client-carried cursors — Base64-encoded PageState containing a byte-array position, offset, and total count [19] [20]. They survive normal restarts but there's no explicit cross-version compatibility guarantee [21]. Clients should be prepared to restart pagination from the first page after an upgrade.

Summary for the matrix: The two BytesBuffer copies live behind separate boundaries (server encodes, store reads), external tools don't touch the binary format, Kryo compatibility is unguaranteed, and page tokens are ephemeral. These findings reinforce that a per-copy characterization fixture is the right shape for BytesBuffer, and that LongEncoding → NumericUtil remains the natural fixture ordering since those are the codec paths with the cleanest boundaries.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.