[Roadmap] Incremental Rust modernization for HugeGraph core modules

[imbajin]

Goal

Explore an incremental Rust implementation for selected HugeGraph Server, HStore, and PD components, especially performance- or memory-sensitive paths where Rust can provide measurable safety and efficiency benefits.

This is a parent tracking issue, not a commitment to rewrite the whole server. The current implementation remains the compatibility baseline, and Java/Rust components may coexist behind stable process, protocol, or storage boundaries.

Roadmap at a glance

Toolchain foundations
        │
        v
┌─ This repository ─────────────────┐
│  Server core -> HStore -> PD      │
└───────────────────────────────────┘
        │
        v
Computer kernels -> Vermeer/OLAP

Each arrow is a review gate, not an automatic rewrite.

Proposed priority

1. Server core
   • Begin with bounded leaf components, memory management, codecs, or performance-sensitive data structures rather than the whole request path.
   • Preserve REST/Gremlin behavior, authentication, observability, and plugin contracts.
2. HStore
   • Evaluate storage/replication paths only after workload baselines, on-disk and wire compatibility, failure recovery, and rolling-upgrade requirements are explicit.
3. PD
   • Evaluate scheduling and metadata components after their distributed-system invariants and compatibility contract are covered by repeatable tests.

The sequence may change based on benchmark evidence and maintainer consensus.

Delivery model

proposal/RFC -> isolated component -> differential/fault tests
             -> benchmark -> opt-in deployment -> staged adoption

Planning guardrails (required)

Before implementation starts, each child issue must define:

• scope, owner/mentor, and non-goals;
• API, protocol, storage-format, and operational compatibility;
• differential, concurrency, fault-injection, and upgrade/rollback tests;
• reproducible latency, throughput, memory, and recovery benchmarks;
• security/dependency checks and supported platforms;
• observability, deployment, coexistence, migration, and rollback plans;
• explicit acceptance criteria and user/operator/contributor documentation.

No large rewrite PR should start before its design and acceptance criteria are reviewed. Correctness, recoverability, and compatibility take precedence over headline benchmark numbers. Small, reviewable PRs and stable interoperability boundaries are required; an unplanned rewrite PR may be closed or asked to return to the RFC/design stage.

Candidate work items

• Select the first small server-side proof of concept
• Document stable Java/Rust interoperability boundaries
• Build shared differential and compatibility test fixtures
• Establish representative OLTP and storage workload baselines
• Define CI, packaging, security, and release conventions
• Define rolling-upgrade and rollback requirements before HStore/PD work
• Split approved work into newcomer-friendly child issues

This checklist will link to concrete child issues as proposals are accepted.

Project-wide roadmap

• Toolchain: [Roadmap] Incremental Rust modernization for HugeGraph Toolchain hugegraph-toolchain#748
• Server / HStore / PD (this issue): [Roadmap] Incremental Rust modernization for HugeGraph core modules #3110
• Computer / Vermeer: [Roadmap] Incremental Rust modernization for graph computing hugegraph-computer#355
• Documentation hub: [Roadmap] Documentation hub for HugeGraph Rust modernization hugegraph-doc#462

How to participate

Interested in open source, Rust, databases, distributed systems, storage engines, or infrastructure? Comment with the area you want to work on and any relevant experience. RFC review, failure-model analysis, benchmarks, tests, and documentation are all valuable.

First-time contributors can start with the HugeGraph contribution guide and the repository's good first issue / help wanted tasks.

[dpol1]

I'd be interested in taking the interop/build/test foundation work before proposing a server component.

Fair warning, I'm not a Rust person. My work here has been compatibility and failure handling (#3062). Where I'd be useful is the build, test and licensing plumbing someone has to do anyway.

Poked around a bit first. Few things that might save time for whoever picks this up:

• root pom is JDK 11, so there is no stable FFM API at the project baseline. JNI is the obvious in-process baseline; an out-of-process boundary is the other architectural option.
• hugegraph-commons is still Java 8, so a native seam there has a tighter floor than one under hugegraph-server.
• HugeGraph now supports x86_64, aarch64 and riscv64. Any eventually distributed native component will need an explicit support/fallback policy; feat(server): support build & use(RocksDB) on RISC-V #3102's smoke-test infrastructure looks reusable.
• Apache OpenDAL already ships a Rust JNI binding: exec-maven-plugin for the cargo build, classifier jars via os-maven-plugin, crate licenses in a DEPENDENCIES.rust.tsv. Not saying we copy it, but the release-side questions have an answer somewhere. One catch: their non-classifier jar simply has no native lib in it, it's not a Java fallback, the loader throws if nothing loads. Whether we want graceful degradation instead is a real decision. Worth noting the root pom already pins os.detected.classifier to osx-x86_64 on ARM Macs, so anything keyed on that property would need handling there.

On picking the first PoC, I'd rather not name a component. A small matrix (test oracle? how many call sites? work per call? does it touch a persisted format?) plus a rule for when a JNI hop pays for itself makes it easy to move on when the first pick doesn't hold up.

Small first thing, no Rust until there's agreement: a compat inventory of the codec classes, a golden fixture corpus in plain Java that can merge on its own, and an interop draft (JNI seam, failure semantics, packaging) as a child issue.

Questions: does this fit the toolchain -> server ordering, or should it wait behind apache/hugegraph-toolchain#748? Park packaging for now, or is the classifier jar shape what you'd want? And is anyone up for mentoring / picking the first component?