4.0.2

JVector Release Notes

Version: 4.0.1 - 4.0.2

---

New Features

Added optional batched vector access and a bounded file source for quantizer
training, encoding and existing full-precision/NVQ disk serialization. Capable
algorithms can consume samples or full scans without requiring an upfront preload
of the original base dataset; existing vector sources remain supported.

BenchYAML and AutoBenchYAML support useDatasetPrefetching: true in their index
configuration. The flag defaults to false, preserving resident loading and scoring.
Comparison-heavy uncompressed configurations retain resident input. The existing
jvector.dataset_loader VM option overrides the flag, while configurations
requiring resident raw vectors still preload; an explicit buffered request logs
the reason for that fallback.

See batched vector access for ownership,
resource limits, examples and the benchmark opt-in. Encoded vectors, training
working memory and graph structures retain their existing memory requirements.

---

Google Highway native SIMD is no longer experimental and is preferred by default
on supported Java 22+ runtimes. jvector.enable_native_vectorization is the supported
flag; the former name remains a deprecated compatibility alias.

Successful native selection uses MemorySegmentVectorFloat instead of
ArrayVectorFloat. Consumers should use VectorFloat, or FloatArray.array()
when array access is needed; direct casts and assumptions about get()'s backing
storage must be updated. copyFrom supports both built-in float vector types.
Disable automatic native selection with -Djvector.enable_native_vectorization=false
to retain the previous provider choice. Native operations require provider-created
vectors; copying between storage types does not make mixed-storage arithmetic valid.

Pass --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED
for classpath deployments. Missing native access can produce JVM warnings; denied
native access, an unsupported ABI, a missing ABI query, or missing required library
symbols cause automatic selection to fall
back to the Java provider. Explicitly forcing native fails during initialization.

Java requires ABI 1 and checks the native library's explicit supported set before
calling kernels. Compatible native releases keep the same ABI ID. Libraries that
predate the ABI query are rejected; the bundled library supports ABI 1.

---

Performance Improvements

Native SIMD Acceleration on AArch64 (NEON, SVE, SVE2)

PR: #723

Description

This PR extends JVector's native Highway SIMD backend to AArch64 (64-bit ARM), bringing
the same native acceleration that x86-64 users already have to ARM-based servers and
development machines (e.g. AWS Graviton, Apple Silicon under Linux, Ampere Altra).

AArch64 support was previously absent from the native layer — ARM hosts silently fell back to
the Panama Vector API path. The native binary now ships three AArch64 ISA tiers, all compiled
as vector-length-agnostic (scalable) code:

Tier	Highway target	Notes
NEON	HWY_NEON	Baseline; all AArch64 CPUs
SVE	HWY_SVE	Scalable Vector Extension
SVE2	HWY_SVE2	SVE2 + i8mm + bf16

SVE and SVE2 are compiled as VL-agnostic (scalable) code, so the same binary runs correctly
on any SVE/SVE2-capable CPU regardless of its physical vector length. NEON operates on fixed
128-bit vectors. At JVM startup, the best available tier is selected automatically via CPU
feature detection — identical to the existing CPUID-based dispatch on x86-64.

The following changes were made:

• NativeVectorizationProvider — the hard-coded x86_64 architecture check is extended
  to include aarch64, so the provider is offered on ARM Linux hosts.
• jvector_arch.h — new header introducing JV_ARCH_X86_64 / JV_ARCH_AARCH64
  preprocessor macros; all arch-specific code is guarded with #if/#elif/#endif.
• jvector_cpu_features.h — AArch64 feature probing (NEON, SVE, SVE2 presence and
  vector-length queries) added alongside the existing x86 CPUID path.
• Meson build — three AArch64 ISA variant targets added; SVE/SVE2 targets skip gracefully
  on Clang < 22 or non-Linux hosts where SVE is unavailable.
• Cross-compilation — Meson cross-files and updated READMEs provided for building the
  AArch64 native library from an x86-64 host.
• CI — a dedicated AArch64 GitHub Actions runner added to the matrix so native builds and
  kernel correctness are verified on real ARM hardware on every PR.

Purpose / Impact

Benchmarked on 1M-scale datasets on both AWS Graviton 3 and Graviton 4, Native SIMD (Highway)
consistently outperforms Panama SIMD on AArch64:

• Up to 40% lower search latency at the same recall level

• Up to 29% faster index construction

• All kernels already ported to Highway for x86-64 (FP32 similarity, PQ, NVQ, element-wise
  arithmetic) benefit immediately on AArch64 — no separate ARM implementation was required.

• On AArch64 Linux with native vectorization enabled, the Highway backend is used exclusively
  for all similarity and quantization operations; the Panama Vector API path is bypassed.

• On platforms where the native library cannot be loaded, JVector falls back to the Panama or
  pure-Java provider transparently, as before.

How to Enable

Both libjvector.so variants — x86-64 and AArch64 — are built and bundled into the release
JAR. At JVM startup, NativeVectorizationProvider detects the current architecture and loads
the matching native library automatically. The same flags work on both architectures:

java --enable-native-access=ALL-UNNAMED \
     -Djvector.experimental.enable_native_vectorization=true \
     -jar your-app.jar

To cap the ISA tier for debugging or benchmarking:

JVECTOR_MAX_ISA=neon java --enable-native-access=ALL-UNNAMED \
     -Djvector.experimental.enable_native_vectorization=true \
     -jar your-app.jar

Building releases:

To produce a release JAR with native libraries for all supported
architectures (x86-64 and AArch64), the build must be run with the -Dnative.crossarch
Maven property. Without this flag, only the native library for the current host architecture
is built and bundled.

The release Maven profile (in jvector-native) automatically sets -Dnative.crossarch,
so no extra flag is needed on the command line:

# verify (run before the verification script)
mvn verify -Prelease

# publish
mvn -Prelease clean deploy

Notes

• SVE/SVE2 compilation requires Clang ≥ 22; older compilers silently skip those targets and
  fall back to NEON.
• There is no change to the on-disk index format; AArch64 and x86-64 indexes are fully
  interchangeable.

---

Compaction: level-0 candidates found by scanning cells instead of searching

PR: #740

Description

Compaction merges several pre-built partition indexes into one. At level 0 it must give every node
its nearest neighbours in each of the other partitions, and until now it found them with one
graph search per node per other partition — a walk whose every hop reads a different record from a
different place on disk. Cost scales with nodes times hops, and the reads are scattered, so once the
inputs stop fitting in page cache the merge becomes bound by major faults rather than by
computation.

This change computes the same thing by scanning. Three pieces make it work:

1. Cells. The compactor's own upper-layer merge now runs first, and its merged level-1 nodes
   become cells. Every live node is assigned to the cell its vector descends to.
2. Ordinals in cell order. Output ordinals are assigned cell by cell, so one partition's members
   of one cell occupy consecutive rows of the output, and reading a cell is one sequential read.
3. A shared scan. Each node names the cells it wants, ranked by a width-64 beam over the merged
   top layer and cut off after a fixed budget of 4,096 vectors per larger partition. Those requests
   are then inverted, so a cell is read once per group and scored against every node that
   asked for it, rather than once per node.

Partitions are processed smallest first: a node scans only the partitions larger than its own, and
the reverse direction arrives as a bounded set of offers, so the largest partition scans nothing.

The merge remains exact. Every score that orders a candidate is computed on the vectors themselves.
Approximation appears in exactly two places: which cells a node asks for, and the pairwise
comparisons inside diversity selection, which run on vectors decoded from a wide PQ code. The wide
code measures slightly better there than exact comparisons, because coarse codes collide
near-duplicates and near-duplicate detection is what the diversity test exists to do.

Breaking API change

OnDiskGraphIndexCompactor now assigns its own output ordinals, because locality in the output
ordinal space is what makes a cell's members contiguous and therefore readable in one pass. A
caller-supplied mapping cannot provide that. Both constructors lose their remappers parameter and
the mapping is published afterwards:

// before
new OnDiskGraphIndexCompactor(sources, liveNodes, remappers, similarityFunction, executor);
new OnDiskGraphIndexCompactor(sources, sourceCompressed, liveNodes, remappers, similarityFunction, executor);

// after
new OnDiskGraphIndexCompactor(sources, liveNodes, similarityFunction, executor);
new OnDiskGraphIndexCompactor(sources, sourceCompressed, liveNodes, similarityFunction, executor);

// then, once compact() has run:
List<OrdinalMapper> mappers = compactor.ordinalMappers();

Callers that previously passed OrdinalMapper.OffsetMapper simply drop the argument and read
ordinalMappers() afterwards to learn where each source node landed. oldToNew of a deleted node
is OrdinalMapper.OMITTED; live ordinals are dense.

Also removed: setRefineAfterCompaction and the optional refinement pass it gated.

Both compact(Path) and compact(Path, Path) are unchanged, as is the requirement that every
source carries the INLINE_VECTORS feature. Full-precision, fused-PQ and PQ-sidecar sources are all
still supported, through the same entry points as before.

How to Enable

No configuration required, and no system properties are introduced. The behaviour is fixed at the
measured operating point: merged cells, probe beam width 64, scan budget 4,096 vectors per larger
partition.

The cell join engages when the merged upper layers yield level-1 nodes and the similarity function
is dot product, Euclidean or cosine. If either does not hold — for example merging flat indexes with
no hierarchy — level 0 falls back to the existing per-node graph search and output ordinals fall
back to a PQ-code prefix. The merge is correct either way; only the level-0 strategy changes, and
the compactor logs which path it took.

Results

Three datasets of 6M to 10M vectors, four partitions each, graph degree 32, 64 threads, cold page
cache before every run, no swap, one run at a time on a 32-physical-core host with 495 GB RAM and
NVMe RAID 0. Memory is a cgroup v2 MemoryMax set as a fraction of the on-disk input index size S.
Ratio is S divided by that limit, so ratio above 1 means the index does not fit in the memory the
process may use.

Speedup is the previous implementation's compaction time divided by the new one's.

Dataset	dim	unconstrained	ratio 1.00	ratio 1.33	ratio 1.97
CAP-6M	1536	2.10×	2.54×	est. 16×	est. 47×
DPR-10M	768	2.58×	3.05×	est. 28×	est. 76×
Cohere-10M	1024	2.63×	2.50×	9.72×	est. 40×

Values marked est. are runs where the old implementation was stopped once it had taken ten times
the new one's wall clock; their throughput is measured over the nodes actually completed, so those
figures are lower bounds on the gap. 9.72× is the one above-RAM figure where both implementations
ran to completion.

The mechanism is visible in the fault counts. The old implementation services major faults at a
near-constant rate, so its wall clock is essentially its fault count divided by a constant, and
memory pressure multiplies that count:

Dataset	major faults/node, ratio 1.00	ratio 1.33
CAP-6M	43.1	472
DPR-10M	40.9	582

The cell join moves from 4.07 to 4.26 and from 2.64 to 2.86 over the same step. On CAP-6M at ratio
1.97 both implementations spend almost identical user CPU, 44,764 s against 46,816 s, so they
compute comparable distances; the old one then adds 349,979 s of kernel time servicing page faults
and spends 80% of its wall clock stalled on I/O at 17% CPU utilisation, against 76% for the cell
join.

Throughput, nodes per second, shows the same thing from the other side: the cell join varies by only
1.24× to 1.39× across the entire memory range and tracks vector width rather than available memory,
while the old implementation falls by 20× to 41×.

Quantized inputs

At ratio 1.33, by input layout:

Layout	Speedup	Recall change
fused PQ	2.34× – 4.58×	−0.23 to +1.54 pt
full precision	est. 16× – 28×	see below
PQ sidecar	est. 20× – 47×	not measured

The ordering is explained entirely by whether the old implementation can navigate on codes. Fused
codes are inline and it uses them, so its faults fall to 43–75 per node and the gap narrows.
Full-precision sources have no codes. A PQ sidecar is the worst case for it: codes exist but are
only consulted when written inline, so a sidecar's codes are retrained and rewritten yet never
navigated on, and its faults rise to 474–962 per node.

Notes and trade-offs

Recall. On full-precision inputs the cell join gives up 0.3 to 1.0 points of recall@10. The old
implementation's recall is identical to five decimal places at every memory setting, so it is
deterministic; the cell join varies about 0.4 points run to run, from thread scheduling and a
per-run wide-code retrain, which puts the smallest deficits inside its own noise. On fused inputs
the deficit disappears.

Attribution. The figures above compare against main, which is what users have today, so they
credit this change with everything in the PR rather than the cell join alone. Measured against the
same branch with the cell join switched off and its other changes kept, the cell join is 1.4× to
1.6× slower in RAM: most of the in-RAM gain comes from processing partitions smallest-first with
bounded reverse offers, which halves the number of searches. The cell join's own contribution is the
above-RAM columns, where it converts scattered reads into sequential ones.

Scope. Every behavioural change is confined to the compaction path. Outside it, four files are
touched and all four are additions only: a dotProductMulti kernel added to VectorUtilSupport as
a default method, so existing implementors are unaffected, its Panama override, a static entry
point on VectorUtil, and three read-only getters on ProductQuantization. Eighty-four lines added,
none removed, and no existing non-compaction code path is modified or re-routed.

Measurement scope. All figures are a single run per configuration; repeats were not taken. Where
the old implementation was stopped the effect sizes are above 10×, far outside the run-to-run
variation observed elsewhere, but the completed in-RAM and ratio-1.00 comparisons should not be read
to two decimal places.

See docs/compaction.md for the pipeline description.

---

Deferred batch graph serialization

PR: #747

Parallel graph serialization can now write complete layer-zero records in bounded
batches of 64, with an optional positive batch-size override. When the parallel writer
is selected, Grid supplies inline NVQ or full-precision features during final
serialization, after graph cleanup, and uses buffered vector ranges when the input
source supports them. With the sequential writer selected, Grid keeps encoding those
features during insertion.

The core writer accepts application-supplied feature states independently of the
benchmark harness or dataset prefetching. Complete-record batching applies equally
to fully preloaded inputs; the source does not have to preload the whole dataset.

Integrators should use the parallel writer when deferring NVQ/FP encoding to
final serialization. A sequential final writer performs that encoding on one
thread and can regress construction time. Sequential callers should retain
encoding during insertion and writeFeaturesInline.

The existing V6 layout, readers, ordinary vector sources, and incremental
writeFeaturesInline API remain supported. Prewritten features are preserved.
Concurrent feature suppliers must keep their input source available until writing
finishes. BenchYAML and AutoBenchYAML default to parallel writing, with
-Djvector.parallel_writer=false selecting the sequential path. The library
writer default remains unchanged.

Recorded construction studies with dataset prefetching reduced ample-memory final
write time from 136.10 to 50.69 seconds on MPNet 10M, 148.50 to 64.02 seconds on
Cohere 10M, and 162.64 to 51.76 seconds on CAP 6M.¹ These are separate fresh runs,
not a universal throughput or memory guarantee. See
Deferred batch serialization for usage.

Construction tutorials and examples now demonstrate final feature suppliers for
deferred FP/NVQ writing, with per-worker source views and an explicit source-lifetime
requirement. The query algorithms and quantization settings are unchanged.

---

Extend Native SIMD Coverage for Vector Operations

PR: #753

Description

Move minimum and sum reductions, in-place scaling, and vector and scalar
subtraction from Panama Vector API fallbacks to native Google Highway kernels,
using the existing x86-64 and AArch64 ISA dispatch.

Subtraction ranges and input lengths are validated before processing.
Minimum reduction preserves Java’s NaN propagation and signed-zero behavior.

How to Enable

These operations use the existing experimental native-vectorization setting.
No new configuration is required.

Notes

Hamming distance continues to use the Panama Vector API implementation.
No public API or index-format changes.

---

Batch Native Highway Operations

PR: #761

Description

Reduce Java/native calls and temporary allocation during PQ training, PQ encoding,
and NVQ processing. Batch closest-centroid scans and KMeans updates, use lazy
array-backed native views, batch NVQ growth-rate search, and keep cosine results
packed.

How to Enable

No new configuration is required. These optimizations apply when the native
provider is selected, which is the default on supported Java 22+ runtimes.

Notes

Existing native function signatures and ABI 1 are unchanged. Four additional
native kernels are included in the bundled library; deployments supplying their
own native library must include these kernels. No index-format changes.

---

Bug Fixes and Issue Resolutions

Enforce Neighbor-ID Uniqueness

PR: #738

Description

Asymmetric scoring, such as A · quant(B) ≠ B · quant(A), can assign different scores to the same neighbor, defeating score-local duplicate checks and reducing effective graph degree.

NodeArray now checks IDs independently of scores and deduplicates complete candidate merges, retaining existing scores and score ordering. ConcurrentNeighborMap also deduplicates initial candidate batches and rejects duplicate offers before copying or changing pruning state.

For an existing neighbor, subsequent backlink offers are rejected regardless of their score; the stored edge score is retained. During concurrent construction, which score is installed first can depend on thread timing.

More distinct edges can improve recall but increase traversal work and reduce QPS at fixed search settings. Results vary by dataset and construction seed.

How to Enable

Automatic; no configuration required.

Notes

Library API and index format are unchanged. A full rebuild is required to remove previously stored duplicate neighbor IDs: loading an existing index or incrementally adding nodes does not guarantee that all old neighborhoods are repaired. Supersedes #522.

---

Publish S3 Dataset Downloads Only After Completion

PR: #745

Description

Download S3 dataset files into private temporary files and atomically publish the
cache path after successful completion and content-length validation. This prevents
other benchmark processes from reading an in-progress download or losing a completed
cache file when another download attempt fails.

How to Enable

No configuration is required. This applies to S3 downloads made by the example
benchmark dataset loader; HTTP downloads already use atomic publication.

Notes

Normal failures remove their temporary files. A hard process termination may leave
an unreferenced temporary file, which is not accepted as cached data. Existing cache
files are not retroactively validated; remove any files known to come from an
interrupted download before reusing that cache.

---

Fix Hierarchical Search Failure with Non-Normalized Dot Product Vectors

Description

Resolves #713. When searching a hierarchical graph with VectorSimilarityFunction.DOT_PRODUCT and vectors that are not unit-normalized, search could return no matches for an upper layer and fail with AssertionError in GraphSearcher.internalSearch.

The descent through layers above level 0 searched each layer with a threshold of 0.0f, treating it as "accept everything". That only holds for similarity functions bounded to (0, 1] (COSINE and EUCLIDEAN). DOT_PRODUCT is only bounded when vectors are unit length; with larger-magnitude vectors, scores can be negative. If every candidate reachable in an upper layer scored below 0, all were discarded, no entry point was found for the next layer, and the assertion fired. This surfaced in Cassandra's VectorDotProductWithLengthTest.testTrueDotproduct, which uses non-unit 2D vectors with components in [-100, 100].

Upper-layer descent now uses a threshold of Float.NEGATIVE_INFINITY. The search(scoreProvider, topK, acceptOrds) overload, which has no threshold parameter, now also defaults to Float.NEGATIVE_INFINITY instead of 0.0f.

How to Enable

Automatic; no configuration required.

Notes

Results from COSINE, EUCLIDEAN, and unit-normalized DOT_PRODUCT searches are unchanged. For non-normalized DOT_PRODUCT, the search(scoreProvider, topK, acceptOrds) overload can now return results with negative scores that were previously filtered out. Callers who pass an explicit threshold and need to accept every result for non-normalized DOT_PRODUCT should pass Float.NEGATIVE_INFINITY rather than 0. Index format and library API are unchanged. Regression coverage is in TestHierarchyDotProductSearch.

---

Testing Enhancements

Optional Compaction Benchmark and Dedicated Runner for Regression Benchmarks

PR: #736

Description
Two changes to the automated regression benchmark (AutoBenchYAML / run-bench.yml):

• CompactionBench is now opt-in. AutoBenchYAML previously ran CompactionBench for every dataset after the main benchmark. Compaction benchmarking now runs only when a dataset's YAML configuration enables it with the new top-level withCompaction parameter. When it is not enabled, the step is skipped and logged, and the -compaction.csv / -compaction.json output files are not produced.
• Benchmarks run on the large runner. The test-avx512 job in run-bench.yml previously matched any self-hosted runner with the isa-avx512f label, so results could come from different machines between runs. The job now also requires the jvector-runner-large label, so every benchmark runs on the same hardware and results are comparable across runs.

How to Enable
Compaction benchmarking is disabled by default; no existing YAML files were changed. To enable it for a dataset, add withCompaction at the same level as yamlSchemaVersion and dataset:

yamlSchemaVersion: 1
onDiskIndexVersion: 6

dataset: cap-10M
withCompaction: [Yes]

construction:
  ...

If the parameter is absent or set to [No], CompactionBench is not run. When --config is passed to AutoBenchYAML, that single file controls compaction for all datasets.

The runner change requires no configuration in the repository, but the self-hosted runner must carry the custom label jvector-runner-large, as runs-on matches labels rather than runner names.

Notes

• Used internally by JVector benchmarking, no client impact.
• With both branch runs of a PR pinned to the same runner, they now execute sequentially rather than potentially in parallel on separate runners.

---

1. GCE VM, 48-core Intel Xeon 6985P-C (96 threads), local SSD RAID10, JDK 23, 128 GiB heap; native Highway, 48 workers. ↩