4.0.0-rc.9

JVector Release Notes

Version: 4.0.0-RC.8 - 4.0.0-RC.9

---

New Features

Streaming N:1 Graph Index Compaction

PR: #659

Description
OnDiskGraphIndexCompactor merges multiple on-disk HNSW graph indexes into a single compacted index. This targets write-heavy workloads where data is continuously ingested into small segment indexes that accumulate over time; periodically compacting those segments into one larger index improves search throughput and recall without rebuilding from scratch.

Each source is an OnDiskGraphIndex paired with a FixedBitSet marking its live (non-deleted) nodes. The compactor merges all live nodes into a single graph, remaps each source's ordinals into a contiguous global ordinal space via user-supplied OrdinalMappers, and rebuilds each node's neighbor list using Vamana-style diversity selection over candidates gathered from the node's own source and from graph searches in the other sources. Sources are processed in parallel, batch-by-batch, with a bounded backpressure window so memory stays flat regardless of dataset size.

Purpose / Impact

• Consolidates many small segment indexes into one larger index, improving search throughput and recall without a full rebuild
• Excludes deleted nodes from the output, reclaiming space as part of the merge
• Bounded heap footprint — compaction streams per-source batches in parallel rather than materializing the full merged graph in memory. All benchmark datasets compact under -Xmx5g, including a 10M-vector, 2560-dimension dataset
• Measured recall is on par with or better than building a single index from scratch across a range of datasets

How to Enable
Construct an OnDiskGraphIndexCompactor from the source indexes, their live-node bitsets, and an ordinal remapper per source, then call compact():

List<OnDiskGraphIndex> sources = List.of(index0, index1, index2);

// Mark all nodes live (no deletions)
List<FixedBitSet> liveNodes = sources.stream()
    .map(s -> { var bs = new FixedBitSet(s.size()); bs.set(0, s.size()); return bs; })
    .collect(toList());

// Sequential ordinal remapping: source[s] node i -> global offset[s] + i
int offset = 0;
List<OrdinalMapper> remappers = new ArrayList<>();
for (var src : sources) {
    remappers.add(new OrdinalMapper.OffsetMapper(offset, src.size()));
    offset += src.size();
}

var compactor = new OnDiskGraphIndexCompactor(
    sources, liveNodes, remappers,
    VectorSimilarityFunction.COSINE,
    /* executor= */ null                  // null = use the core JVector thread pool
);

compactor.compact(Path.of("compacted.index"));

Deleted nodes are dropped by clearing their bit in the corresponding FixedBitSet and remapping only the surviving ordinals (for example with OrdinalMapper.MapMapper).

Notes

• The public compactor API is marked @Experimental and may change in a future release.
• Quantization is preserved across the merge. For FUSED_PQ sources the PQ codes are stored inline and the codebook is retrained on the combined dataset; non-fused sources can supply a sidecar CompressedVectors that is retrained and rewritten as a single merged file.
• PQ retraining uses balanced proportional sampling across all sources (up to ProductQuantization.MAX_PQ_TRAINING_SET_SIZE vectors total, at least 1000 per source).
• A new CompactorBenchmark (in benchmarks-jmh) measures compaction time, heap footprint, recall, and search latency. See docs/compaction.md for the full algorithm description and benchmarking instructions.

---

Performance Improvements

Fixes #629.

This pull request simplifies AVX-512 feature detection logic in the jvector_simd_check.c file by replacing manual CPUID bit checks with built-in compiler functions (available only on gcc and llvm compilers). The earlier code was missing XSAVE support at runtime. On VMs where 512‑bit register state saving is disabled, this can result in #UD faults when executing AVX‑512 instructions.

CPU feature detection improvements:

1. Replaced manual CPUID register checks for AVX-512 features with calls to __builtin_cpu_supports for each required feature, improving code readability and portability.
   Added a call to __builtin_cpu_init() to ensure proper initialization before feature checks.
2. . Rename check_compatibility to check_avx512_compatibility.

---

Faster Index Construction via Bulk Vector Serialization

PR: #681

Description
Optimized vector serialization in MemorySegmentVectorProvider by replacing
per-element scalar writes with bulk operations. At high core counts,
per-element scalar writes become a serialization bottleneck: multiple threads
contend on a single lock, stalling index construction and negating the benefit
of additional cores. Bulk writes eliminate that contention, restoring scaling
at large core counts.

• writeFloatVector — uses backing float[] with writeFloats() instead
  of per-element writeFloat()
• writeByteSequence — uses backing byte[] with bulk write() instead
  of per-element writeByte()

How to enable
No configuration required. The improvement is applied automatically when using
the native SIMD backend (jvector-native).

Performance (AWS x8i.24xlarge)

• Bulk serialization throughput: ~1.3× to 1.9× improvement across tested vector sizes
• openai-1536-1m index build: ~57% faster (105.59s → 45.22s)
• openai-3072-1m index build: ~38% faster (164.63s → 102.75s)

---

Bug Fixes and Issue Resolutions

This PR makes dataset loading more robust in the face of file corruption errors.
It also indirects dataset access through DataSetInfo, caching the attached DataSet locally, and enabling access via getDataSet(). This will vastly speed up testing flows which are uneccesssarily processing datasets into memory right when the actual data is not yet accessed.

[UPDATE]

Due to a request for fixing some longer-standing issues, the scope of this PR has increased moderately:

• DataSet loader formats which can not provide metadata (VSF, ...) on their own now pull details from dataset_metadata.yml for VSF. If none such is available, an error should be thrown.

• Addtionally, the contract type (DataSetProperties) is now a layer of requirements added onto the DataSetInfo type, including an indicator for dataset preprocessing aspects (zero vectors, dupes) and normalization.

• surefire tests were disabled in examples. enabled, and fixed up a unit test

• DataSetProperties is a carrier type in the new DataSetInfo contract

---

Disable Graph Search Pruning

PR: #686

Description
Resolves Issue #678. Deprecated and disabled GraphSearcher.usePruning(boolean) for TopK and filtered searches.
The existing TopK/filter pruning heuristic could reduce recall and did not show reliable production value.
The method remains available for source and binary compatibility, but enabling it no longer activates pruning for threshold == 0 searches.
Legacy threshold-search behavior is preserved: searches with threshold > 0 continue to use the existing threshold early-termination path.

How to Enable
No API change. The usePruning setting will be ignored and set to false by the library.

Notes
The recall regression for usePruning = true is present in release 4.0.0-rc.4 onward (possibly earlier).
PR #501 4.0.0-rc.3 inadvertently partially addressed it.

---

Documentation and Tutorials

This includes:

• A set of tutorials covering basic JVector concepts

---

This PR adds links to the introductory tutorials in the main README. The existing step-by-step guide is moved out to it's own file, considering that it's rather cryptic for new users to decipher. I elected not to remove it entirely since it still contains useful information that isn't (yet) documented elsewhere.

---

---

Testing Enhancements

Summary

This PR introduces YAML-driven benchmark compute + reporting selection, and adds run-scoped logging artifacts (sys_info.json, dataset_info.csv, experiments.csv) so Grid benchmark runs are reproducible and easier to analyze offline. Reporting/benchmark policy is now run-level (via yaml-configs/run.yml), while dataset YAML files are simplified to dataset-tuned construction/search settings only.

---

Key changes

Run-level policy file: yaml-configs/run.yml

PR: #625

• Added a new run-level config file that defines:
  • benchmark computation policy (benchmarks: what to compute)
  • console projection (console: subset + named metrics)
  • experiments logging policy (logging: selection + sink type + run metadata)
• Dataset YAMLs are now focused on dataset-tuned parameters only (construction/search grids). Reporting/benchmark controls are no longer expected in dataset YAMLs.

Config versioning clarity

• Renamed the old vague YAML version field to onDiskIndexVersion (still validated against OnDiskGraphIndex.CURRENT_VERSION).
• Introduced yamlSchemaVersion (starting at 1) to version YAML schema independently of on-disk index header format.
• Added deprecated back-compat for legacy files that still use version: 6.

Compute vs display vs logging separation

• Benchmarks specify what is computed; console/logging are projections only.
• Added stable Metric.key identifiers for all benchmark outputs and telemetry so selection and CSV columns are robust (no header-string matching).
• Enforced:
  • strict subset validation for benchmark-stat selections (must be computed)
  • best-effort warnings for runtime-unavailable telemetry/metrics (omitted from output/CSV, with warnings)

Run-scoped artifacts and logging

• When logging is enabled, each BenchYAML invocation creates a run directory under:
  • logging/<run_id>/
• The run directory contains (see examples below):
  • sys_info.json — host/OS/JVM/SIMD/threading/memory + stable system_id
  • dataset_info.csv — one row per dataset used in the run
  • experiments.csv — flat, append-only table of fixed params + selected output-key columns
• experiments.csv schema stability:
  • fixed columns are defined centrally in ExperimentsSchemaV1
  • output-key columns are an ordered union across all topK scenarios encountered in the run
  • non-applicable / missing values are written as empty CSV fields (easy for pandas/matplotlib)

Plumbing / ergonomics

• Added RunArtifacts to centralize:
  • run setup + validation
  • dataset registration (dataset_info.csv)
  • row logging (experiments.csv)
• Updated BenchYAML and HelloVectorWorld to use run.yml + RunArtifacts (no manual wiring of compute/display/logging maps).
• Refactored Grid signatures to take a single RunArtifacts object instead of many reporting parameters.
• Legacy callers continue to work via the legacy Grid overload + RunArtifacts.disabled().

Legacy compatibility

• Legacy YAML files without yamlSchemaVersion are treated as schema 0:
  • parsed leniently (unknown fields ignored)
  • emit a deprecation warning
  • optional extraction of legacy search.benchmarks is supported for legacy behavior when run.yml is absent
• AutoBenchYAML is not migrated to run.yml yet; it continues to work with legacy configs.

---

Behavior

• Logging enabled (run.yml has logging.type):
  • BenchYAML writes run artifacts under logging/<run_id>/
  • prints a startup message pointing to the run directory and how to delete it to reclaim space
  • appends experiment rows as configurations execute
• Logging disabled (no logging section or blank type in run.yml):
  • run artifacts are not created
  • RunArtifacts becomes a no-op via RunArtifacts.disabled()
• Console output continues to work via run-level selection; dataset configs tune only construction/search parameters.

---

Notes / limitations

• experiments.csv uses Metric.key strings as column names for benchmark outputs/telemetry; missing values are empty CSV fields.
• sys_info.json SIMD reports the configured vector width via io.github.jbellis.jvector.vector_bit_size (with simd_config_present), avoiding brittle runtime probing.
• Threading in sys_info.json distinguishes build executor parallelism vs parallel streams (common FJP), explaining why build/query values can differ.
• This PR intentionally does not overhaul per-index build time attribution (e.g., cache hits have no build time by design); deeper build-timing correctness is left for a future pass.
• Dataset YAML breaking changes are intentional: version → onDiskIndexVersion, plus removal of dataset-level benchmarks/console/logging fields in favor of run.yml.

Example of run.yml and yaml-config for dataset

Examples of artifacts (sys_info, dataset_info, experiments)

---

This PR fixes a resource leak where BenchmarkDiagnostics (and its DiskUsageMonitor/NIO WatchService) was created without being closed in several benchmark execution paths. Over repeated runs/config sweeps this leaked OS watch resources/file handles and could fail with “User limit of inotify instances reached.”

Changes:

• Wrap BenchmarkDiagnostics usage in try-with-resources in QueryTester and Grid run paths so it is always closed.
• Refactor ThroughputBenchmark to create BenchmarkDiagnostics within runBenchmark using try-with-resources (instead of storing it as a field), ensuring proper cleanup.

---

This PR expands the capability of the DiskUsageMonitor in the BenchYAML and autoBenchYAML testing applications to be able to monitor more than a single directory and to provide detailed information on disk used and number of new files for each directory monitored, as well as a summary of the total bytes used and number of new files across all monitored directories.

Although this allows DiskUsageMonitor to monitor an arbitrary number of directories in practice as of this PR it is used to monitor the work directory and the cache directory which contains indexes that may be reused for testing purposes.

example rollup summary at index construction time:

Disk Usage Summary Graph Index Build:
  [testDirectory]:
    Total Disk Used: 1.61 GB
    Total Files: 1
    Net Change: 1.61 GB, +1 files
  [indexCache]:
    Total Disk Used: 0 B
    Total Files: 0
    Net Change: 0 B, +0 files
  [Overall Total]:
    Total Disk Used: 1.61 GB
    Total Files: 1
    Net Change: 1.61 GB, +1 files
Index build time: 956.781921 seconds


---

## Summary

This PR adds quantization timing telemetry to BenchYAML reporting (console + experiments.csv), and makes it selectable from `run.yml`. The new metrics capture quantizer **compute** and **encoding** time for both **index construction** and **search-time vector encoding**, with stable `Metric.key` identifiers.

## Key changes

### Quantization timing metrics (new)

**PR:** [#636](https://github.com/datastax/jvector/pull/636)
- Emit quant timing metrics as `Metric` outputs:
    - **Index construction:** `construction.index_quant_time_s.<QT>.(compute_time_s|encoding_time_s)`
    - **Search-time:** `search.search_quant_time_s.<QT>.(compute_time_s|encoding_time_s)`
- Per-quantizer keys are emitted for PQ/BQ; NVQ reports **compute only** (NVQ encoding is intentionally not measured).
- Human-readable headers are phase-tagged to disambiguate:
    - `Idx PQ compute (s)`, `Idx PQ encoding (s)`, `Search PQ compute (s)`, etc.

### Run-level selection support (prefix expansion)
- Added prefix-style named metric selection in `run.yml`:
    - `metrics.construction.index_quant_time_s` → expands to all `construction.index_quant_time_s.*` metrics
    - `metrics.construction.search_quant_time_s` → expands to all `search.search_quant_time_s.*` metrics
- Selection + application now supports “prefix families” of metrics (similar to `recall_at_{topK}` expansion).

### Logging fixes (experiments.csv)
- Updated logging schema planning to include prefix-selected metrics so quant timing columns appear in `experiments.csv` when selected via run.yml.
- CSV column keys remain `Metric.key` strings; missing/non-applicable values remain empty fields.

### Console ergonomics
- Added 2-line header rendering with a max column width cap so wide metric names wrap cleanly.
- Manual header breaks are supported via `\n` in the `Metric.header` string (useful for specific columns like Avg QPS).

### Runtime warnings (availability)
- Preserved “warn once” behavior for missing exact-key metrics (e.g., index build time on cache hit).
- Quant timing warnings were adjusted so we do not warn about NVQ encoding (not measured by design).

## Notes / limitations
- PQ compute time is recorded when the PQ compressor is computed (cache miss). On cache hits, compute time is intentionally blank (no compute performed).
- NVQ encoding time is intentionally not measured; NVQ reports compute only.
- Prefix selection is used for quant timing because concrete per-quantizer keys are only known once the run produces metrics.


---

This extends the v1 experiments.csv schema with build_compressor and search_compressor columns so logged runs record the resolved compressor configurations used during index build and search. Values use the compressor’s printable toString() (e.g., ProductQuantization[...]) and are CSV-escaped to preserve commas. Search compressor logging is null-safe when compression is disabled (cv == null).
<img width="1400" height="345" alt="build-and-search-compressor-info" src="https://github.com/user-attachments/assets/cda7634c-b5f6-4f98-9fc3-1d91955f630b" />


---

## Investigation Complete: Root Cause Found and Fixed

### Key Findings

**PR:** [#645](https://github.com/datastax/jvector/pull/645)

#### 1. The "Recall Degradation" Was Actually a Test Harness Bug
Test results showed:
- **BenchYAML + fusedGraph: No** → recall: 0.65
- **BenchYAML + fusedGraph: Yes** → recall: 0.65
- **AutoBenchYAML + fusedGraph: No** → recall: 0.77 ← **ANOMALY**
- **AutoBenchYAML + fusedGraph: Yes** → recall: 0.65

The anomaly was that AutoBenchYAML with fusedGraph:No produced artificially **high** recall (0.77), not that FusedPQ was producing low recall. All configurations using FusedPQ correctly produced consistent recall of 0.65.

#### 2. Root Cause: Missing Vector Encoding in `runAllAndCollectResults`
**Location:** `jvector-examples/src/main/java/io/github/jbellis/jvector/example/Grid.java`, line 833

**The Bug:**
```java
CompressedVectors cvArg = (searchCompressorObj instanceof CompressedVectors) ? (CompressedVectors) searchCompressorObj : null;

This line attempted to cast a VectorCompressor (ProductQuantization) to CompressedVectors, which always failed, resulting in cvArg = null. When cvArg is null and fusedGraph is disabled, the ConfiguredSystem.scoreProviderFor method falls back to exact scoring using uncompressed vectors (line 1084 / 1094), which artificially inflates recall.

The Fix:

// Encode vectors for reranking if a compressor is provided
CompressedVectors cvArg;
if (searchCompressorObj == null) {
    cvArg = null;
} else {
    cvArg = searchCompressorObj.encodeAll(ds.getBaseRavv());
}

This fix properly encodes the vectors using the compressor, matching the behavior of runOneGraph (the correct implementation used by BenchYAML).

Impact

• Before fix: AutoBenchYAML with fusedGraph:No was using exact (uncompressed) scoring, giving misleadingly high recall
• After fix: AutoBenchYAML will now correctly use PQ-compressed vectors for approximate scoring during graph traversal, with NVQ reranking, producing consistent recall across all test harnesses

Verification

After applying this fix:

• AutoBenchYAML + fusedGraph:No → recall: ~0.65 (matching other configurations)
• All test harnesses will produce consistent recall measurements
• The perceived "recall degradation with high dimensionality" will disappear, as it was never a real issue with FusedPQ

Additional Notes

The investigation also examined the FusedPQ implementation thoroughly and confirmed that:

• The storage and retrieval of fused PQ data is correct
• The scoring mathematics are correct
• The SIMD implementations are correct
• FusedPQ works correctly for all dimensionalities when properly configured

---

Summary

This PR optimizes AccuracyMetrics to improve the performance and readability of recall and precision measurements. The primary focus is shifting from $O(N^2)$ list-scanning to $O(N)$ Set-based lookups and reducing object allocation overhead by interacting directly with primitive NodeScore arrays.

Key changes

Logic cleanup and "dead code" removal

PR: #647

• Unused Signature Removal: Removed the redundant topKCorrect(List, List, ...) signature, consolidating logic into a single private method that operates directly on SearchResult.
• Branch Elimination: Removed the unreachable if (gtView.size() > retrieved.size()) branch. Due to existing guard clauses (kGT <= kRetrieved and kRetrieved <= retrieved.size()), this condition was mathematically impossible.
• Parity Preservation: Preserved specific, argument-focused exception messaging to ensure clear feedback for invalid k parameters.

Performance optimizations

• Algorithmic Shift: Replaced $O(N)$ List.contains() calls with $O(1)$ HashSet.contains(). This moves the core intersection logic from quadratic to linear time complexity.
• AP Nested Scan Removal: In averagePrecisionAtK, replaced the $O(i)$ subList(0, i).contains(p) duplicate check with a HashSet lookup. This transforms the AP calculation from an $O(K^2)$ operation to $O(K)$.
• Allocation Reduction: Eliminated intermediate boxing and ArrayList creation in the SearchResult path. The logic now iterates directly over the NodeScore[] array, significantly reducing GC pressure during large benchmark runs.
• Loop Efficiency: Replaced Stream API calls with manual for loops in high-frequency paths to avoid the object overhead and "setup tax" of the Stream API.
• HashSet Pre-Sizing: Sets are now initialized with explicit capacities ($K / 0.75$) to prevent expensive internal rehashing.

Test coverage

• Functional Validation: Verified Recall, AP, and MAP calculations against hand-calculated results to ensure mathematical parity with standard IR definitions.
• Duplicate Handling: Confirmed that duplicate IDs in search results are handled via the "seen" Set, preventing artificial score inflation while maintaining $O(1)$ lookup speed.
• Exception Parity: Explicitly verified that IllegalArgumentException strings remain 1:1 identical to the original implementation to prevent breaking downstream error-parsing.

Performance results

Benchmarks were conducted using 10,000 queries on the gecko-100k dataset (768 dimensions).

Metric	Original Time	Optimized Time	Delta
recall@10	~8ms	~same	-
recall@100	~58ms	~25ms	~57% reduction
AP@100 / MAP@100	~62ms	~26ms	~58% reduction

While the benefit is negligible for $K=10$, the optimization becomes critical as $K$ increases. The reduction in AP measurement time is particularly significant as it eliminates the $O(K^2)$ complexity previously caused by nested sublist scans.

Notes / limitations

---

Summary

This PR restructures benchmark dataset loading so scrubbing behavior is explicit, metadata-controlled, and uniform across HDF5 and MFD loaders. The primary goal is to stop hard-coding legacy load-time scrubbing behavior into loader-specific paths and prepare a safe transition toward prescrubbed datasets whose offline ground truth matches the stored vectors exactly.

Key changes

Scrubbing behavior becomes explicit

• Added DataSetProperties.LoadBehavior with:
  • LEGACY_SCRUB
  • NO_SCRUB
• Added load_behavior support to dataset_metadata.yml.
• Added DataSetUtils.processDataSet(...) as the new metadata-aware entry point for benchmark dataset processing.
• Preserved the prior load-time scrubbing implementation behind legacyScrubDataSet(...).
• Kept getScrubbedDataSet(...) temporarily as a deprecated compatibility shim.

Unified metadata-driven loader flow

• Updated both DataSetLoaderHDF5 and DataSetLoaderMFD to carry full DataSetProperties through the load path instead of collapsing metadata down to only similarity_function.
• Routed both loaders through DataSetUtils.processDataSet(...) so load behavior is applied in one place.
• Removed the HDF5-specific filename inference path and now require explicit metadata for curated HDF5 datasets.

Metadata coverage updates

• Added load_behavior to existing curated dataset entries.
• Added explicit metadata entries for curated ann-benchmarks HDF5 datasets that previously relied on filename inference.
• Added metadata entries for remaining MFD datasets in the static registry so known datasets fail clearly if metadata is missing.

Visibility / debugging

• Added runtime printing of the dataset similarity function in the graph run path so the effective metadata-supplied similarity can be seen during execution.

Behavior changes

What changes now

• Curated benchmark datasets no longer rely on HDF5 filename suffix inference for similarity.
• Known datasets are now expected to have an explicit dataset_metadata.yml entry.
• Benchmark dataset load behavior is controlled centrally through metadata rather than implicitly by format-specific code paths.
• NO_SCRUB now loads vectors and ground truth exactly as stored.
• LEGACY_SCRUB preserves the existing load-time behavior:
  • zero-vector removal
  • duplicate base-vector removal
  • query filtering against invalid / overlapping vectors
  • ground-truth remapping
  • conditional normalization

What does not change yet

• Existing deployed datasets can remain on LEGACY_SCRUB during the transition.
• The previous scrubbing behavior is still available and intentionally preserved for compatibility while prescrubbed datasets are rolled out.

Why this change

• Benchmark datasets with precomputed offline ground truth should not be silently mutated by loader-specific cleanup logic unless that behavior is explicitly intended.
• The old behavior broke the correspondence between stored vectors and offline ground truth for unscrubbed datasets, even though runs still completed.
• Scrubbing policy is transitional benchmark-loader behavior, not an intrinsic property of the raw dataset format.
• Unifying HDF5 and MFD under the same metadata architecture makes the code easier to reason about, easier to maintain, and safer to evolve as prescrubbed datasets become available.

Notes / limitations

• LEGACY_SCRUB remains the default during the transition to avoid breaking currently deployed datasets.
• getScrubbedDataSet(...) is still present as a deprecated compatibility API and should be removed after downstream callers are fully migrated.
• Locally added datasets in DataSetLoaderMFD must also be added to dataset_metadata.yml to participate in the new configuration model.
• This PR does not move scrubbing into the core library yet; it only prepares the benchmark loader path for a clean transition.

---

This dataset loader is like the previous MDF loader but with very targeted changes to make things easier and more robust.

• It uses HTTP, not requiring S3 to fetch data. It's still plenty fast enough and less complicated.
  • Ted improved the transport to support both. It will do HTTP* and S3 as specified.
• It indirects dataset name to data facets through a datasets.yaml file. This is just a name to facet mapping file.
  • It uses the catalog entries file catalog_entries.yaml within the source directory for clarity.
• It supports local and remote, and simply caches the remote dataset.yaml file locally and uses it by default.
• If will still inform the user if the remote datasets.yaml file has changed if they choose.
• It does the same kind of loading as the previous MFD loader once initialized. All other dataset and run settings are unaffected.

---

Updating regression tests (AutoBenchYAML) to use the datasets.yml configuration rather than hard-coding its datasets.

• AutoBenchYAML.java updated to dynamically load datasets for testing from yaml
• DataSets.java updated to return a specific subsection of datasets.yml based on section key passed
• new subsection ("regression-tests") added to datasets.yml
• new yml files added for the datasets used by regression (primarily the only difference is that we don't want to run as many permutations of overquery or high k value in regression due to time concerns)

---

buildInMemory (the no-build-compressor path) never recorded the index build time into indexBuildTimes, so getIndexBuildTimeSeconds() always returned null for it. This caused construction.index_build_time_s to be omitted from benchmark results, triggering a warning whenever the metric was selected in the logging config:

WARNING: selected logging output not available; skipping metrics.construction.index_build_time_s (construction.index_build_time_s)

Fix: capture the duration of builder.build() in buildInMemory and store it in indexBuildTimes, consistent with how buildOnDisk already does it. Write time is intentionally excluded — only graph construction is measured.

Before:

Index build time: null seconds

After:

Index build time: 32.192336 seconds


---

This adds the name entries for the openai datasets in our shared config.


---



---

This PR updates the tracking of offheap memory usage by calculating max usage comparing the pre and post phase snapshots instead of just using the final snapshot, and adds offheap memory usage to the tabular results generated by BenchYAML.

Note that the reported offheap usage will appear static across configurations for a given dataset in this test. This is because off-heap memory is allocated during index loading (once per dataset). It includes memory-mapped index files and Direct ByteBuffers for vector storage. All query configurations reuse the same loaded index, so off-heap usage remains constant but different datasets have different index sizes, hence different off-heap usage.

Example output:

Disk Usage Summary Graph Index Build:
[testDirectory]:
Total Disk Used: 115.99 MB
Total Files: 1
Net Change: 115.99 MB, +1 files
[indexCache]:
Total Disk Used: 0 B
Total Files: 0
Net Change: 0 B, +0 files
[Overall Total]:
Total Disk Used: 115.99 MB
Total Files: 1
Net Change: 115.99 MB, +1 files
Index build time: 26.442781 seconds

cohere-english-v3-100k: ProductQuantization(M=128, clusters=256, centered=false) codebooks loaded from PQ_cohere-english-v3-100k_128_256_false_-1.0
cohere-english-v3-100k: ProductQuantization(M=128, clusters=256, centered=false) encoded 99685 vectors [13.17 MB] in 1.75s
cohere-english-v3-100k: Using OnDiskGraphIndex(layers=[LayerInfo{size=99685, degree=32}, LayerInfo{size=3224, degree=32}, LayerInfo{size=95, degree=32}, LayerInfo{size=2, degree=32}], entryPoint=NodeAtLevel(level=3, node=75059), features=NVQ_VECTORS):

Index configuration:
featureSetForIndex [NVQ_VECTORS]
M 32
efConstruction 100
neighborOverflow 1.2
addHierarchy true
refineFinalGraph true

Query configuration:
usePruning true

Overquery Avg QPS ± Std Dev CV % Mean Latency Avg Visited Recall@10 Max heap Max offheap
(of 3) (ms) usage (MB) usage (MB)

1.00 52315.1 378.7 0.7 0.163 355.3 0.78 755.2 116.4
2.00 44155.5 619.7 1.4 0.201 515.2 0.92 834.3 116.4
5.00 28832.6 264.1 0.9 0.306 951.1 0.98 834.3 116.4
10.00 17835.6 1134.1 6.4 0.504 1536.7 0.99 786.6 116.4

Overquery Avg QPS ± Std Dev CV % Mean Latency Avg Visited Recall@100 Max heap Max offheap
(of 3) (ms) usage (MB) usage (MB)

1.00 18204.6 119.4 0.7 0.522 1536.7 0.85 803.5 116.4
2.00 11269.1 83.9 0.7 0.846 2495.2 0.97 803.5 116.4

---

Allows all CI workflows to run correctly on PRs from forks. This mainly applies to the workflow which runs AutoBenchYAML.

- The workflow replaces `github.head.ref` (`fork-workflow-2`) with `github.ref` (`/refs/pull/688/merge`). The checkout action fails for the former if the branch is from a fork, but the latter works in both cases.


---


### Provide a possibility to store and load an empty graph

**Description**  
Provide a possibility to store and load an empty graph from/to disk

**Purpose / Impact**
- 
- Helps to preserve graph metadata (similarity function, features, dimensions, even if the graph is empty)