RVF SDK silently drops metadata; native metadata is not durable or returned

[thesoulpole]

Versions

• @ruvector/rvf@0.2.3
• @ruvector/rvf-node@0.2.0
• darwin-arm64 native binary 0.1.7
• Node 26.4.0, macOS arm64

Public SDK reproduction

const db = await RvfDatabase.create(path, {
  dimensions: 2,
  metric: 'cosine',
}, 'node');

await db.ingestBatch([
  { id: 'red-item', vector: new Float32Array([1, 0]), metadata: { color: 'red' } },
  { id: 'blue-item', vector: new Float32Array([0, 1]), metadata: { color: 'blue' } },
]);

Observed:

• Ingest reports both vectors accepted.
• Unfiltered results contain no metadata.
• A field-0 equality filter for red returns no results.

Native N-API reproduction

Using the low-level binding, pass one positional metadata record per vector:

db.ingestBatch(
  new Float32Array([1, 0, 0, 1]),
  [101, 102],
  [
    { fieldId: 0, valueType: 'string', value: 'red' },
    { fieldId: 0, valueType: 'string', value: 'blue' },
  ],
);

A field-0 red filter works while that database instance remains open. However:

• Results still contain only ID and distance.
• After close/reopen, the same filter returns no results.

Root cause

• npm/packages/rvf/src/backend.ts accepts RvfIngestEntry.metadata but calls native ingestBatch(flat, ids) without metadata.
• Native SearchResult contains only ID/distance.
• Runtime inserts metadata into an in-memory MetadataStore, but no metadata segment is written and reloaded.
• The SDK filter declaration omits the native parser's required valueType.

Impact

Applications can successfully ingest records while silently losing metadata. Filtering may appear to work through the native API and then stop working after restart, which can produce incomplete or incorrect search results without an error.

Requested fix

As an immediate safety measure, reject non-empty SDK metadata until it is supported rather than silently accepting it.

A complete fix needs:

• A per-vector metadata shape across the SDK and N-API.
• Metadata segment persistence and reload.
• Aligned filter types, including valueType.
• Either metadata-bearing search results or getMetadata(id).
• Close/reopen tests, including vectors with unequal metadata-field counts.

[ruvnet]

Release verification update (2026-07-28): the interim safety fix from #708 is present in published @ruvector/rvf@0.3.4. A clean registry install rejects non-empty metadata with MetadataNotSupported before ingest, and store status remains at zero vectors, so the public SDK no longer silently accepts or loses metadata. Keeping this issue open for the remaining native durable-metadata/search-result design described above.

[thesoulpole]

thx(!) for the RVF/SONA batch (#708) and agentdb #15. saw your release-verification pass ystrdy.

One thing still blocks me: durable metadata (#704). use case is merchant catalog distribution — one RVF file a merchant hands over that carries its own product attributes (brand, size, colour, price). today those live in a sidecar manifest next to the file, so it's two artifacts that have to stay in sync, which defeats the point. MetadataNotSupported is the right interim call, but we need the real thing.

read through main at 597be6a to size the gap. smaller than I expected:

• Meta (0x07) and MetaIdx (0x0D) are reserved and unused
• write_meta_seg exists in write_path.rs, no callers
• MetadataStore is rebuilt empty at all four construction sites in store.rs, incl. open() and open_readonly()

so the shape looks like write_cow_state_segments / restore_cow_state applied to MetadataStore over the Meta segment, plus a field-name→fieldId dictionary in the same payload.

happy to write it. four things I'd want your call on first, since they're format commitments rather than implementation:

1. payload encoding for the Meta segment — dictionary + per-vector values in one blob, or split across Meta and MetaIdx?
2. are fieldIds file-local or stable across files? decides whether two merchant catalogs can ever merge.
3. COW branches — does a child inherit the parent's Meta segment, or always write its own?
4. does filtered search stay a linear scan over MetadataStore, or is MetaIdx meant to back it?

say the word on 1-4 and I'll do the PR. if it's already scoped your side, ignore me and just say roughly when.