Adding elements to an index without rebuilding it?

[iliane5]

Hi!

Thank you for the great library. I was wondering if there was any way to add/delete elements in an index without having to rebuild it entirely?

It seems like hnswlib does support it but I haven’t found any documentation about it in hnswlib-node.

Thanks again!

[yoshoku]

@iliane5 Thank you for having interest in this library. The hnswlib-node is a thin wrapper for hnswlib, so it supports similar operations. An example script would look like this:

import { HierarchicalNSW } from 'hnswlib-node'

const numDimensions = 4;
const maxElements = 5;

const index = new HierarchicalNSW('l2', numDimensions);
index.initIndex(maxElements);

index.addPoint([1, 2, 3, 1], 1);
index.addPoint([1, 2, 3, 2], 2);
index.addPoint([1, 2, 3, 3], 3);
index.addPoint([1, 2, 3, 4], 4);

const numNeighbors = 2
const query = [1, 2, 3, 5];
console.table(index.searchKnn(query, numNeighbors));

index.writeIndexSync('foo.dat');

// ---

const readedIndex = new HierarchicalNSW('l2', numDimensions);
readedIndex.readIndexSync('foo.dat');

// Adding new point closest to the query.
readedIndex.addPoint([1, 2, 3, 5], 5);

console.table(readedIndex.searchKnn(query, numNeighbors));

// Removing an indexed point.
readedIndex.markDelete(4);

console.table(readedIndex.searchKnn(query, numNeighbors));

Execution result:

$ yarn ts-node sample.ts
┌───────────┬───┬───┐
│  (index)  │ 0 │ 1 │
├───────────┼───┼───┤
│ distances │ 1 │ 4 │
│ neighbors │ 4 │ 3 │
└───────────┴───┴───┘
┌───────────┬───┬───┐
│  (index)  │ 0 │ 1 │
├───────────┼───┼───┤
│ distances │ 0 │ 1 │
│ neighbors │ 5 │ 4 │
└───────────┴───┴───┘
┌───────────┬───┬───┐
│  (index)  │ 0 │ 1 │
├───────────┼───┼───┤
│ distances │ 0 │ 4 │
│ neighbors │ 5 │ 3 │
└───────────┴───┴───┘

[yoshoku]

@iliane5 It has been two weeks since I responded, so I will close this issue. Search index algorithm based on graph structure adds new data as nodes and does not require rebuilding the index.