HttpANN algorithm to support language-angostic implementations (Re: Issue #20)

[alexklibisz]

Thanks @maumueller, @gosha1128 and others for the fruitful discussion over in #20. I think I've arrived at an implementation that could fit the purpose of language-agnostic (big) ANN.

The HttpANN algorithm is designed to make HTTP calls to a server. The server executes all indexing and querying, thus enabling language-agnostic ANN implementations with minimal overhead. The only requirements for the server are:

1. It should implement the JSON-over-HTTP API documented below (copied from httpann.py). Note that this is a 1:1 copy of the BaseANN Python Class API.
2. It should be able to read the vector dataset in the standard binary format used by this competition.

It could in theory even run remotely, although the intended use-case is that the server runs in the same container.

The overhead for data transfer and serialization is minimal. The server only needs to parse the 10k JSON-encoded query vectors and encode the resulting 10k lists of neighbors.

I also included an example implementation which uses scikit-learn. It's too slow for the large datasets, but it works on the smaller random-xs and random-range-xs. So it should be good enough to demonstrate that this algorithm works.

---

Here is the API that a server must implement:

Method	Route	Request Body	Expected Status	Response Body
POST	/init	dictionary of constructor arguments, e.g., {“metric”: “euclidean”, “dimension”: 99 }	200	{ }
POST	/load_index	{ "dataset": <dataset name, e.g. "bigann-10m"> }	200	{ "load_index": }
POST	/set_query_arguments	dictionary of query arguments	200	{ }
POST	/query	{ “X”: , “k”: }	200	{ }
POST	/range_query	{ “X”: , “radius”: }	200	{ }
POST	/get_results	{ }	200	{ “get_results”: }
POST	/get_additional	{ }	200	{ “get_additional”: }
POST	/get_range_results	{ }	200	{ “get_range_results”: <list of three 1-dimensional lists (lims, I, D)> }

[alexklibisz]

There were two sections called "deep-10M" so I just moved the "diskann-t2" section up here to deduplicate the sections.

[alexklibisz]

It looks like CI is failing because the fit method is never called. I think I can use the --rebuild flag to force this, but how is it working for the other algos without this flag? Nevermind, it looks like I need to just return false from load_index.

[maumueller]

This looks very nice, thanks @alexklibisz! Have you been able to measure the overhead vs a naive implementation?

My only suggestion here is to split up httpann.py into base-http.py (or whatever you deem to fit) and http-example-sklearn.py to show the interaction between the base http wrapper and the actual implementation that you suggest others to use.

[maumueller]

I'm wondering what the performance penalty of this will be. I would be happy to

• change the arguments such that the query vector file is exposed
• or add some kind of prepare for providing the query vectors

[alexklibisz]

I've done some local testing, and I think it will be fine since the HTTP overhead and JSON serialization is only incurred one time. It was a bigger deal with ann-benchmarks because that framework required a request and serialization for every query.

[alexklibisz]

My only suggestion here is to split up httpann.py into base-http.py (or whatever you deem to fit) and http-example-sklearn.py to show the interaction between the base http wrapper and the actual implementation that you suggest others to use.

Sounds good. I'll break it into two files.

[maumueller]

Great, thanks. Is this ready to be merged?

[alexklibisz]

I think so. Please squash if you can, as there are several intermediate/incomplete commits in there.

[alexklibisz]

I'll resolve the conflicts and also need to add one bit of documentation.. one moment..

[maumueller]

Sorry for not getting back to you early. Shall I squash and merge with main?

[alexklibisz]

No problem. Yes, good to go. Thanks!

[maumueller]

Looking forward to how this is going to be used. Thanks again.