Provide support for RAFT-based indexes

[wphicks]

This is still a work-in-progress. Complete PR description will be added once it is ready for review. Resolve #715.

issue: #715

[sre-ci-robot]

Welcome @wphicks! It looks like this is your first PR to milvus-io/knowhere 🎉

[mergify]

@wphicks Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

[xiaofan-luan]

@wphicks Welcome !

[cjnolet]

Integration looks pretty clean so far! Leaving some feedback as we will probably want to allow sweeping through most, of not all, of the available parameters to get accurate benchmarks.

[matrixji]

I see raft support for faiss, right now Knowhere reuses some faiss's code for some GPU Index. Does it possible to reuse the faiss's impl, if Faiss would support RAFT?

[liliu-z]

I see raft support for faiss, right now Knowhere reuses some faiss's code for some GPU Index. Does it possible to reuse the faiss's impl, if Faiss would support RAFT?

We will definitely do that. Since it takes time for Faiss' integration to be done, we can first support RAFT to fillfull users' needs.

[wphicks]

I see raft support for faiss, right now Knowhere reuses some faiss's code for some GPU Index. Does it possible to reuse the faiss's impl, if Faiss would support RAFT?

Yes, as @liliu-z mentioned, integrating RAFT is mostly to make it available within Milvus sooner. It may also be an integration path that will be useful to support longer term, since ANN improvements are a very active area of research and development in RAFT. New innovations there may take a bit of time to be supported through FAISS, so having RAFT support will allow Milvus to more quickly take advantage of the latest and greatest we can come up with.

[Presburger]

I see raft support for faiss, right now Knowhere reuses some faiss's code for some GPU Index. Does it possible to reuse the faiss's impl, if Faiss would support RAFT?

Yes, as @liliu-z mentioned, integrating RAFT is mostly to make it available within Milvus sooner. It may also be an integration path that will be useful to support longer term, since ANN improvements are a very active area of research and development in RAFT. New innovations there may take a bit of time to be supported through FAISS, so having RAFT support will allow Milvus to more quickly take advantage of the latest and greatest we can come up with.

raft allows knowhere to rely on fewer libs, faiss lacks modularity in design, so I prefer raft.

[wphicks]

I've just updated with support for a pool allocator and all of RAFT's IVF parameters.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Presburger, wphicks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Presburger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Presburger]

@wphicks hi, if you can merge the commit into single one and signed the commits, then we will merge the pr.

[matrixji]

seems here index_raft should be ivf_raft