feat(indexer): allow user switch indexer in query time

[hanxiao]

Scenario explained

In most of examples and tutorial, we use NumpyIndexer for the vector index. In the query time, the NumpyIndexer stores vectors using compressed binary, then in the query time it uses exhaustive search (dot products).

The motivation behind this PR is to let user switch to advanced indexer in the query time, without re-indexing again, e.g.:

• exhaustive dot product is slow, I want to switch to annoy
• annoy is still slow, I need to check faiss
• faiss is fine but I need to tune the parameter here and there.

This feature is possible because all vector indexers implemented so far are inherited from NumpyIndexer. In #589 , I made a preparation PR for refactoring NumpyIndexer into BaseNumpyIndexer.

Results

Say user uses the following YAML for indexing.

    !NumpyIndexer
    metas:
      name: wrap-npidx
    with:
      backend: numpy
      compress_level: 1
      index_filename: wrap-npidx
      metric: euclidean

With the new feature enabled in this PR, in the query time, the user can switch to AnnoyIndexer, simply by quoting the above YAML in ref_indexer:

!AnnoyIndexer
with:
  ref_indexer:
    !NumpyIndexer
    metas:
      name: wrap-npidx
    with:
      backend: numpy
      compress_level: 1
      index_filename: wrap-npidx
      metric: euclidean

or to NMSLibIndexer via

!NmslibIndexer
with:
  ref_indexer:
    !NumpyIndexer
    metas:
      name: wrap-npidx
    with:
      backend: numpy
      compress_level: 1
      index_filename: wrap-npidx
      metric: euclidean
  space: l2

No touch on all binary dumps is required.

Generalization

ref_indexer is implemented at BaseNumpyIndexer level. So all indexers inherited from it are switchable, e.g. you can also "downgrade" an AnnoyIndexer to NumpyIndexer by creating a !NumpyIndexer YAML and putting !AnnnoyIndexer inside ref_indexer.

One can also use it in a compound manner, e.g. in ChunkIndexer:

!ChunkIndexer
components:
  - !AnnoyIndexer
    with:
      ref_indexer:
        !NumpyIndexer
        with:
          index_filename: vec.gz
        metas:
          name: vecidx  # a customized name
          workspace: $WORKDIR
  - !ChunkPbIndexer
    with:
      index_filename: chunk.gz
    metas:
      name: chunkidx
      workspace: $WORKDIR
metas:
  name: chunk_compound_indexer
  workspace: $WORKDIR

[github-actions]

Jina CLA check ✅ All Contributors have signed the CLA.

[github-actions]

This PR closes: #587

[nan-wang]

LGTM👍