Support inference-free SPLADE models (asymmetric doc-side encoding, tokenizer-only queries)

[KShivendu]

Summary

FastEmbed currently ships only one SPLADE model — prithivida/Splade_PP_en_v1 — which is a symmetric SPLADE variant: the ONNX encoder runs on both documents and queries. There is no support for inference-free SPLADE (IF-SPLADE), where document vectors are produced by neural expansion at index time, but queries are processed by plain tokenization into raw BERT token IDs — no model, no GPU, no neural inference at query time.

This is a meaningful gap because inference-free SPLADE gives essentially the same retrieval quality as full SPLADE while making queries dramatically cheaper, which is exactly the regime FastEmbed targets (lightweight, ONNX, CPU-friendly serving).

Motivation

Benchmarks of inference-free vs. full SPLADE on BEIR scifact (writeup):

• ~13× faster queries — median query time dropped from ~57ms to ~4.3ms.
• <1.3% relative quality loss — only 0.0093 NDCG@10 lower than full SPLADE.
• Both substantially outperform BM25 at comparable query speed.
• The cost shifts entirely to index time (neural expansion of documents), which for static / slowly-changing corpora is an acceptable one-time investment.

Because queries skip the transformer entirely, IF-SPLADE removes the 50–100ms per-query encoder latency that currently makes Splade_PP_en_v1 expensive to serve, while keeping learned sparse retrieval quality.

What's missing in FastEmbed

1. No inference-free SPLADE models are registered. fastembed/sparse/splade_pp.py only lists prithivida/Splade_PP_en_v1 (and a duplicate misspelled prithvida/... alias).
2. No inference-free query path. SpladePP runs the ONNX model for both embed() and query_embed(). There is no code path that tokenizes a query and emits raw BERT token-id sparse vectors (with IDF/learned token weights) without invoking the encoder.

Proposal

• Add support for asymmetric / inference-free SPLADE models, e.g.:
  • opensearch-neural-sparse-encoding-doc-v3-gte (asymmetric)
• Implement an inference-free query path: tokenize the query into BERT token IDs and assign token weights without running the ONNX encoder, so query_embed() requires no model inference, while embed() (documents) continues to run neural expansion.

Notes:

Unfortuantely, naver/splade-v3-doc (asymmetric, inference-free optimized) cannot be integrated in Fastembed due to license issues.

References

• Blog: https://kshivendu.dev/blog/if-splade
• naver/splade-v3-doc, naver/splade-v3 on Hugging Face

[pablopupo]

Hi @KShivendu, I'd like to pick this up.

My plan is to add the new model as an asymmetric entry, reuse the existing document-side path, and make the query side inference-free (tokenizer + IDF weights, like bm42/miniCOIL) for the ~13x speedup.

There are two things I'd like to confirm first. First, for the hosting: the document encoder needs an ONNX export and the query side needs the IDF file. Do you want these mirrored under Qdrant/ on HF, or should I do the conversion and hand them over for you to upload? Second, for the license: since naver/splade-v3-doc is non-commercial and the OpenSearch model is Apache-2.0, I'm assuming we target the OpenSearch one and drop naver. Can you confirm?

[pablopupo]

Hi @KShivendu, circling back on the two questions above, HF hosting for the query-side artifacts and the base model license. If it helps, I can open the PR with the document side plus the inference-free query side and we can settle hosting during review.