Fast Sentence Embeddings

fse 2.0 is in development on master and not released yet; this README describes it, and its API may still change before 2.0.0. The current release is fse 1.1.0 on PyPI, in maintenance mode on the branch 1.x (bug and compatibility fixes only), with its own README and notebooks. 2.0 changes the API and the file format of saved models. Feature requests are closed until then; bug reports are welcome.

If you start a new project, also look at actively maintained libraries such as sentence-transformers or model2vec.

fse 2.0 encodes text with finished static embedding models: a model (fse.Encoder) is one .safetensors file that holds a table of word rows, its tokenizer and whether every row is normalized. A sentence becomes the mean of the rows of its words, computed in parallel by a native kernel, bit for bit the same on every CPU. fse 2.0 trains nothing: weights, removed components or projections of a model (such as SIF or uSIF) are folded into its rows before it is saved. Vectors go into memory, or into a store on disk that fse.VectorStore searches.

2.0 is not on PyPI yet. From a checkout of master, with uv and a C compiler:

make install

fse.gensim (converting gensim word vectors) needs the extra fse[gensim].

On Apple silicon with macOS 14 or newer, the extra fse[mlx] adds the backend "mlx", which averages the word vectors on the GPU within a small tolerance of the CPU's rows (usually with their bits): encoder.encode(texts, backend="mlx"). For tiny inputs it is slower than the default "cpu".

Load a finished model and encode:

import fse

encoder = fse.Encoder.load("model.safetensors")
rows = encoder.encode(["the cat sleeps", "a dog runs"])  # one float32 row per text

Make a model from word vectors (every word weighs 1) and save it:

import fse
from fse.gensim import from_gensim
from gensim.models import KeyedVectors

wv = from_gensim(KeyedVectors.load("glove-wiki-gigaword-300.model"))
encoder = fse.Encoder.from_word_vectors(wv)  # tokenizer fse_words@1, rows normalized
encoder.save("glove.safetensors")  # one file with the table

Which call to use when:

• encode returns float32 rows by default; dtype="float16" gives float16 rows and dtype="int8" a fse.QuantizedRows (int8 codes and one scale per row).
• store.search returns a fse.SearchResult with the ids (positions in the store) and the cosine scores, best first. Its default mode="auto" takes the exact arithmetic of fse for up to 8 queries at once and BLAS above, whose bits depend on how many queries are asked together; mode="exact" gives the same results for every batch.
• Texts are tokenized by fse_words@1 by default: lowercase, split at whitespace and punctuation, contractions expanded (ARCHITECTURE.md, "Tokenizers"). Subword tokenizers (WordPiece, Unigram), which models such as POTION use, follow in a later PR; until then such models run with tokenizer="external" and tokens you make.
• FastText word vectors keep their ngram rows: an unknown word adds the rows of its first max_ngrams ngrams (default 40) with the weight oov_weight (default 1).

What 2.0 no longer has, against fse 1.1.0 and the earlier development versions of 2.0:

• training: Average, SIF and uSIF with train or fit, and any learning from a corpus. SIF and uSIF tables are made offline; a notebook with the 1.1.0 recipe (word weights and the removal of principal components, folded into the table) follows before the release.
• stored recipes and learned components: a model holds its finished table, not the recipe, the weights or the components it was made with.
• word frequencies: word vectors hold no counts, and from_gensim takes none.
• shared external tables: every model holds its own table, so two models of the same word vectors hold two copies.
• Corpus: encode takes lists, paths and streams directly.
• Files written by a development version of 2.0 before this change are refused (fse.UnsupportedArtifactError); rebuild them.

ARCHITECTURE.md describes the design and the numerical contract, CHANGELOG.md every change against 1.1.0.

These are results of fse 1.1.0, not of 2.0 models; the offline SIF and uSIF recipe will be checked against them. Pearson correlation (x100) on the test set of the STS Benchmark, computed with the procedure of notebooks/STS-Benchmarks.ipynb on the branch 1.x (lang_freq="en") with fse 1.1.0 on Python 3.13, numpy 2.5, scipy 1.18, gensim 4.4, wordfreq 3.1 and nltk 3.10. SIF and uSIF compute their principal components exactly since 1.1.0; against the randomized SVD of scikit-learn 1.9 used before, in the same environment, SIF-10 scores move by at most 0.04 points and uSIF scores not at all. Compared with the table published for fse 1.0, uSIF scores rise by 0.2 to 1.4 points. Most of that comes from a fix in 1.1.0 to how uSIF weights the removed principal components, worth 0.2 to 1.3 points when both formulas run in this environment (see CHANGELOG.md). Newer dependency versions shift SIF scores by up to 0.3 points and CBOW scores by up to 0.04 points.

1. Iyyer M, Manjunatha V, Boyd-Graber J, Daumé III H (2015) Deep Unordered Composition Rivals Syntactic Methods for Text Classification. Proc. 53rd Annu. Meet. Assoc. Comput. Linguist. 7th Int. Jt. Conf. Nat. Lang. Process., 1681-1691.

2. Arora S, Liang Y, Ma T (2017) A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Int. Conf. Learn. Represent. (Toulon, France), 1-16.

3. Ethayarajh K (2018) Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline. Proceedings of the 3rd Workshop on Representation Learning for NLP. (Toulon, France), 91-100.

4. Eneko Agirre, Daniel Cer, Mona Diab, Iñigo Lopez-Gazpio, Lucia Specia. Semeval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. Proceedings of SemEval 2017.

See CHANGELOG.md.

The project uses uv and a Makefile:

make install   # create the environment and build the extension
make check     # format check, lint, type check and tests

See CONTRIBUTING.md for details and SECURITY.md for reporting vulnerabilities.

Built by Dr. Oliver Borchers. For questions that do not belong in a GitHub issue, use the contact page.

Copyright (C) 2019 to 2026 Oliver Borchers, licensed under the GNU General Public License v3.0.

If you found this software useful, please cite it in your publication.

@misc{Borchers2019,
	author = {Borchers, Oliver},
	title = {Fast sentence embeddings},
	year = {2019},
	publisher = {GitHub},
	journal = {GitHub Repository},
	howpublished = {\url{https://github.com/oborchers/Fast_Sentence_Embeddings}},
}