Skip to content

fse 2.1.0

Latest

Choose a tag to compare

@github-actions github-actions released this 15 Nov 14:27
· 30 commits to master since this release
ca91af4

Added

  • Accept custom os.PathLike[str] objects consistently for local model files, model folders, caches, extra files, stores and sinks. Path objects are always local model references; only strings may identify a Hub repository. Byte paths remain invalid, and existing validation error classes are preserved.

    Import PathLike, ModelRef and WorkerCount from fse.typing for common input annotations.

    Accept one-dimensional NumPy text arrays as batches in encoding and prediction, including document IDs and position-based resume across stores of different lengths. Unicode, object and StringDType arrays use the existing dimension and row checks. Results retain the same bits as list batches. Score sink targets and store row locations expose their guaranteed array dtypes.

  • The lint check verifies local file references in Markdown and notebooks without network access. Architecture guidance links to the configured layers and guardrail paths, and contributor rules require validating public inputs before conversion.

  • The speed study compares ten README encoding configurations against released fse 2.0.0, with five paired rounds, fixed ST and paragraph inputs, production memory admission and a shared version-study budget from setup through allocation steps.

  • The versions speed study now pairs the 100-million-sentence writing cell against fse 2.0.0, with full warmups, alternating arms, persisted code and scale sample bit checks, and a streaming memory preflight.

  • Run version measurements, CPU profiles and memory profiles as steps of one allocation, sharing the saved time and money budget. Select the deciding candidate child, isolate profiler installations, and retain missing profile evidence in sealed results.

  • The README has a new section with a picture and a short example for several writing machines on a network drive, with one store for each writer and one view for the reading machine (DR-351: Mehrere Schreiber auf einem Netzlaufwerk in der README mit Grafik zeigen).

  • Complete public static return types before 2.1 (DR-323: Öffentliche Typen und vollständige statische Rückgabetypen vor 2.1 anbieten). Import Encoding, LengthLevel, ScoreHead, SinkRows and Tokenizer from fse.typing. Factories preserve subclass types where they already return subclasses; Encoder.quantize and WordVectors.quantize return their base classes. Overloads distinguish array, int8 and sink results, vectors and length reports. Public batch iterators expose close() through Generator, and guaranteed array dtypes are explicit. Batch fields now include their possible None values, so typed callers must check them, for example assert batch.rows is not None. Heads mapping values have concrete model types. Values, validation errors, arithmetic and artifact formats stay unchanged.

  • Enable the int8 prefilter on arm64 CPUs for searches without rows= (DR-340: int8-Vorfilter für die Suche ohne rows= auf ARM freischalten, part of DR-168: Suche mit int8-Vorfilter und beweisbarer Fehlerschranke beschleunigen). Exact searches of int8 stores (mode="exact", and "auto" up to 8 queries; 1 to 8 queries, k up to 100, 256 to 1024 values per row, at least 65,536 rows) of a store or a view on arm64 CPUs with the dot-product extension now skip the rows that a provable int8 bound rules out and rescore the rest through the exact path; ids and score bits are those of the search without it. A search with rows= or doc_ids=, with max_per_doc, a float store, a smaller store, a larger k or more queries takes the previous path unchanged, as do x86 CPUs (the AVX2 variant stays off) and the numpy backend. A local measurement on Apple silicon found large savings for whole-store searches (not published here) and none for rows=, which is why rows= is excluded. The log event store.prefilter (DEBUG) records each call that enters. No public API, arithmetic or persistence name changes.

  • Add a local Aegis prompt-head preparation recipe (DR-318: Prompt-Köpfe aus dem Aegis-2.0-Datensatz außerhalb von fse trainieren und mit Modellkarte vorbereiten). It compares two pinned encoders, trains binary safety and ten-category heads outside fse, and prepares complete artifacts for grouped inference, with source reports, subset metrics and model cards. No upload or library change. File-only recovery preserves the frozen test result; the deliverable scan audits its narrow exception for pinned source URL literals.

  • Add publication-study search cells for exact float32, exact int8 and binary search, plus an explicit prefilter comparison selection (DR-308: Suchzellen mit Zeit je Anfrage in der Veröffentlichungsstudie anbieten). Whole-pool blocks share a calibrated pass count and report time per request with median and IQR. Check the measured final pass against exact score bits and ids, record binary recall and prefilter path evidence, and admit model groups against phase memory plans and the allocation budget. No library change or published speed numbers.

  • Add an offline Civil Comments classifier preparation tool under tools/hub/heads/ (DR-311: Toxizitäts-Classifier aus google/civil_comments außerhalb von fse trainieren und mit Modellkarte vorbereiten). It compares two pinned encoders on train and validation, trains a six-label head with scikit-learn outside fse, verifies complete and standalone artifacts, and renders a model card from recorded test metrics. Models and data stay outside the checkout. The final prepared model retains toxicity and insult from the full streaming SGD head, with validation-selected thresholds and one frozen official test evaluation. No upload or library change.

  • Add two native-file classification headline cells for 100 million texts (DR-314: Kopfzeile für 100 Millionen klassifizierte Texte in der Veröffentlichungsstudie anbieten), with labels and probabilities, configurable head labels (default: toxicity and insult), resource admission before input preparation, per-round warmup, and sample bit checks against cpu and numpy. The untrained placeholder is restricted to functional rehearsals. Publication waits for a pinned real toxicity head. Protocols expose comparison-arm, resource-plan, and quiet-cell properties. Measurement tooling only; no library change.

  • Profile an addressed development or fse speed child with speed.py --profile and the strict profile@1 contract (DR-313: Profiljob für einen adressierten Kindjob unter der Heavy-Sperre ausführen). One heavy lock covers access checks, offline input reconstruction and execution. Profiles have a separate deadline, fresh folders, checked origin hashes and executing identities, explicit coverage evidence and diagnostic receipts that comparison summaries refuse. Measurement tooling only.

  • Reuse the shared pool for text scans and add pool= to fse.add_sign_bits (DR-309: Gemeinsamen Pool auch für die Text-Zerlegung und add_sign_bits nutzen). File, encoded-string and Arrow scans use the pool threads and task places before encoding. Batch iterators scan on their iteration pool; StreamSource collections still scan inline. Sign-bit backfills share the pool limits and leave it open, including after failures. Chunks, error order and output bytes stay unchanged.

  • Evaluate a named cells@1 decision across development cells (DR-298: Gesamtentscheidung über mehrere Zellen im Mess-Kern auswerten). Main cells must pass their existing noise and confirmation gates plus the declared practical saving. Edge cells must meet their factor bound in comparison and confirmation. Four explicit changes: optional automatic early stop after a final rejection (disabled in the local int8 study), exit 1 for an overall rejection or inconclusive result including partial selections, decision.json also after preparation failure or budget exhaustion, and per-edge factor bounds without a global minimum. Cell decisions are unchanged. Measurement tooling only.

  • Document named linear heads, score stores, document IDs and later filtering, add the Civil Comments filtering recipe, and extend the classification and folding recipes (DR-230: Kopf-Dokumentation und Toxizitäts-Beispielkopf vorbereiten). Documentation only; no change to the library.

  • Add VectorStore.search(..., mode="binary", oversample=None) (DR-267: Binäre Suche mit search(mode="binary", oversample=) anbieten, part of DR-255: Binäre Vorsuche mit exaktem Nachrechnen als Opt-in anbieten). Stage 1 ranks the selected rows by the Hamming distance of their sign bits to the bits of the query (numpy, fse._reference_bits) and keeps the best oversample * k per query (default 40, provisional until DR-264: Recall der Varianten und Kosten der Bitsuche gegen den int8-Vorfilter messen); stage 2 scores only those rows exactly with the path of rescore, so the result is search(mode="exact", rows=sorted(C)) for the candidates C bit for bit, and oversample at least the selection gives mode="exact". rows=, doc_ids=, max_per_doc=1, workers= and pool= work; a shard without sign bits is derived from its rows with one store.bits_derived WARNING per store. New events store.binary and store.bits_derived. "binary" and oversample are public values and freeze with 2.1; no new persistence name; score and score stores are unchanged.

  • Add ShardSink(sign_bits=True) (DR-266: Vorzeichenbits beim Schreiben mit ShardSink(sign_bits=True) mitschreiben, part of DR-255: Binäre Vorsuche mit exaktem Nachrechnen als Opt-in anbieten). Default False; without the switch the bytes written are those of before. With it each shard gets its sign_bits-<first row> artifact: the bits are derived in the hash thread from the codes committed, and published as a step of its own after the shard is renamed and its rows are counted, with the durability of the sink (none tags the artifact). A failure of the bits is an error of the sink that never changes the rows already counted; a stop between a shard and its bits leaves a valid store. A sink made with resume=True writes the bits that the published shards it keeps lack (checked as fse.add_sign_bits checks them), so any stop resumes to the bytes of an uninterrupted run; a sink that only appends does not. The bits equal those of fse.add_sign_bits. Adds the fields sign_bits (store.sink_opened) and sign_bits_shards (store.sink_closed). No new persistence name. No timing was measured; the write cost follows later.

  • Add fse.add_sign_bits(path, *, durability="full", workers=None) and the companion artifact sign_bits-<first row> of the sign bits of a shard (DR-265: Vorzeichenbits als Begleitdatei je Shard lesen und mit fse.add_sign_bits nachrüsten, part of DR-255: Binäre Vorsuche mit exaktem Nachrechnen als Opt-in anbieten). The artifact is an NpyContainer with the tensor sign_bits (rule sign_bits@1), bound to its shard by fse.shard_fingerprint; shards and store.json are unchanged. VectorStore.open, refresh, verify, views and repair_store read and check it (a bound, damaged or unknown artifact is CorruptArtifactError or UnsupportedArtifactError, never skipped; a shard without it stays valid; repair moves bits before their shard, and alone when they do not fit). add_sign_bits writes the missing artifacts after hashing the vector bytes against the checksum of the shard, and returns their number. Adds the event store.sign_bits_added and the fields sign_bits_shards, sign_bits_bytes (store.opened), added_sign_bits (store.refreshed) and moved_sign_bits (store.repaired). VectorStore.search(..., mode="binary") searches with them. New persistence names, which freeze with 2.1: sign_bits-, sign_bits, fse.kind=sign_bits, fse.bits=sign_bits@1, fse.shard_fingerprint. v2.0.0 reads such a store and passes over the entries.

  • VectorStore.search, score and rescore take pool= (DR-283: Gemeinsamen Pool für search, score, rescore und Köpfe über einen Store annehmen): with an fse.Pool their tasks run on its threads, at most workers (default pool.workers) at once, instead of threads of each call, so concurrent calls of a service share one set of threads. Bit-defined calls (mode="exact", "auto" for at most 8 vectors, the int8 prefilter, rescore, head store mode) give the bits of a call without a pool, mode="blas" stays close as before, and a call without pool= behaves as before. workers above pool.workers raises ValueError, a closed pool RuntimeError, before a row is read. A call takes a place of the pool per task until the task ends, and an error, a prefilter that gives up or a stream ended early cancels its unstarted tasks, waits for its running ones and leaves the pool open. Fixed: the store mode of a head (predict, predict_proba, decision_function and predict_batches of a Classifier or Regressor given a VectorStore) accepted pool= and ignored it; it now runs on that pool. See docs/architecture/store.md, "Concurrency".

  • Measure multilingual quality with benchmarks/quality.py (DR-276: Mehrsprachige statische Modelle auf multilingualen Benchmarks vergleichen): MTEB(Multilingual, v2) and NanoBEIR-multilingual of the pinned MTEB 2.26.25 join the benchmarks, and every task keeps the benchmark it came from, so a task name of several benchmarks (STS17 is English only in MTEB(eng, v2)) is refused unless written as "Benchmark/Task". Each task's data is unloaded after its evaluation. Local class K runs need explicit HF_HOME and HF_DATASETS_CACHE, stop at the next task below a free disk floor, abort above a resident set limit, refuse tasks with a larger memory estimate and take a shared study deadline. Rows record the MTEB training relation, the execution path and the snapshot file hashes; summaries add coverage by set equality, tables per language and language pair, a common zero-shot set and reproduction against published MTEB results. New output probes compare fse with the native libraries. Benchmark tooling only; no change to the library.

  • Add recipe 12 (recipes/12-fastapi-search.ipynb) (DR-285: Rezept für FastAPI mit async /search über einen Store schreiben): a FastAPI service with lifespan, a store opened in each process, async /search through anyio.to_thread.run_sync with a CapacityLimiter of the size of one shared fse.Pool, mode="exact", a view that is opened again at a fixed distance and swapped atomically, and a shutdown that waits for running searches; the second part replays three writers with shard=(i, 3) and one view over a shared folder. It checks that parallel requests give the bits of serial ones and that the view gives the bits of one store (claim class correctness, no timing). Network drives stay outside the crash contract. Pins fastapi, uvicorn and httpx in recipes/requirements.txt. Documentation only; no change to the library.

  • Add recipe 11 (recipes/11-reranker-candidates.ipynb, helpers in recipes/_rerank.py) and the reranker study of benchmarks/quality.py (DR-277: Rezept und Experiment für fse-Kandidaten mit nachgelagertem Reranker erstellen): fse finds candidates in a store with document IDs, the texts come from the user's own source, and any local Sentence Transformers cross-encoder orders the top N again; an optional section fuses fse with BM25 by Reciprocal Rank Fusion (k = 60). The study (cells of kind rerank in benchmarks/config/rerank-local.json, run with --run --local-run) measures NDCG@10 and recall of these arms on the 13 NanoBEIR tasks, class K. Documentation and measurement only; no change to the library, no reranker and no API client in fse.

  • Exact searches of int8 stores (mode="exact", and "auto" up to 8 queries) on arm64 CPUs with the dot-product extension skip rows that a provable int8 bound rules out and rescore the rest through the exact path of rescore; ids and score bits are unchanged, and anything outside the proven domain takes the previous path (DR-168: Suche mit int8-Vorfilter und beweisbarer Fehlerschranke beschleunigen, part 3, DR-278: int8-Vorfilter in die exakte Suche einbinden). The domain is 1 to 8 queries, k up to 100, 256 to 1024 values per row and at least 65,536 selected rows, without max_per_doc, rows= and doc_ids=. Adds the log event store.prefilter (DEBUG: queries, k, rows, pairs, fallback, workers). No public API and no persistence name changes.

  • Read sentence-transformers models whose table is float16, such as hotchpotch/bekko-static-embedding-v1, as fse float16 tables (DR-274: float16-Tabellen von sentence-transformers-Modellen als fse-float16 laden): Encoder.from_pretrained keeps the table unchanged (half the memory of float32), the kernel widens it exactly and every sum runs in float32, so the rows are the bits of the encoder of the widened table; truncate_dim and normalize work as for float32. sentence-transformers itself takes the mean of such a table in float16, so the comparison is with its weights made float32, within the bound of the float32 tables. Additive: models that loaded load as before, bit for bit; int8 and float64 tables of sentence-transformers models and every non-float32 table of Model2Vec models are still refused. No persistence name is added or changed.

  • Create the GitHub release in CI for every v* tag (DR-261: GitHub-Release bei jedem Release automatisch anlegen und den Ablauf als Skill festhalten): a new job github-release runs after the PyPI upload and is the only job with contents: write. It names the release fse X.Y.Z, uses the ## [X.Y.Z] section of this file as its text, attaches the sdist and the wheels, and marks aN, bN and rcN versions as pre-releases. tools/release_notes.py (with tests/test_release_notes.py) also runs before the PyPI upload and stops the release if the tag is malformed, the section is missing, empty or duplicated, or a built file does not carry the version of the tag. New repository skill .agents/skills/release. No change to the library.

  • Allow Encoder.from_pretrained(..., truncate_dim=k) for Model2Vec models (DR-260: truncate_dim auch für Model2Vec-Modelle erlauben): the encoder keeps the first k dimensions of the table, as it already does for sentence-transformers models, before normalize@1, and fse.source records the dims of the cut, which save, load, save_pretrained and quantize keep. "model" (the default) and None keep all dimensions, k equal to the width too, a larger k raises. The columns a cut drops are checked for NaN and infinity, so a corrupt table is refused with or without a cut (also for sentence-transformers models). fse encoders still refuse truncate_dim; without the argument nothing changes.

  • Add fse.Pool(workers=None, *, start=False, max_tasks=None) and the keyword-only pool= of the ten methods that tokenize text (DR-241: Heißen Pfad vor 2.1 profilieren und die besten ein bis zwei Engpässe beheben, rank 2, with DR-185: Optionalen Thread-Pool über mehrere Aufrufe anbieten). A pool is a set of threads that runs the tasks of any number of calls, so a server that encodes many small requests, or a loop over many small batches, no longer starts threads for every call; start=True starts them when the pool is made. A stream (encode_batches, predict_batches) now runs all its batches on one pool that lives for the iteration and is closed when it ends, fails, is closed or is collected, where each batch made its own threads before (the profile of the stream cells showed 59 to 81 % waiting and 15,000 to 28,000 thread entries for 3,660 calls). The default stays that no thread outlives a call; a pool passed with pool= is closed by its owner. workers= may not exceed pool.workers. Bits, errors and the order of commits are unchanged. Additive: v2.0.0 behaves exactly as before.

  • Add the guide External indexes on fse shards and recipe 10 (recipes/10-external-ann-indexes.ipynb, helpers in recipes/_ann.py) (DR-169: Shards optimal für externe ANN-Indizes vorbereiten, part 4): how FAISS, USearch and hnswlib take the rows of iter_batches (VectorBatch.normalized(), int8 codes without a copy), how VectorStore.rescore gives their hits the exact score, and how to add rows, continue after a stop, repair and use views. Documentation only; no change to the library.

  • Add VectorBatch.inv_norms and VectorBatch.normalized() (DR-169: Shards optimal für externe ANN-Indizes vorbereiten, part 3): every batch of VectorStore.iter_batches now holds the stored inverse norms of its rows as a read-only float32 view, and normalized() returns a new float32 array of approximately unit-length rows for an external index (FAISS, hnswlib): float32(stored) * inv_norm per element, int8 from the codes without the scale. The bits are fixed in the default IEEE environment; a batch built by hand is checked and refused with ValueError or TypeError if it does not fit. No new persistence name.

  • Add VectorStore.rescore(queries, candidates, /, k=10, *, workers=None, max_per_doc=None) (DR-169: Shards optimal für externe ANN-Indizes vorbereiten, part 2), which scores the hits of an external index (FAISS, hnswlib, USearch) exactly: for every query it returns the best k rows among its own candidate positions (-1 marks a gap; the lists may be unsorted, hold duplicates, be empty or shorter than k), bit for bit as search(query, k, mode="exact", rows=sorted(candidates without -1)), also with max_per_doc=1 and on views. A matrix result is filled up on the right with ids -1, scores -inf and document IDs -1; a single query is not. uint64 candidates are rejected. It adds the log event store.rescore (DEBUG: queries, k, pairs, read_rows, workers, max_per_doc when set), and no persistence name.

  • Add resuming of endless streams at a position that can be read again (DR-252: Unendliche Streams an einer wiederlesbaren Position fortsetzen, part 5 of DR-165: Shards und Resume für große Jobs einführen): Encoder.encode_batches(batches, sink=, positions=True) and predict_batches(..., positions=True) of Heads, Classifier and Regressor take batches (texts, position) (or (texts, doc_ids, position)) into sinks made with resume=True, a store of the new mode fse.resume=source_position@1. A position is an opaque text of 1 to 65,536 UTF-8 bytes (a Kafka offset, a byte offset); fse stores it and never reads it. Every shard keeps the position of the batch it ends in under fse.source_position and fse.source_position_row, and the root keeps the start of the stream. After a crash sink.source_position, also VectorStore.source_position and ScoreStore.source_position, is (P, skip): seek the source to P, give its batches again, and the first skip records, which the store holds, are dropped without being encoded; every other record is written as an uninterrupted run writes it, bit for bit. Every batch of the replay gives one result: first_row is the replay row of its first computed record, n_rows the records computed (0 for a batch the store holds) and durable_rows the replay rows that are durable, so a consumer commits its input up to there. A vector store and a score store written together resume together, also when they hold different numbers of rows (seek by the one that holds fewer). A store of this mode has a store.json of up to 1 MiB, members of a view show the position of each member, and a view reads stores of streams in the order given. A store of one resume mode is not resumed by the other: encode(texts, shard=) and predict(..., shard=) write source_rows@1, the batches with positions=True write source_position@1.

  • Add resumable score stores and resumable heads (DR-250: Score-Stores und Köpfe wiederaufnehmen, part 3 of DR-165: Shards und Resume für große Jobs einführen): fse.ScoreSink(..., resume=True) writes a score store of the mode fse.resume = source_rows@1 (the same six keys as a vector store). Heads.predict, Classifier.predict and Regressor.predict take the new keyword-only return_vectors=, vector_sink=, score_sink=, doc_ids= and shard=: with a sink made with resume=True the call resumes, runs only the rows the stores do not hold yet, bit for bit as an uninterrupted run, and returns None (with lengths the pair (None, LengthReport)). A vector store and a score store of one call may hold different numbers of rows: a first phase catches the shorter one up, a second runs the rest into both. The outputs of the heads are dropped, so the memory of a call does not grow with the rows. A sink made with resume=True is refused by predict_batches and by every other entry. Without the new keywords nothing changes.

  • Add resumable vector stores and fixed partitions (DR-248: Vektor-Stores wiederaufnehmen und feste Partitionen schreiben, part 1 of DR-165: Shards und Resume für große Jobs einführen): fse.ShardSink(..., resume=True) writes a store of the new mode fse.resume=source_rows@1, and Encoder.encode(texts, sink=sink, shard=(i, n)) takes part i of n of a source that can be read again (a list, tuple or array of str, token lists or the path of a text file; parts are whole blocks of 16,384 rows). The call hashes the whole source and compares its fingerprint, the encoder, the dtype and the partition with the store: another source or setup is a ValueError before any row is reserved. A stopped run continues with only the missing rows, bit for bit as an uninterrupted run, and a store that is complete is left alone. Under durability="full" the sink syncs the store directory once when it appends. A report of lengths= covers the rows of that call and its first_row is their source row. Streams and Arrow text columns are refused with a TypeError; encode_batches and the heads refuse a sink made with resume=True. The format (six keys: fse.source_fingerprint, fse.source_form, fse.execution_fingerprint, fse.source_first_row, fse.source_rows, fse.source_shard) is also read and repaired for score stores; their sink follows. These names freeze with 2.1. Recipe 03 uses the new calls.

  • Add VectorStore.open_view(paths) and ScoreStore.open_view(paths) (DR-251: Mehrere Stores als eine Sicht lesen, part 4 of DR-165: Shards und Resume für große Jobs einführen), which read several stores as one store whose rows follow each other, without merging anything on disk. Search and score with mode="exact", take_*, get_*, iter_batches and predict_batches give the bits and the order of ties of one store that holds the same rows (BLAS blocks follow the shards, so mode="blas" and "auto" above 8 queries differ in the last bits, as between two shard sizes). The stores must hold the same encoder, dtype, dimension and document-ID mode (scores: the same heads, columns and paths, and the same thresholds across each boundary); stores written with Encoder.encode(shard=) must be parts of one source (each part at most once, not mixed with stores without a source) and are read in the order of the parts; parts may be missing or unfinished (a snapshot, positions are the concatenation, source is None), and only if all parts are there and finished is a position a row of the source. A view has no path and no refresh (ValueError), verify runs over each store, and close closes them all. New read-only helpers on both store classes and views: paths, members (one mapping per store: path, first_row, rows, source), source (a mapping for a store written from a source, and for a view of all its parts) and locate(ids), which names the store and the row of each position. New events store.view_opened and scores.view_opened.

  • Add VectorStore.refresh() and ScoreStore.refresh(), which take the shards a writer has published since the store was opened into the store itself and return the number of new rows (an int; 0 if there are none). A reader of a store that is still written sees new shards without opening it again, and only the new shards are checked and, if tagged durability="none", hashed. Searches, scores and reads that run, and generators already made, keep the rows they started with: every public read now takes its shards and row count once, when it is called (a generator of iter_batches or predict_batches at the call, not at its first block). A gap, an orphan or a break of the chain raises CorruptArtifactError, a store whose root or last shard changed since it was opened raises RuntimeError; in both cases the rows stay as they were. A repair meanwhile is detected at best (repair_store stays offline). VectorStore.close raises while a refresh runs. New events store.refreshed and scores.refreshed (DEBUG). (DR-249: Stores während des Schreibens mit refresh() nachladen)

  • Add lengths=, count_truncated= and on_truncation= to the ten methods that tokenize text (Encoder.encode and encode_batches, Heads.predict and predict_batches, Classifier.predict, predict_proba, decision_function and predict_batches, Regressor.predict and predict_batches), which show what the length rule of the model cut. lengths="summary" or "per_row" returns (result, fse.LengthReport) (a 2.0 return type becomes a pair, with a sink (None, report)), makes every batch of a *_batches call the pair (batch, report) and gives HeadsBatch the field lengths; EncodedBatch and PredictedBatch are unchanged. The report holds the sums (rows cut by characters, by tokens, empty rows, tokens, unknown ids dropped) and with "per_row" the same per row; count_truncated=True adds the tokens and characters cut. on_truncation="raise" raises the new fse.TruncationError(ValueError) (row, cut, limit) for the first sentence the rule would cut. Without the keywords nothing is recorded and every result is bit for bit the same; a VectorStore input with them raises before a row is read. A sentence that reaches max_tokens costs at most one more word for the flags, and counting tokenizes the rest of a cut sentence. (DR-163: Sichtbar machen, was beim Tokenisieren gekürzt wurde)

  • Let Classifier.predict_batches and Regressor.predict_batches write to sinks without fse.Heads (DR-243: Umwege über fse.Heads bei einzelnen Köpfen entfernen): the new keyword-only vector_sink=, score_sink= and doc_ids= work as on Heads, and a single head and a group of one write the same files and append to each other's stores. sink= keeps exactly its role and behavior of 2.0 and excludes the new keywords; a VectorStore input refuses them (the outputs of a store keep going to sink=). ScoreSink(path, model, name=...) and ScoreStore.open(path, heads=model, name=...) take a single head; the name is required, because it is written to the store. ScoreSink(store=...) is now stored=... (unreleased, no alias). New LinearModel.path(return_vectors=...) names the path a head takes, as Heads.paths does for a group. One validator checks doc_ids for encode_batches, Heads.predict_batches and single heads: a wrong type is a TypeError, a wrong value a ValueError, and the message names the method. Shard sinks of fse in document-ID mode are refused when the call brings no IDs, but a sink of your own with has_doc_ids True runs as in 2.0, also in encode and encode_batches (DR-240 had refused it).

  • Add VectorStore.take_vectors(rows, /) and take_doc_ids(rows, /), which read the rows at a list of store positions (a list or an int64 array of one or two dimensions, so result.ids works as it is) in that order with duplicates kept, and VectorStore.iter_batches(block_rows=, start=, stop=) with the new fse.VectorBatch, which gives blocks of read-only views of the mapped shards (int8 as QuantizedRows). Negative positions raise. docs/document-ids.md lists store positions apart from run positions. (DR-237: Vektoren und Dokument-IDs nach einer Liste von Zeilen und blockweise lesen)

  • Add durable scores of a group of heads (DR-162, Scores und Vorhersagen dauerhaft speichern): Heads.predict_batches(..., score_sink=sink, doc_ids=...) writes the scores of every head to a score store (fse.ScoreSink(path, heads, stored=None, doc_ids=False, shard_rows=None, durability="full")), and fse.ScoreStore reads it without an encoder: iter_batches gives the stored values as read-only views, predict_batches gives labels or values as Heads does, with new multilabel thresholds through thresholds=, without new inference. A classifier stores its decision (the float32 logits, from which probabilities and labels follow bit for bit) or its probabilities, a regressor its values. The format prediction_scores 1 (store_version 2) holds one tensor scores for all heads, optional doc_ids and thresholds.<head>; store.json may be 1 MiB (stores of vectors keep 64 KiB); a head keeps the path it ran on (folded, unfolded, ...) and an append with another path is refused. HeadsBatch.durable_rows reports the durable rows of a run; fse.ScoreBatch carries store_row, outputs and doc_ids. The Encoder refuses a ScoreSink, and so does sink= of Classifier and Regressor, with a hint to score_sink=. fse.repair_store repairs both kinds of stores, and store.repaired gains the field kind. New events scores.sink_opened, scores.shard_published, scores.sink_closed, scores.opened and scores.verified on fse.store. A root whose fse.kind names no kind of fse is now an UnsupportedArtifactError; with keys of its own it was a CorruptArtifactError before.

  • Add vector_sink= to Heads.predict_batches (DR-229: Vektoren und Scores in einem Lauf in zwei Stores schreiben): one pass writes the vectors of the encoder to a ShardSink (or any fse.Sink) and, with score_sink=, the scores to a ScoreSink, two folders with the same doc_ids; HeadsBatch.durable_rows is the minimum of both sinks. vector_sink needs return_vectors=True (heads on raw means take the unfolded path). Each folder survives a process crash on its own, and a power loss under durability="full"; rows beyond the confirmed count may be in one folder only and are written again on a replay. Two shard stores of unequal length take a run only with doc_ids. Classifier and Regressor take the same vector_sink= (DR-243: Umwege über fse.Heads bei einzelnen Köpfen entfernen).

  • Export fse.DocumentSearchResult, the type VectorStore.search returns with document IDs; ThresholdInput is no longer exported (it was never released). (DR-240: Kleine API-Korrekturen aus dem zweiten API-Review vor 2.1 umsetzen)

  • Add fse.Heads and fse.HeadsBatch: an immutable, read-only group of named heads of one encoder (Heads(quality=q, toxic=t), names [a-z][a-z0-9_]*, equal encoder fingerprints, all complete or all standalone) whose predict and predict_batches tokenize each text once and score all heads in one pass, with the bits of each head alone (softmax, multilabel, regression, folded, normalized and mixed heads, with or without the vectors). Heads.paths shows the path each head takes. A lone Classifier or Regressor now runs through the same internal runtime with unchanged public API, outputs and sinks. The group has no store mode and no file format of its own; its sinks are the vector_sink= and score_sink= of later entries. (DR-161: Mehrere Köpfe in einem Durchgang rechnen (Softmax und Mehrfach-Labels))

  • Import finished linear Model2Vec classifier heads (model2vec 0.10): Classifier.from_pretrained also reads the export of a StaticModelPipeline (local folder or Hub repository at one commit), and the new Classifier.from_model2vec takes a StaticModelPipeline or a trained StaticModelForClassification (only to_pipeline runs). One linear layer with softmax (softmax@1) or sigmoid (multilabel@1, cutoffs 0.5, changeable with with_thresholds or thresholds=), the fine-tuned table with its token weights multiplied in, normalize and the classes in order. Hidden layers, regression heads, quantization, separate token weights, non-float32 heads, a legacy pipeline.skops export (never opened) and unknown keys raise with a reason; nothing is trained and no pickle or skops is opened. Float64 tables and separate token weights of any Model2Vec model now raise UnsupportedArtifactError in Encoder.from_pretrained too (before: KeyError: 'F64' and the vocabulary quantization message). (DR-164: Lineare Model2Vec-Köpfe importieren)

  • Add max_per_doc=1 to VectorStore.search: the k best documents per query, each by its best-scoring row (ties to the smaller position), combinable with doc_ids and rows filters. Only the value 1 is allowed in this version; larger values may follow. Behavior change: search now returns at most k results, min(k, candidates), also when k is larger than the number of rows of an unfiltered search, which earlier raised a ValueError; empty results have the usual shapes and types. Exact mode keeps the bits of the ordinary search for the returned rows. (DR-218: Suche mit höchstens einem Treffer je Dokument anbieten)

  • Import Platt-calibrated scikit-learn classifiers (CalibratedClassifierCV with method="sigmoid" and ensemble=False, binary or one-vs-rest) by folding the calibration into the head's weights and bias in float64 under the kernels' floating-point environment, rounded once to float32. Add softmax temperature to Classifier.from_arrays (finite, greater than zero). Both write linear_head format 2 with the label rule linear@2, now also defined for single-label heads as the first class of largest probability, so predict costs more than under linear@1; no new metadata key. ensemble=True or "auto", isotonic and temperature calibration, and calibrated OneVsRestClassifier estimators are refused. Existing format 1 heads keep their identifiers, fingerprints and predictions. (DR-227: Kalibrierte Klassifikatoren importieren und Softmax-Temperatur unterstützen)

  • Add pydantic-validated thresholds, read-only classifier thresholds and immutable with_thresholds updates. Allow explicit class names for all sklearn classifier imports, preserving output order, and document external training through deployment. (DR-226: Mehrfach-Labels mit benannten Schwellen unterstützen)

  • Add independent multilabel classifiers (multilabel@1, linear@2) with strict per-label thresholds, including partial named overrides such as thresholds={"sport": 0.4}. Import fitted multilabel OneVsRestClassifier models with supported affine sigmoid heads. New models use linear_head format 2 and head.thresholds; existing format 1 models keep their identifiers, fingerprints and predictions. (DR-199: Lineare Köpfe zum vollwertigen Bestandteil von fse machen)

  • Add document-ID write, unfiltered-search and filtered-search cost profiles to shared local diagnosis, including normalized public-call latency and filter-workspace measurements. Numerical checks, timing floors and resource guards remain enforced.

  • Add optional int64 document IDs to vector stores, per-batch ID input, and exact filtered search by document IDs or physical rows. Duplicate document IDs retain every matching row. New ID-enabled stores use format 2 while vector-only stores remain format 1. DR-215 (Eigene Dokument-IDs im Store speichern (row_ids)).

  • Add an evidence-grounded fse walkthrough skill with offline synthetic demonstrations and one-item pacing. Skill-only instruction and metadata changes skip heavy PR and ordinary master CI jobs; release publication still requires full master CI.

  • Add a shared fse-benchmark skill for Codex and Claude Code, with measurement planning, validity checks and sourced lessons.

Changed

  • The documentation, the docstrings and the recipes now match fse 2.0 and the code of master (DR-326: Veraltete Dokumentation und Mac-Tempozahlen vor 2.1 bereinigen). The recipes label their timings with the Apple M1 Pro MacBook that produced them, dead links and local paths in notebook output are gone, and the migration guide, the security policy and the architecture pages describe the current behavior.

  • Correct the remaining contradictions between the documentation and the code (DR-336: Weitere Widersprüche der Dokumentation aus der Prüfung vor 2.1 korrigieren). The README marks which examples need fse 2.1 and lists all recipes, the architecture pages state the tokenizer, dtype, resume and shard-chain rules as the code applies them, the new docs/README.md indexes the guides, and the docstrings of public properties, Encoder.encode, ShardSink, Classifier.load and Regressor.load describe their parameters. Text, docstrings and two recipe code cells only; no library behavior changes.

  • Collect user-visible changes as separate towncrier fragments instead of editing the shared Unreleased section (DR-343: CHANGELOG-Einträge als Fragmente mit towncrier sammeln). Release preparation assembles them into CHANGELOG.md, and publishing refuses leftover fragments.

  • Raise fse_revision in benchmarks/config/pins.json from a1e9bf5 to 9a63400, the master after the fixes of the user stress test of DR-289 (Nutzer-Stresstest über alle Funktionen von 2.0 bis 2.1 mit Codex durchführen), so the Linux studies install the 2.1 candidate that the version comparison against fse 2.0.0 measures (DR-350: fse_revision auf den 2.1-Kandidaten für die Benchmarks setzen). Historical results keep their own pin. Measurement tooling only; no change to the library.

  • Document that binary search with oversample=None uses factor 40 throughout fse 2.x,
    including minor releases. Other factors require an explicit positive integer.

  • Reject unordered sets and mappings as class labels, boolean elements mixed into head weights, biases or word-vector indices, and masked arrays before conversion discards their masks, including temperature imports. These stricter checks affect existing 2.0 APIs. Ordered dictionary keys remain accepted. Valid inputs retain their float32 bits.

  • Narrow the new 2.1 surface before the freeze (DR-322: Neue öffentliche 2.1-Oberfläche vor dem Freeze bereinigen (Pool, score_schema, dropped_total)). fse.Pool keeps its constructor, workers, max_tasks, closed, close() and the with block; submit, try_acquire, acquire, release, check_open and inside are internal (submit bypassed max_tasks). ScoreSink.score_schema is internal; read what a store holds through ScoreStore.heads. LengthReport.dropped_total is now dropped_unknown_total, the sum of dropped_unknown (unknown ids removed by fse.unknown=drop; tokens cut by the length rule are truncated_tokens_total). All three names were never released, so there are no aliases; no persistence names change.

  • Raise fse_revision in benchmarks/config/pins.json from 624251d (fse 2.0) to a1e9bf5, the master after DR-317 (DR-320: Pin der fse-Revision für die Linux-Studien auf den aktuellen master heben), so the Linux studies install an fse with fse.Heads. Historical results keep their own pin. The speed allocation gains a base environment (fse 2.0 at 624251d, lock speed-base.lock.txt) passed as --base-python, so --study all is no longer refused for its versions cells; the plans of the existing cells are unchanged. The development Dockerfile now uses the image of pins.json (CPython 3.13.18) instead of 3.13.16-slim. The speed rehearsal machine check allows 6 GiB instead of 4 GiB, because the bekko cells fail in 4 GiB with a MemoryError (maintainer approval 2026-11-08). Measurement tooling only; no change to the library.

  • Start sink threads only with their first work (DR-317: Sink-Threads eines nicht geschlossenen Sinks nicht unbefristet weiterlaufen lassen). Constructing or inspecting a ShardSink or ScoreSink starts no thread. Write and hash start at the first commit, publish at the first finished shard. Used sinks still require close() or a with block. Thread-start failures follow the shared writer error and cleanup protocol.

  • Serve development lock waiters in registration order and report queue position (DR-315: Wartende an der Heavy-Sperre in Ankunftsreihenfolge bedienen). Canceled and crashed waiters leave the queue; existing lock owners are never taken over. FIFO requires draining old waiting controllers and wrappers before rollout. Measurement tooling only.

  • Name the claim classes of measurements correctness, descriptive, regression, optimization and publication (previously K/D/R/O/P) (DR-294: Messklassen mit lesbaren Namen statt Einzelbuchstaben benennen), in the contributor guides, the measurement guide, the study descriptions and the local quality runs, whose rows now record "class": "correctness". Results written before keep their one-letter value; benchmarks/quality.py --summarize maps it to the same class, so old and new rows of one configuration give the same summary.

  • Make the task of a head give up the GIL far less often (DR-241: Heißen Pfad vor 2.1 profilieren und die besten ein bis zwei Engpässe beheben, rank 1): the checks of the backend methods compared arrays with np.may_share_memory, which releases the GIL, up to 20 times per task of fse.Heads (11 for one Classifier); they now compare the byte bounds without a numpy call (fse._backends._base_checks.overlaps, and _overlaps in the extension), with the same result. A task whose heads read only the folded table also pools in one section without the GIL, as Encoder.encode does, instead of building and checking a Batch with about ten more numpy passes over its tokens; a failed task is run again on the old path, which raises the error of its row. No arithmetic, API or format changes: the outputs are bit-identical to those of the numpy reference.

  • VectorStore.close now raises RuntimeError("the store is in use ...") while a search, score or read runs; the message said "being searched" before. (DR-237: Vektoren und Dokument-IDs nach einer Liste von Zeilen und blockweise lesen)

  • ScoreStore.iter_batches and predict_batches raise RuntimeError after close() also before the first block; they ended quietly before. The functions are unreleased; their entry for DR-162 (Scores und Vorhersagen dauerhaft speichern) stands in this section. (DR-242: ScoreStore.iter_batches nach close() vor dem ersten Block mit Fehler beenden)

  • A second run into a ShardSink raises an error that says a sink takes exactly one run. Add EncodedBatch.vectors, an alias of rows. Explain store positions, run positions, batch lengths, durable prefixes and document IDs in docs/document-ids.md. (DR-240: Kleine API-Korrekturen aus dem zweiten API-Review vor 2.1 umsetzen)

  • Document which rows the sink of predict_batches takes with and without return_vectors in a table, and name that role in the error for a sink of the wrong width. (DR-240: Kleine API-Korrekturen aus dem zweiten API-Review vor 2.1 umsetzen)

  • A sink is in document-ID mode only with has_doc_ids exactly True; other values (0, 1, None) are ignored unless the call brings document IDs, where a value that is no bool raises a TypeError, so sinks of 2.0 keep running. encode(doc_ids=True) raises a TypeError that names the difference to encode_batches(doc_ids=True). A shard sink of fse in document-ID mode is refused before a batch is read when the call brings no IDs, with a hint for the working route; a sink of your own with has_doc_ids True runs as in 2.0 (DR-243: Umwege über fse.Heads bei einzelnen Köpfen entfernen). Document has_doc_ids and commit(..., doc_ids=) of fse.Sink. (DR-240: Kleine API-Korrekturen aus dem zweiten API-Review vor 2.1 umsetzen)

  • Classifier.thresholds is None for heads without cutoffs instead of raising; with_thresholds still raises ValueError there. (DR-240: Kleine API-Korrekturen aus dem zweiten API-Review vor 2.1 umsetzen)

  • Rename the unreleased document-ID persistence names of store format 2 from row_ids to doc_ids (doc_ids.npy, fse.doc_ids, fse.doc_ids_min, fse.doc_ids_max, fse.doc_ids_sorted); stores written by a pre-release of 2.1 are refused with UnsupportedArtifactError (also by verify()) and must be written again. Add VectorStore.has_doc_ids and VectorStore.get_doc_ids(start=, stop=). DocumentSearchResult keeps doc_ids in _replace, _asdict and repr, and _make raises instead of dropping them. encode_batches takes doc_ids=True for (texts, doc_ids) batches and checks it against the sink when called. DR-235 (Dokument-ID-Persistenz vor 2.1 in doc_ids umbenennen und Lesewege ergänzen).

  • Refuse local speed.py --diagnose timing runs whose controller does not run at nice 0, before any directory, preparation or budget is used, and name zsh's default BG_NICE (which adds 5 to the nice value of cmd & jobs) as a usual cause. Diagnostic rehearsals and the prepare phase are exempt, measured children still require nice 0 and the heavy wrapper for builds and checks still targets nice 10. (DR-234: zsh-Falle BG_NICE bei lokalen Messungen abfangen und dokumentieren)

  • Require a Linux speed comparison only for published speed numbers and release guarantees (class P) or when the maintainer has explicitly approved a Linux machine for the ticket. Pull requests name their claim classes and their actual Linux comparison status, and state explicitly when none was run. Numerical conformance, bit identity and persistence guarantees are unchanged.

  • Make all macOS PR jobs optional, including wheels. Add a targeted manual macOS scope bound to an open PR and its immutable current head SHA, with explicit source/artifact identities and stale-head rejection. Preserve automatic master/tag coverage, full manual verification and release gates. CI summaries fail closed on failed, cancelled, abandoned or unknown started jobs.

  • Couple measurement effort to claim classes. Add candidate-only descriptive costs, a conservative five-pair regression screen, small independent adapter probes and a total local wall-time admission budget including preparation and load/lock waits. Existing numerical, persistence, optimization and Linux publication guarantees remain unchanged.

  • Limit macOS tests on pull requests and release tags to Python 3.13, alongside arm64 wheel builds. Master pushes retain the full Python 3.11 to 3.15 macOS matrix and sentence-transformers comparison. Release publication now requires successful full master CI for the exact tagged commit, verified through the GitHub API.

  • Avoid a second stored-row copy when rescoring candidates, including stage 2 of binary search (DR-310: Doppelte Zeilenkopie beim exakten Nachrechnen in rescore und der binären Suche vermeiden). Sort private candidate sections in place and gather ascending parts directly into their output. Scores, IDs and tie order keep the same bits. Public candidate arrays remain unchanged.

Removed

  • From the next release, macOS Intel (x86_64) wheels are no longer built, tested or published. macOS wheels target Apple Silicon (arm64) only. Source distributions remain available, but builds on Intel Macs are unverified and no longer tested in CI. Existing 2.0.0 wheels remain available.

Fixed

  • Describe the thresholds of ScoreStore.predict_batches correctly (DR-244: Doku und CHANGELOG für 2.1 auf den Endstand bringen): the default thresholds=None keeps the stored cutoffs of every head, and only a head mapped to None resets its labels to 0.5. The behavior is unchanged. The fse module docstring now also names Heads, ShardSink, VectorStore, ScoreSink, ScoreStore and doc_ids.

  • MemorySink, ShardSink and ScoreSink now reject boolean, floating-point, negative and oversized row arguments before allocation, reservation or commit state changes. Target sizes must fit the platform array index and byte-size limits, including sinks used by Heads. Targets spanning shards allocate their own buffer before reserving any shard rows, so a failed allocation cannot create a large chain of reservations. Python and NumPy integers remain supported.

  • FastText parameters accept NumPy integers like other integer API parameters, check their
    types and bounds before conversion, and store them as Python integers. Invalid types,
    including booleans and floats, raise an error naming the affected parameter instead of
    failing during JSON serialization.

    Batch calls with a resume sink now explain the supported positions=True route for
    Encoder, Classifier, Regressor and Heads, alongside the eager and non-resume alternatives.

  • Correct the documentation of nested calls on a pool: inline text encoding and prediction calls are allowed, while reentry into an already active pool guard is refused, including single-task store calls with more than one worker. An outer inline text call does not activate that guard. Clarify that views returned by VectorStore and ScoreStore remain readable after close, and that an unknown keyword to StaticModel.from_pretrained raises pydantic.ValidationError.

  • Recipe helpers reject masked ANN candidate keys, hit counts, and validity flags before conversion, and the reranker recipe downloads complete pinned model snapshots into its temporary workspace. Integration instructions reflect the published fse and adapter packages, and the recipe guide includes installation with fse from PyPI. Recipe provenance records the installed fse version and permits release archives without Git metadata.

  • Store search and rescoring now reject masked queries and booleans mixed into
    numeric queries. Store scoring also rejects these inputs in weights and bias,
    before conversion can discard a mask or turn a boolean into a number.
    Mappings and sets are rejected before conversion where an ordered sequence is required.

  • Reject masked arrays with TypeError before copying or taking ownership of word-vector tensors in WordVectors.from_arrays and Encoder.from_arrays, including scales, ngrams and vocabulary arrays. The gensim adapter also rejects masked source rows before conversion. Classifier labels and internal encoder weights reject masks instead of silently discarding them. Scikit-learn and Model2Vec imports also validate original weights, biases, labels and embedding tables before conversion; scikit-learn numeric attributes reject boolean arrays and scalars.

  • The first table fold for a classifier, regressor or group of heads now respects the call's worker limit and shared pool. The caller helps when pool capacity is occupied, avoiding deadlocks with nested sink calls. Concurrent first calls share one completed fold; later calls keep their own execution settings.

  • Loading structurally damaged WordVectors, encoders and linear heads now raises CorruptArtifactError. Unknown formats still raise UnsupportedArtifactError, and array constructors retain their argument errors.

  • Pool now validates constructor arguments consistently with the other public
    entry points, raising pydantic.ValidationError for invalid worker counts,
    task limits or start flags. Positive Python and NumPy integer counts remain accepted.
    Its docstring now distinguishes task places held until commit during text encoding
    and prediction from store task places released at task completion, with finished
    store results bounded by each call's own window.

  • Fix the unreleased 2.1 ScoreStore outputs of a one-label multilabel classifier to keep shape (n, 1), including empty selections and joined views. Head and store outputs now share one shape rule. Threshold updates validate supplied label keys before overlaying current cutoffs.

  • Fix the type check of k and capacity in the int8 prefilter pass under Cython 3.1.0 and 3.1.1 (DR-329: Typprüfung von k und capacity im int8-Vorfilter unter Cython 3.1.0 reparieren). Those versions compile isinstance(x, int | np.integer) to a check that is always false (Cython issue 6420, fixed in 3.1.2), so int8_pass(..., k=10) raised ValueError. Both checks use the tuple form now, the build requires Cython>=3.1.2, and the three measurement locks (speed-main, speed-base, quality, and the input quality.in) pin cython==3.1.8 instead of 3.1.0, which changes the package identity of the measurement state. The prefilter is not enabled on master, so no user was affected.

  • Return (result, vectors) from Heads.predict, Classifier.predict and Regressor.predict with return_vectors=True and no sinks (DR-312: return_vectors=True in Heads.predict ohne Sinks nicht still übergehen); previously the call raised ValueError. With lengths the return is (result, vectors, LengthReport). Outputs and encoder vectors match predict_batches(return_vectors=True) bit for bit, including the unfolded path of raw-mean heads. Resume sink returns stay unchanged. Standalone heads, VectorStore inputs and Regressor.predict(sink=...) refuse this switch.

  • Write manifest.json of npy artifacts and store shards with LF line endings on every platform; Windows wrote CRLF before. Files, checksums and fingerprints are unchanged on Linux and macOS, and CRLF manifests written by 2.0.0 on Windows stay readable. (DR-238: Manifeste auf allen Plattformen mit LF-Zeilenenden schreiben)

  • Preserve subnormal multilabel decisions under foreign floating-point settings and reject Hub folders whose declared format version disagrees with the model. Correct the one-label multilabel output description. (DR-226: Mehrfach-Labels mit benannten Schwellen unterstützen)

  • Reject already released Arrow document-ID schemas before reading their fields.