📢 Spark NLP 6.4.2: BM25 Retrieval, SaT Sentence Detection, and Model Conversion Workflows

Spark NLP 6.4.2 is a feature and reliability release focused on strengthening retrieval, sentence segmentation, model conversion, and release infrastructure. This release introduces BM25Approach and BM25Model for Okapi BM25 lexical retrieval, adds SentenceDetectorSaTModel for transformer-based sentence boundary detection with SaT / Segment any Text models, and expands model conversion examples with new Hugging Face to Spark NLP notebooks for ONNX, OpenVINO, and GGUF workflows.

In addition, this release improves production robustness by properly closing resource streams and file systems during local resource copying, which avoids leak warnings that can affect dependent pipelines. It also updates jsl-llamacpp to 2.0.3.

🔥 Highlights

BM25Approach + BM25Model: New Okapi BM25 lexical retrieval annotators implemented as a fit-once, query-many Estimator/Model pair. They complement semantic retrieval components such as DocumentSimilarityRanker with a fast, corpus-statistics-based lexical ranking option.
SentenceDetectorSaTModel: New sentence detector based on SaT / Segment any Text transformer models. It uses ONNX-backed XLM-R SentencePiece models to produce sentence spans from per-token boundary probabilities and supports long documents through overlapping token windows.
Model conversion notebooks: New end-to-end notebooks demonstrate Hugging Face to Spark NLP conversion flows for ONNX, OpenVINO, and GGUF models.

🚀 New Features & Enhancements

BM25 Lexical Retrieval

Spark NLP 6.4.2 introduces BM25 lexical document retrieval through a two-stage Spark ML design. This implements SPARKNLP-1145 and adds a purely lexical ranking option that complements semantic retrieval components such as DocumentSimilarityRanker.

BM25Approach scans the tokenized corpus during fit() and learns corpus-level statistics: document count, per-term document frequency, average document length, and inverse document frequency.
BM25Model reuses those learned statistics at transform time to score each document against a query.

This design is necessary because BM25 scores depend on corpus-level statistics that are only known after fitting over the full dataset. Once fitted, the model can be reused across many queries without rescanning the corpus.

Key capabilities:

Emits BM25_RANKINGS annotations.
Adds bm25_score, num_query_terms_matched, query, and doc_len metadata.
Supports range-validated k1, b, minDocFreq, and caseSensitive parameters.
Uses the non-negative Lucene / Elasticsearch-style IDF variant.
Supports setQuery(...) for raw query strings.
Supports setQueryTokens(...) for pre-analyzed query terms when the corpus pipeline includes normalization, stemming, lemmatization, or other token transformations.
Makes caseSensitive read-only on fitted BM25Model to avoid corrupting scores after the IDF vocabulary has been built.
Includes Scala and Python tests for ranking, save/load, exact-score verification, query reuse, parameter propagation, analyzer symmetry, and invalid parameter handling.

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import Tokenizer, StopWordsCleaner, BM25Approach
from pyspark.ml import Pipeline
from pyspark.sql.functions import col, explode

corpus = spark.createDataFrame([
    (1, "Apples are a great source of dietary fiber and vitamin C."),
    (2, "Machine learning uses neural networks and statistical models."),
    (3, "Vitamin C deficiency can affect the immune system."),
], ["id", "text"])

document = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")

stop_words = StopWordsCleaner() \
    .setInputCols(["token"]) \
    .setOutputCol("clean_token") \
    .setCaseSensitive(False)

bm25 = BM25Approach() \
    .setInputCols(["clean_token"]) \
    .setOutputCol("bm25_rankings") \
    .setK1(1.2) \
    .setB(0.75) \
    .setMinDocFreq(1) \
    .setCaseSensitive(False)

pipeline = Pipeline(stages=[document, tokenizer, stop_words, bm25])
model = pipeline.fit(corpus)

# Re-query the fitted model without recomputing corpus statistics.
model.stages[-1].setQuery("vitamin C health")
result = model.transform(corpus)

result.select(col("id"), explode(col("bm25_rankings")).alias("ranking")) \
    .select(
        col("id"),
        col("ranking.metadata")["bm25_score"].cast("double").alias("bm25_score"),
        col("ranking.metadata")["num_query_terms_matched"].cast("int").alias("terms_matched"),
    ) \
    .orderBy(col("bm25_score").desc()) \
    .show(truncate=False)

For pipelines that transform tokens before BM25 training, use setQueryTokens(...) with query tokens produced by the same analysis pipeline. This avoids query/document analyzer asymmetry and ensures query terms match the learned IDF vocabulary.

See the new BM25 Retrieval notebook for a complete example.

SentenceDetectorSaTModel

This release adds SentenceDetectorSaTModel, a new transformer-based sentence segmentation annotator built around SaT / Segment any Text models.

SentenceDetectorSaTModel predicts sentence boundaries from per-token probabilities using an ONNX-backed XLM-R SentencePiece model. It supports documents longer than a single model window by slicing text into overlapping sub-word token windows, merging the boundary probabilities, and projecting them back to character spans.

Key capabilities:

Input annotation type: DOCUMENT.
Output annotation type: DOCUMENT.
Supports SaT ONNX models such as segment-any-text/sat-12l-sm and segment-any-text/sat-12l.
Default pretrained model: sat_12l_sm with language xx.
Supports long text via blockSize and stride.
Supports batched ONNX inference via satBatchSize.
Supports overlap weighting strategies: hat and uniform.
Supports whitespace trimming and sentence-row explosion behavior.
Supports length-constrained segmentation with minSentenceLength and maxSentenceLength.
Supports local model loading through loadSavedModel(...) when the model folder contains model.onnx and assets/sentencepiece.bpe.model.

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import SentenceDetectorSaTModel
from pyspark.ml import Pipeline

assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

sentence_detector = SentenceDetectorSaTModel.pretrained() \
    .setInputCols(["document"]) \
    .setOutputCol("sentence")

pipeline = Pipeline().setStages([assembler, sentence_detector])
data = spark.createDataFrame([["This is a sentence. This is another one."]]).toDF("text")

result = pipeline.fit(data).transform(data)
result.selectExpr("explode(sentence.result) as sentence").show(truncate=False)

A new example notebook demonstrates Hugging Face ONNX usage with SentenceDetectorSaTModel:

HuggingFace ONNX in Spark NLP SentenceDetectorSaT

Hugging Face to Spark NLP Model Conversion Notebooks

Spark NLP 6.4.2 adds three end-to-end model-conversion notebooks under examples/python/model-conversion/:

These notebooks make it easier to validate and document model import paths across the main runtime families used by Spark NLP: ONNX, OpenVINO, and GGUF.

Runtime, Build, and Maintenance Updates

This release also includes several infrastructure and maintenance improvements:

Resource cleanup: ResourceHelper now closes copied resource streams and associated file systems, avoiding stream/file-system leak warnings that could affect dependent pipelines even when the underlying issue appears as a warning rather than a hard error.
jsl-llamacpp upgrade: The jsl-llamacpp dependency is bumped from 2.0.0 to 2.0.3.
CI hardening: GitHub Actions now disables sbt/setup-sbt@v1 disk-cache behavior in Spark 3.3, 3.4, and 3.5 build jobs to avoid unrelated hashFiles(...) cache hashing failures before the Spark NLP build starts.
Issue template routing: GitHub issue templates now route new bug reports, documentation improvements, and feature requests to the current maintainer by default, improving visibility and triage ownership.

🐛 Bug Fixes

Fixed resource stream cleanup in ResourceHelper by closing the copied resource and the associated file system.
Fixed an empty similarity/__init__.py issue in Python API wiring.
Hardened GitHub Actions Spark build jobs by disabling sbt/setup-sbt disk cache.

✅ Tests and Validation

Validation added or updated in this release includes:

Scala BM25TestSpec coverage for statistics learning, ranking, query reuse, save/load, exact-score verification, non-default parameters, minDocFreq pruning, analyzer symmetry, and parameter-range rejection.
Python bm25_test.py coverage for fit/query/reuse, save/load, parameter propagation, exact score checks, setQueryTokens(...), and read-only caseSensitive behavior.
The BM25 retrieval notebook was validated in Google Colab against a built Spark NLP JAR.
Scala SentenceDetectorSaTSpec coverage for the new SaT sentence detector.
Python sentence_detector_sat_test.py coverage for the Python wrapper.
Generated Scala and Python API docs refreshed for the new annotators.

❤️ Community Support

Slack real-time discussion with the Spark NLP community and team
GitHub issue tracking, feature requests, and contributions
Discussions community ideas and showcases
Medium latest Spark NLP articles and tutorials
YouTube educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.4.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.2

Maven

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

FAT JARs

CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.4.2.jar
GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.4.2.jar
Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.4.2.jar
AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.4.2.jar

What's Changed

Properly close stream and avoid warning #14774 by @albertoandreottiATgmail
[SPARKNLP-1384] Add Hugging Face to Spark NLP model conversion notebooks for ONNX, OpenVINO, and GGUF #14775 by @AbdullahMubeenAnwar
[SPARKNLP-1145] Add BM25 lexical retrieval annotator, including BM25Approach, BM25Model, Scala/Python APIs, docs, tests, and example notebook #14776 by @AbdullahMubeenAnwar
[SPARKNLP-1395] Update GitHub issue routing, disable fragile sbt disk-cache behavior, and bump jsl-llamacpp to 2.0.3 #14777 by @danilojsl
Introduce SentenceDetectorSaTModel sentence segmentation annotator with Scala/Python APIs, tests, and ONNX example notebook #14782 by @ahmedlone127
Spark NLP 6.4.2 release aggregation, version bump, generated Scala API docs, and generated Python API docs #14778 by @danilojsl

Full Changelog: 6.4.1...6.4.2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

6.4.2

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

📢 Spark NLP 6.4.2: BM25 Retrieval, SaT Sentence Detection, and Model Conversion Workflows

🔥 Highlights

🚀 New Features & Enhancements

BM25 Lexical Retrieval

SentenceDetectorSaTModel

Hugging Face to Spark NLP Model Conversion Notebooks

Runtime, Build, and Maintenance Updates

🐛 Bug Fixes

✅ Tests and Validation

❤️ Community Support

💻 Installation

Python

Spark Packages

CPU

GPU

Apple Silicon

AArch64

Maven

spark-nlp

spark-nlp-gpu

spark-nlp-silicon

spark-nlp-aarch64

FAT JARs

What's Changed

Contributors

Uh oh!