Clarifying the two weighting mechanisms

[mrienstra]

I was confused and briefly thought that the default title weight of 5.0 was comparable to the default h3 weight of 5.0. Below is the result of my attempts to find clarity. Note that this is largely LLM-written, however it has been through many revisions. Some of the quirks (such as which other codebases were analyzed to try to characterize the flavor of BM25 in pagefind) are my own -- I just picked a few that seemed reasonably popular (and one which is itself a comparison of approaches).

Perhaps bits of this could serve as starting points for new docs?

---

Pagefind Scoring Weights: Heading Weights vs. Title Weight

Pagefind exposes two distinct weight systems, documented separately:

• Content weights (Weighting) — a pre-BM25 mechanism, controlled via heading level or data-pagefind-weight. They scale the term-frequency input for words in a section of the page body before the BM25 body-ranking calculation, and also affect how those words are weighted for excerpt selection and display.
• Meta weights (Ranking) — a post-BM25 mechanism, controlled via options.ranking.metaWeights. They scale an additive score boost derived from metadata field matches, which is added on top of the BM25 body score.

The docs present both systems with similar-looking numbers, but they affect different stages of scoring. Content weights act inside body-text ranking; meta weights act as a separate metadata boost added afterward.

Heading weights (content weights)

Content weights multiply the term-frequency input for those words by N before BM25's TF saturation is applied — so the effect is real but bounded; keyword-stuffing an h1 cannot produce unbounded scores.

In the implementation, defined in parser.rs line 365–371:

• Each word is stored with its weight as a u8, multiplied by 24 internally (e.g. h1 → 7 * 24 = 168, plain text → 1 * 24 = 24).
• For ranking: at search time, weighted_term_frequency is divided back by 24.0 and fed into the BM25 TF calculation (search.rs line 696). The actual effective contribution is also mediated by length_bonus from fuzzy-match similarity.
• For display and excerpt selection: a separate per-word balanced_score is computed as (weight as f32).powi(2) (line 188) and serialized to the JS client (lib.rs line 534). This squared value determines which passage Pagefind surfaces as the excerpt preview — consistent with the docs' "quadratic scale" description — but does not affect how pages are ordered in search results.

In practice, an explicit data-pagefind-weight attribute always wins over the auto-assigned heading weight — data-pagefind-weight="1" on an <h1> cancels the default 7.

Title weight (meta weight)

Meta weights scale an additive boost computed outside BM25. They don't touch term frequency at all.

In the implementation, defined in lib.rs line 103:

• At search time, metadata fields are scored in a separate pass from body content (search.rs line 740–745):

  field_weight * matched_idf * coverage * coverage
  

  Where coverage = matched_idf / query_total_idf — the fraction of the query's total IDF weight matched in that metadata field.
• This score is added on top of the page's BM25 body score.
• Coverage is squared, so partial matches are penalized heavily. For three equally rare terms (each IDF i) where only one matches the title: matched_idf = i, coverage = 1/3, contribution = field_weight * i/9. A full match yields field_weight * 3i. The ratio is 1/27 ≈ 3.7% — under 4% of what a full match contributes.

Why they can't be compared directly

Setting data-pagefind-weight="10" on a div and setting metaWeights: { title: 10 } don't produce the same kind of effect, even though they use the same number. One scales term frequency inside BM25 (pre-BM25, saturating); the other scales an additive boost on top of BM25 (post-BM25, additive). Doubling a content weight doesn't double the score; doubling a meta weight does double that field's contribution to the boost.

---

How Pagefind's BM25 compares to standard BM25 and its variants

Pagefind uses BM25 as its core ranking algorithm, but with several modifications. Comparing against reference implementations (bm25s, rank_bm25, Manticore Search, pg_textsearch) and the BM25 Wikipedia article, we can characterize Pagefind's approach precisely:

Pagefind uses the Lucene IDF variant with the standard (k1+1) TF numerator, BM25F-style pre-weighted term frequencies (single-b normalization), a custom BM25/linear TF interpolation knob, and an additive metadata boost layer outside BM25.

IDF — Lucene variant

The IDF calculation (pagefind_web/src/search.rs, line 130–135):

log(1 + (N - n + 0.5) / (n + 0.5))

The + 1 inside the log(...) is what makes this specifically the Lucene IDF formula (as labeled by bm25s, also used by pg_textsearch). It guarantees non-negative IDF values. This is distinct from:

Variant	IDF formula	Can go negative?
Lucene (Pagefind uses this)	log(1 + (N - df + 0.5) / (df + 0.5))	No
Robertson/Okapi	log((N - df + 0.5) / (df + 0.5))	Yes (df > N/2)
ATIRE	log(N / df)	No
BM25L	log((N + 1) / (df + 0.5))	No
BM25+	log((N + 1) / df)	No

The BM25L and BM25+ IDF formulas are from the original papers (Lv & Zhai, SIGIR 2011 and Lv & Zhai, CIKM 2011 respectively), not just implementation choices. Both explicitly modify the Robertson IDF to avoid negative values, but they do so differently — BM25L retains the +0.5 smoothing on df, while BM25+ drops it.

Core TF — standard BM25 with weighted input

The TF component (pagefind_web/src/search.rs, line 155–156):

(k1 + 1) * f / (k1 * (1 - b + b * |D| / avgdl) + f)

This includes the (k1 + 1) factor in the numerator, which is present in most academic formulations of BM25 (including Robertson's original papers, Wikipedia's formula, and Elasticsearch/Lucene). It's a constant multiplier that doesn't affect ranking order.

Note: some reference implementations drop (k1 + 1) as an optimization. In bm25s's taxonomy, the version with this factor is labeled "ATIRE", while Robertson and Lucene omit it. This is a taxonomy choice specific to bm25s (following Kamphuis et al. 2020), not a universal convention.

Defaults (lib.rs, line 104–108): k1 = 1.4 and b = 0.75. The public API exposes these under different names:

BM25 parameter	Pagefind API name (options.ranking)	Default	Range
k1	termSaturation	1.4	0–2
b	pageLength	0.75	0–1
(custom)	termFrequency	1.0	0–1
(custom)	termSimilarity	1.0	—

The key difference from any standard variant: f is not a raw term count. It's weighted_term_frequency * length_bonus, where:

• weighted_term_frequency incorporates content weights (heading level, data-pagefind-weight) divided back by 24.
• length_bonus (controlled by the termSimilarity ranking parameter) is a term-similarity penalty that reduces score for fuzzy/partial matches.

BM11 / BM15 — supported implicitly

BM11 (b = 1) and BM15 (b = 0) are just special cases of BM25. Since Pagefind lets users set pageLength (which maps to b) anywhere in [0, 1], both are available.

BM25F — closest named relative, but not a strict implementation

BM25F applies per-field weights to term frequency before feeding into a single BM25 calculation, rather than running BM25 separately per field and combining. Pagefind's content weight system is conceptually similar — heading/section weights are baked into the term frequency that goes into one BM25 calculation.

But it's not a strict BM25F implementation. Manticore Search shows what strict BM25F looks like: each field has its own length normalization against its own average field length (b per stream, avgdl per stream). Pagefind uses a single b for the whole document regardless of which weighted section a word came from.

BM25+ / BM25L — not implemented

BM25+ (Lv & Zhai, CIKM 2011) adds a + δ constant inside the summation to prevent long documents that match from being unfairly scored below short documents that don't match at all. BM25L (Lv & Zhai, SIGIR 2011) addresses the same problem differently, by adding δ into both numerator and denominator of a pre-normalized TF. Pagefind includes neither correction. (See also bm25s implementations for comparison.)

Pagefind-specific: term_frequency interpolation

The term_frequency ranking parameter (pagefind_web/src/search.rs, line 158–163) is entirely custom. It interpolates between BM25's saturating TF curve and a simpler linear TF:

pagefind_tf = (1 - term_frequency) * raw_tf + term_frequency * bm25_tf

Where raw_tf (line 161–162) is:

raw_tf = min(weighted_with_length / (average_page_length / 5), k1 + 1)

Here weighted_with_length is weighted_term_frequency * length_bonus — the same weighted input f that feeds into BM25 TF (see line 148). The /5 divisor is a hardcoded scaling factor; there are no source comments explaining its rationale. The result is clamped at k1 + 1.

At the default term_frequency = 1.0, this is pure BM25. At 0.0, it uses only raw_tf — a linear function of weighted term count that grows until it hits the k1 + 1 ceiling. This doesn't correspond to any named variant in the literature or reference implementations surveyed.

---

This analysis is based on Pagefind at commit 50b519e. Links to other repos are also pinned to what was the most recent commit during analysis.