Search

Search

Three engines. The spelling picks the engine, so a build flag never changes what a query means.

Form	Matches	Order	Needs	Index
"term" 'term'	folded substring	positional	nothing	none
?"term" ?'term'	words, stemmed, BM25	ranked	fulltext feature	22 MB
*"term" *'term'	vector similarity	ranked	vector feature	28 MB

If a feature or its index is missing, the query is refused with QQL_UNSUPPORTED naming the fix. It never falls back to a different engine — answering a different question than the one asked is worse than saying no.

Scoping

The reference in front of the term says where to look. This is the same for all three engines:

"text"            the default source — the Quran
q:"text"          the whole Quran
q:1:"text"        Surah 1
q:1:3~5:"text"    ayat 3–5 of Surah 1
b:1:"text"        Bukhari chapter 1
b:"text"          all of Bukhari

~ marks a search scope so it cannot be read as the - of an ordinary selector: Q:1:3-5 returns three ayat, Q:1:3~5:"x" searches them.

Arabic folding

The corpus is fully diacritized. A typed الحمد shares no substring with the stored ٱلْحَمْدُ, so all three engines fold before comparing:

• harakat and sukun (U+064B–U+0652), superscript alef (U+0670), the Quranic annotation marks (U+06D6–U+06ED), tatweel (U+0640) — dropped
• alef seats أ إ آ ٱ → ا
• ى → ي, ة → ه
• ASCII lowercased

Folding is for comparison only. Returned records keep every mark.

Exact — "term"

Always available, no index, no feature. A folded substring scan over ar and en, returned in positional order.

q:1:"الحمد"       1 hit
q:2:"prayer"      14 hits
q:1:'Allah'       2 hits    either quote works
b:1:"Allah's"     7 hits    the other quote carries an apostrophe

Plain substring, so no word or stem matching:

q:1:"mercy"       0 hits — "Merciful" does not contain "mercy"
q:1:"mercif"      finds it

Covers ar and en only, not metadata — "Al-Fatihah" finds nothing.

An unmatched search is an empty results, not an error.

Full text — ?"term"

Backed by tantivy: an inverted index with an English stemmer and BM25 ranking.

cargo run --features fulltext --bin qql-index      # build the indexes
cargo run --features fulltext -- 'q:1:?"mercy"'

q:1:"mercy"       0 hits
q:1:?"mercy"      2 hits    stemming reaches "Merciful"

The term carries tantivy's own query syntax:

q:?"prayer AND charity"~3
q:?"mercy OR forgiveness"~3
q:?"prayer -charity"~3
q:?'"straight path"'~3         a phrase — note the outer single quotes

Arabic is indexed folded; English under the en_stem tokenizer. The indexes are committed, so a checkout searches without a build step. Rebuilding all 18 sources takes about eleven seconds.

Similarity — *"term"

Ranked by vector similarity rather than by words.

cargo run --features vector -- 'q:*"worship"~3'

q:*"worship"~3        109:2, 109:3, 109:4    Surah al-Kafirun
q:1:*"الحمد"          1:2
b:*"intentions"~3     9:29, 81:88, 90:3

The indexes are committed (sources/vectors/*.qv, one file per source), so this needs no build step either. Rebuild after changing text:

python3 scripts/build-vectors.py            # all 18 sources, ~45 s
python3 scripts/build-vectors.py --source Q

What it actually does

The shipped embedder is a signed hash projection of folded tokens — whole words plus character trigrams, hashed onto 256 dimensions, normalized and quantized to int8. No model, no weights, and embedding a query is a handful of hashes rather than a transformer. That is what makes it viable on a low-end device.

It is fuzzy lexical matching, not semantic. It tolerates diacritics, prefixes and suffixes — worth a great deal for Arabic — but it does not know that charity and zakat are related, and a nonsense query returns weak noise rather than nothing.

Real semantic vectors are a build-time swap: emit an index with a different embedder id and teach src/vector.rs to embed queries the same way. The file format, the scan, the scoping and the result shape are unchanged by that.

There is deliberately no ANN index. At ~40,000 records a flat int8 scan is a few million integer operations; a graph index would add a large dependency and a second artifact that can drift from the text, to beat something already fast enough.

Ranked results

The two ranked engines are the only place in QQL where output is ordered by relevance rather than position. Their hits carry two extra fields:

{ "surah": 1, "ayah": 3, "score": 12.017, "ranked": true, "ar": "…", "en": "…" }

~N caps the count; the default is 20.

q:?"mercy"~5
q:*"worship"~3

Weak matches are dropped, so a ranked search can return fewer than the cap, or nothing. Scores are not comparable between engines — BM25 and cosine are different scales.

Which to use

• "term" when you know the exact wording, want every occurrence, and want them in order. Always works.
• ?"term" for ordinary keyword search: word forms, boolean and phrase queries, best-first. The one most users want.
• *"term" when the wording is uncertain and you want approximate matches — currently fuzzy-lexical, upgradeable to semantic.