Skip to content

v0.8.2 — Natural-language queries in hybrid retrieval

Choose a tag to compare

@jpmanson jpmanson released this 07 Aug 10:27
· 17 commits to main since this release
844e137

A user-typed question was a syntax error rather than a search. SQLite FTS5 parses its MATCH argument as a query language, so the punctuation people actually type went to the parser instead of the index:

db.hybrid_search("what does ENOSPC mean?")   # sqlite3.OperationalError: fts5: syntax error near "?"
db.text_subgraph("graph databases?")         # sqlite3.OperationalError
db.text_search("cost-benefit analysis")      # sqlite3.OperationalError: no such column: benefit

hybrid_search caught DatabaseError, but text_search leaked the raw sqlite3.OperationalError, so the exception escaped the fusion logic entirely.

Wrapping the error was only half of it

Catching the error still discarded the whole lexical half, quietly turning hybrid retrieval into vector-only search — precisely for the identifiers, surnames and error codes that are the reason to fuse two retrievers in the first place. A question mark should not cost you the BM25 side.

Unparseable queries are now retried with each token quoted as an FTS5 string literal (graph databases? → "graph" "databases"), which is the same AND-of-terms FTS5 applies to a bare word list, minus the parse failure.

db.hybrid_search("what does ENOSPC mean?")   # lexical half preserved
db.hybrid_search("AT&T")                     # works
db.hybrid_search("graph OR nothing")         # valid FTS5 operators still pass through untouched

Which methods tolerate what

Method Punctuation Why
text_search raises Its documented contract is FTS5 syntax (AND/OR/NEAR/10/term*). Silently rewriting those operators would break queries that rely on them.
text_subgraph tolerates Retrieval, not query syntax. Propagates if the retry also fails — with no second half to fall back on, an empty subgraph would report "no matches" for a broken search.
hybrid_search / hybrid_subgraph tolerates Falls back to the vector side if the retry also fails.

SQLiteFTSIndex.search got the same treatment, so custom text-index backends stop leaking raw sqlite3 errors through text_search_custom.

Compatibility

Patch release, no new API. The one behaviour change to code that already worked: text_search now raises DatabaseError instead of sqlite3.OperationalError — the type the rest of the library already raised. Catch DatabaseError if you were catching the sqlite3 error directly.

Full Changelog: v0.8.1...v0.8.2