Skip to content

v0.8.0 — Hybrid retrieval, path context and graph analysis

Choose a tag to compare

@jpmanson jpmanson released this 07 Aug 00:16
· 23 commits to main since this release
17f1a2d

Six features built on top of the 0.7 line, plus the test harness that found two of the bugs fixed along the way.

Hybrid retrieval

RRF fusion lived only inside DocumentIngestor. It is now available on any graph:

db.hybrid_search("ENOSPC retry policy", k=10)
db.hybrid_subgraph("ENOSPC retry policy", k=20, expand=1)

The two modes fail differently, which is the reason to fuse them — vector search misses exact tokens it never learned (identifiers, error codes), lexical search misses paraphrase. RRF merges by rank, so cosine similarity and BM25 never have to be made comparable. Degrades rather than fails: with only one index configured, that side is returned alone.

Path context

When the answer is what connects two concepts, not what is nearest either of them:

sub = db.path_context(["Roman Empire", "Battle of Hastings"], max_hops=3,
                      exclude_rel_types=["SEMANTIC_SIMILAR"])
# Roman Empire → Saxons → Vikings → Battle of Hastings

hops counts position along the route, not distance from a hit — an intermediate document is the answer here. An empty result means the concepts are not connected within max_hops, which is a real answer and one a top-k search would have hidden behind plausible-looking hits.

Patterns as boolean expressions

Previously a syntax error in every form, despite the docs showing it:

MATCH (p:Paper) WHERE NOT (p)-[:CITES]->() RETURN p.title
MATCH (p:Paper) RETURN p.title, (p)-[:CITES]->(:Paper) AS cites_a_paper

Anything MATCH accepts works inside: types, labels, direction, inline properties, multi-hop and variable-length paths. Matching stops at the first hit, so asking whether a node has a neighbour costs the same whether it has one or a thousand.

Mutual k-NN

create_semantic_graph(symmetrize="union" | "mutual" | "directed"). Reciprocity removes the edges that damage clustering most — a node at the edge of a cluster gets pulled into distant nodes' neighbour lists without them appearing in its own.

It is not a free improvement, and the docs carry the measurements. On overlapping clusters, mutual at k=3 cut cross-cluster edges from 40% to 14% and shattered three planted clusters into eight fragments; at k=5 the fragmentation was gone and so was most of the advantage. It trades coverage for precision and needs a larger k than union.

Community labelling

for community in db.communities("louvain", label_terms=3):
    print(f"[{community.label}] — {community.size} documents")
[graph, nodes, databases] — 3 documents
[search, embeddings, nearest] — 3 documents
[pasta, carbonara, cooking] — 3 documents

tfidf by default, which discounts words common to every community without a stopword list. This is the cheap end of topic modelling: clusters come from the graph, and the words only describe what landed in each — labelling never improves a partition, it reports one.

An invariant harness

Every bug fixed in 0.7.1 and 0.7.2 was a violated invariant found by reading code, not by a failing test. tests/invariants/ now states those properties and checks them across a matrix of parameters on generated corpora with planted clusters.

It found two defects on its first run:

  • max_edges cut builds mid-node. A node stopped half way through its neighbours still counted as processed, so no later refresh would finish it, and repeated capped refreshes converged to a graph missing ~11% of its edges.
  • The refresh loop does not terminate on nodes_processed — which is what the docs suggested until I ran it.

The harness is validated by mutation: reverting each 0.7.x fix fails 4, 12, 12 and 2 properties respectively.

Behaviour change

max_edges is no longer a hard ceiling. It is enforced between nodes and can overshoot by up to k-1. Cutting a node off mid-search left it looking processed with an incomplete neighbourhood that no later refresh would finish; every node a capped build processes is now complete, so repeated capped refreshes reach the whole graph.

Everything else is additive. undirected= still works as shorthand for symmetrize, community labelling is off by default, and no existing signature changed.

Still missing

Retrieval quality remains unmeasured. The invariant harness covers structural properties — atomic rebuilds, converging refreshes, no stranded nodes — but expand depth, reranker choice, min_score, RRF weights and symmetrize all change quality in ways nothing here measures. Build a small labelled set for your corpus before tuning.

Full changelog: v0.7.2...v0.8.0