Skip to content

v0.7.2 — Refresh no longer strands nodes

Choose a tag to compare

@jpmanson jpmanson released this 06 Aug 14:26
· 29 commits to main since this release
4438d01

Bug-fix release. If you use refresh_semantic_graph() with max_edges, upgrade — the incremental refresh could exclude nodes from the semantic graph permanently.

refresh could exclude nodes permanently

approximate treated a node as already linked if it appeared at either end of a generated edge. But a node that merely received an edge was never searched — it has no neighbours of its own — and a build cut short by max_edges leaves exactly those behind. Every later refresh skipped them:

truncated build:   3 edges, 2 nodes processed
                   sources: [d0, d1]    receivers only: [d2]
later refresh:     3 skipped            → d2 neighbourless, permanently

This is the same "excluded forever" failure v0.7.1 fixed for hand-made edges, reached by a different route — and max_edges is exactly the parameter you use on the large corpora where incremental refresh matters most.

A node now counts as processed only if it is the source of a generated edge: the node whose neighbourhood was actually searched.

Refresh converges instead of accumulating

Being strict re-searches nodes whose edges were all deduplicated away, so builds now deduplicate against edges already stored. Two consequences:

  • refresh_semantic_graph() is idempotent. Running it repeatedly is a no-op instead of adding copies.
  • replace=False no longer duplicates. It previously stacked a full second copy of the graph; it now adds only the edges a wider k discovers.
db.create_semantic_graph(k=1, min_score=0.0)
db.create_semantic_graph(k=3, min_score=0.0, replace=False)  # adds, does not duplicate

Behaviour change, and the old behaviour was a bug — appending an identical copy of every edge is not something a caller wants.

min_score rejects booleans

bool subclasses int, so min_score=True silently became a threshold of 1.0 and built nothing:

db.create_semantic_graph(k=2, min_score=True)   # was: 0 edges. now: DatabaseError

SIMILAR() already refused non-numbers; the two APIs agree again.

A failed ingest no longer leaves an orphan FTS index

index_documents(configure_fts=True) registered the index for documents that then never loaded. It is now removed if the ingest raises — unless the index already existed, which is not this call's to undo.

Docs correction

v0.7.1 claimed "FTS5 indexes existing rows when created, so order does not matter". That is wrong. create_text_index registers the configuration and indexes rows as they are written, so registering it after a load leaves those documents unsearchable until rebuild_text_index():

db.create_text_index("node", "Paper", ["text"])   # before loading
db.index_documents(rows, label="Paper")

# or, if already loaded
db.create_text_index("node", "Paper", ["text"])
db.rebuild_text_index()

index_documents is also now documented as not atomic: rows are written as they are read, because the alternative is holding a write transaction open across every embedding call. To resume after a failure, re-run — upsert=True makes replaying the whole batch safe.

Full changelog: v0.7.1...v0.7.2