Repository navigation
Chat with a corpus from any OpenAI client, a knowledge graph every process
shares, and benchmarks that say when a stage fell back. Thank you to
@segevshlomovIBM, for the stage isolation
check's move to judging each setting an arm names, on each track that ran
(#25), and for the evidence runs in the Compose stack that found hybrid search
failing over with no trace in the bundle (#50) and, under speed, running
dense-only (#52), both fixed here.
Upgrading:
- New runs get new stage digests, and a run that writes no answers no longer
records a judge, so their comparability keys differ from 0.2.x results; and
seven settings undersystem.stages.<stage>.settingsare recorded per
track. - A benchmark graph arm's corpus is ingested once more: its key now holds the
extraction model. - Documents uploaded with
enable_graph_retrievalbefore 0.3.0 have no stored
graph (0.2.x kept it in one process's memory), so graph expansion finds
nothing for them until they are reindexed:
POST /api/v1/documents/{document_id}/reindexfor one, or
POST /api/knowledge/reindexfor every document of a session. The reindex
reads the stored file, and needs an LLM to extract with. - The new graph tables are created on their own, on SQLite and PostgreSQL;
there is nothing to migrate.
Deprecated
- The old names renamed in 0.1.0 (
lasagna benchmark compare,lasagna benchmark leaderboard,results interaction --bootstrap, and the names
dropped fromlasagna.__all__) and the unversioned/api/config/knowledge
routes stay for one more release, with their warnings: the old names are
removed in 0.4.0, and the routes move under/api/v1then, once the
configurator calls them there. 0.2.0 said 0.3.0. lasagna.retrieval.graph.get_or_create_session_graph: it builds a graph no
other process sees, and retrieval no longer reads it. Usegraph_forand
index_document_graph.
Added
- A benchmark's run record says how each graph arm's corpus got its knowledge
graph:system.graph.stateisbuilt,reused, orrebuiltwhen a reused
corpus had none (ingested before the graph was stored in the metadata
database), whose documents are then extracted again from the chunks
already stored (#55), as are documents another extraction model did.
incomplete, with how many documents aremissing, when some could not be
extracted (no LLM, a failed call); every query of that arm counts as a
graphragfallback, so the bundle says so. A document that names no entity,
or whose replies held no readable JSON (unparsed_chunks), counts as
extracted. The benchmark ingest stores a
document's graph only once the document is in, so a corpus whose ingest
failed leaves none. Two graph arms that differ ingraph_extraction_model
orgraph_entity_typesget a corpus each, and an arm that names no model is
keyed by the default model. That key is new: a graph arm's corpus ingested
before it is ingested once more. POST /api/v1/chat/completionsandGET /api/v1/models: chat with a
session's documents from any OpenAI client library or chat front end (#13).
The route searches with the session's pipeline, as/search/citationsdoes,
under the same session header, auth, deadline and quota, and writes the
answer with thecitedgenerator from the configured LLM. It answers in the
OpenAI format, at once or streamed as server-sent events ending with
data: [DONE], with the passages ascitations([n]in the answer is
citations[n-1]) anddegraded_stages. The answer runs under the new
RAG_CHAT_TIMEOUT(120 s); the LLM is checked before the search, and an
answer that fails is not retried by the client (x-should-retry: false), so
nothing is searched for an answer that cannot be written. The endpoint reference, the REST
guide and the README show how; every stated endpoint count is now checked
against the API's OpenAPI schema, and every route against the reference.
Fixed
- A run that writes no answers names no judge and no answer-similarity
model, since only answers are judged. A live retrieval sweep recorded the
judge fromRAG_EVAL_JUDGE_MODELorjudge_model:, and the similarity
model fromRAG_EVAL_SIMILARITY_MODELor its default: with the judge set to
an alias (llama3.2) it failedlasagna results validate --strictwith
UNRESOLVED_JUDGE_ALIAS, and two identical retrieval runs split on a
leaderboard by which judge the environment named. A new run's key differs
from an older identical one's; an older results file keeps its key however
it is bundled, and bundled again it loses the warning. A benchmark whose
mode resolves togeneration_only(TriviaQA underauto) answers and is
judged whatever the tracks, so it keeps its judge, and now records the
default judge when none is named, which changes its key too. - The knowledge graph
enable_graph_retrievalbuilds is kept in the metadata
database every process shares, so a search finds the graph whichever process
built it (Part of #55). It lived in the memory of the process that ingested:
in the Compose stack the Celery worker built it and every API worker searched
an empty one, silently, and a restart lost it. It is now stored per session
and collection (SQLite, or PostgreSQL in the stack), each mention and edge
with its document, after the document's chunks are stored, and each process
loads it again once the session's documents change (an upload, a delete, a
benchmark's ingest). Storing a document again replaces its part, so a
reindex leaves no old chunk ids behind, and an upload that fails keeps none.
A search no longer creates an empty graph. Concurrent writers lock entities
in one order, so PostgreSQL does not deadlock them. In
lasagna.retrieval.graph,get_or_create_session_graphis deprecated (it
builds a graph no other process sees, and retrieval no longer reads it); use
graph_forandindex_document_graph.GraphRetriever.expandtakes
collection_name, defaulting to the session's. - Deleting a document removes its part of the session's knowledge graphs, on
every delete path, and deleting a session drops what is left; a reset
forgets the graphs a process had loaded, and a stale benchmark corpus drops
its graph with its vectors (#55). - Hybrid search asks about the collection it searches, and a connection pool
passes the question on (#50, #52). The Compose stack wraps its Milvus server
in a pool (RAG_CONNECTION_POOL_SIZE=10) that did not pass on
collection_supports_native_hybrid, so every collection counted as able to
run native hybrid search. The collections Lasagna creates have no
sparse_embeddingfield, so every hybrid search tried the native call,
failed and fell back (all 12 queries of astandardrun onsmoke). The
same answer skipped filling the BM25 index from the store, so wherever a
process had not indexed the chunks itself, the fallback found it empty and
returned dense results only. In the stack, the Celery worker ingests
uploads, so this was every REST and MCP hybrid search an API worker ran: in
a test of 22 searches on one upload, all 4 that ran (one per worker) fell
back to dense-only, and the search cache served the other 18 from them. In a
benchmark, it was a hybrid arm on a corpus ingested with hybrid off: underspeed
it scored exactly the control, ndcg@10 0.9464 onsmoke, where Milvus Lite
gives 0.9505. In the stack it now gives 0.9505, with map@10 and R-precision
as on Milvus Lite. The base adapter answers per collection from its
capabilities (pgvector, OpenSearch and Weaviate as before), Milvus reads the
collection's fields, and the pool asks one of its instances. The searcher
never answers yes without asking, and asks about the collection a session
searches, which the benchmark names before its first search. The legacy
Milvus searcher serves only the collection it was built for. On a collection
without the field,MilvusBackend.hybrid_searchreturns dense results and
reports a degradedhybridstage instead of failing. Numbers move only for
hybrid search on a Milvus server where it had run dense-only. - A stage that falls back while a benchmark's queries run is named in the
results and the bundle (Part of #50). Hybrid search, the graph expansion
and other stages catch their own failure, carry on with less and record it;
REST, MCP and the SDK reported those stages, and the benchmark harness
never collected them. In the Compose stack, every query of a hybrid arm
underspeedfailed the native call and then ran dense-only, scoring
exactly the control (#52), and its bundle carried no fallback. Each task now
recordssystem.degraded_stages: per track, the stage, on how many of its
queries, and up to three reasons with the queries each covered, so a
dense-only fallback is told apart from one that changed nothing. The bundle
raisesACTIVE_FALLBACKSwith one line per arm, track and stage
(hybrid fell back during the run: smoke, arm 'hybrid', retrieval track, 12 of 12 queries (12 queries: ...; 12 queries: BM25 index empty; dense results only)), whichlasagna results validateprints and--strict
refuses;lasagna benchmark run,benchmark experimentand
lasagna results viewshow the same lines. A task where nothing fell back
records nothing new, so its bundle is what it was. Reasons are redacted
like the rest of a bundle. Adegraded.collecting()block opened inside
another now hands its stages to the outer one. - A document turns
readyonly once a search finds it. On a Milvus server,
where searches read at the Bounded level, a search sent the moment a
document turned ready missed it: in the default Compose stack 27 of 30 such
searches came back without it, and each found it a second later. Before it
marks a document ready, an upload or a reindex now waits until the server
has applied its chunks (one Strong count per document, about 200 ms; a
benchmark's ingest waits once per corpus);
searches keep the fast default. Milvus Lite, Qdrant, OpenSearch, pgvector
and Weaviate searches already saw a write when it returned, so nothing
changes there. Pinecone is eventually consistent and is not covered: there a
search sent right after ready can still miss the document. - The stage isolation check excuses only the settings an arm declares, and
what the harness sets on their account, rather than every setting in a
stage an arm names, and judges an inert arm setting by setting. An arm that
namedhybrid_alphahid adefault_top_kthat leaked into the retriever
stage; a reranker sweep, where every arm names a reranker, hid a leaked
rerank_top_k; an arm that namedhybrid_alphaandrrf_kpassed as live
when onlyrrf_karrived. The first two now mark the bundle
UNDECLARED_STAGE_CHANGE, naming the settings and the track each leaked on
that acts on it (default_top_k on the generation track); the third is
flagged inert for the setting that never arrived. The run's isolation line
andresults viewsay the arms differ only in the settings they name,
which is what is checked. - The stage manifest records the routing and retrieval-depth settings
(use_agentic_rag,agentic_pipeline,routing_default_tier,
similarity_threshold,chunk_cite_retrieval,default_top_k,
enable_context_reordering) per track, as{"retrieval": ..., "generation": ...}, the waycontext_orderalready was, and
system.tracksrecords which tracks ran. It digested the values the
retrieval track forces on every arm, so on a generation track an arm that
setuse_agentic_rag: trueran the loop, digested like its control, and
was called inert; two arms that differed only inrouting_default_tier
could not be told apart. The check now judges such an arm per track: inert
on retrieval, where the setting is forced, and live on generation. For
anyone parsingresults.json: each of these settings, under
system.stages.<stage>.settings, is now that dict where it was a single
value. Stage digests, and so comparability keys, change for every run from
this change on: a bundle written beforesystem.trackswas recorded keeps
its verdict underresults validate, which reads the integrity block the
bundle records, and groups on a leaderboard only with bundles written
before it;results bundleon a results.json written before it runs the
new check, so it can refuse what passed then. - The isolation check calls a setting only the agentic loop acts on
(max_agent_iterations,enable_crag,self_ask_*,ircot_*,flare_*
and the rest) inert on the retrieval track, which runs no loop, and so
speculative RAG's (enable_speculative_rag,speculative_*), which only the
answer step reads, since that track makes no answers; it called such an arm
live there. An arm inert on one track only marks the bundleINERT_ARM, a
warning, soresults validate --strictfails a full-run sweep of such a
setting. What the harness sets on one track only (the depthmetric_kasks
retrieval for) is excused on that track only, so adefault_top_kleaked
into ametric_karm's generation config is caught.
A pair of arms that shares no track is not compared, and neither is a row
written beforesystem.trackswas recorded paired with a newer one (a
sweep resumed across the upgrade); each is listed as unverified, and with
no pair compared the run says why.
Earlier changes: https://github.com/dolev31/lasagna/blob/v0.3.0/CHANGELOG.md