Repository navigation
Releases
Lasagna 0.3.0
Chat with a corpus from any OpenAI client, a knowledge graph every process
shares, and benchmarks that say when a stage fell back. Thank you to
@segevshlomovIBM, for the stage isolation
check's move to judging each setting an arm names, on each track that ran
(#25), and for the evidence runs in the Compose stack that found hybrid search
failing over with no trace in the bundle (#50) and, under speed, running
dense-only (#52), both fixed here.
Upgrading:
- New runs get new stage digests, and a run that writes no answers no longer
records a judge, so their comparability keys differ from 0.2.x results; and
seven settings undersystem.stages.<stage>.settingsare recorded per
track. - A benchmark graph arm's corpus is ingested once more: its key now holds the
extraction model. - Documents uploaded with
enable_graph_retrievalbefore 0.3.0 have no stored
graph (0.2.x kept it in one process's memory), so graph expansion finds
nothing for them until they are reindexed:
POST /api/v1/documents/{document_id}/reindexfor one, or
POST /api/knowledge/reindexfor every document of a session. The reindex
reads the stored file, and needs an LLM to extract with. - The new graph tables are created on their own, on SQLite and PostgreSQL;
there is nothing to migrate.
Deprecated
- The old names renamed in 0.1.0 (
lasagna benchmark compare,lasagna benchmark leaderboard,results interaction --bootstrap, and the names
dropped fromlasagna.__all__) and the unversioned/api/config/knowledge
routes stay for one more release, with their warnings: the old names are
removed in 0.4.0, and the routes move under/api/v1then, once the
configurator calls them there. 0.2.0 said 0.3.0. lasagna.retrieval.graph.get_or_create_session_graph: it builds a graph no
other process sees, and retrieval no longer reads it. Usegraph_forand
index_document_graph.
Added
- A benchmark's run record says how each graph arm's corpus got its knowledge
graph:system.graph.stateisbuilt,reused, orrebuiltwhen a reused
corpus had none (ingested before the graph was stored in the metadata
database), whose documents are then extracted again from the chunks
already stored (#55), as are documents another extraction model did.
incomplete, with how many documents aremissing, when some could not be
extracted (no LLM, a failed call); every query of that arm counts as a
graphragfallback, so the bundle says so. A document that names no entity,
or whose replies held no readable JSON (unparsed_chunks), counts as
extracted. The benchmark ingest stores a
document's graph only once the document is in, so a corpus whose ingest
failed leaves none. Two graph arms that differ ingraph_extraction_model
orgraph_entity_typesget a corpus each, and an arm that names no model is
keyed by the default model. That key is new: a graph arm's corpus ingested
before it is ingested once more. POST /api/v1/chat/completionsandGET /api/v1/models: chat with a
session's documents from any OpenAI client library or chat front end (#13).
The route searches with the session's pipeline, as/search/citationsdoes,
under the same session header, auth, deadline and quota, and writes the
answer with thecitedgenerator from the configured LLM. It answers in the
OpenAI format, at once or streamed as server-sent events ending with
data: [DONE], with the passages ascitations([n]in the answer is
citations[n-1]) anddegraded_stages. The answer runs under the new
RAG_CHAT_TIMEOUT(120 s); the LLM is checked before the search, and an
answer that fails is not retried by the client (x-should-retry: false), so
nothing is searched for an answer that cannot be written. The endpoint reference, the REST
guide and the README show how; every stated endpoint count is now checked
against the API's OpenAPI schema, and every route against the reference.
Fixed
- A run that writes no answers names no judge and no answer-similarity
model, since only answers are judged. A live retrieval sweep recorded the
judge fromRAG_EVAL_JUDGE_MODELorjudge_model:, and the similarity
model fromRAG_EVAL_SIMILARITY_MODELor its default: with the judge set to
an alias (llama3.2) it failedlasagna results validate --strictwith
UNRESOLVED_JUDGE_ALIAS, and two identical retrieval runs split on a
leaderboard by which judge the environment named. A new run's key differs
from an older identical one's; an older results file keeps its key however
it is bundled, and bundled again it loses the warning. A benchmark whose
mode resolves togeneration_only(TriviaQA underauto) answers and is
judged whatever the tracks, so it keeps its judge, and now records the
default judge when none is named, which changes its key too. - The knowledge graph
enable_graph_retrievalbuilds is kept in the metadata
database every process shares, so a search finds the graph whichever process
built it (Part of #55). It lived in the memory of the process that ingested:
in the Compose stack the Celery worker built it and every API worker searched
an empty one, silently, and a restart lost it. It is now stored per session
and collection (SQLite, or PostgreSQL in the stack), each mention and edge
with its document, after the document's chunks are stored, and each process
loads it again once the session's documents change (an upload, a delete, a
benchmark's ingest). Storing a document again replaces its part, so a
reindex leaves no old chunk ids behind, and an upload that fails keeps none.
A search no longer creates an empty graph. Concurrent writers lock entities
in one order, so PostgreSQL does not deadlock them. In
lasagna.retrieval.graph,get_or_create_session_graphis deprecated (it
builds a graph no other process sees, and retrieval no longer reads it); use
graph_forandindex_document_graph.GraphRetriever.expandtakes
collection_name, defaulting to the session's. - Deleting a document removes its part of the session's knowledge graphs, on
every delete path, and deleting a session drops what is left; a reset
forgets the graphs a process had loaded, and a stale benchmark corpus drops
its graph with its vectors (#55). - Hybrid search asks about the collection it searches, and a connection pool
passes the question on (#50, #52). The Compose stack wraps its Milvus server
in a pool (RAG_CONNECTION_POOL_SIZE=10) that did not pass on
collection_supports_native_hybrid, so every collection counted as able to
run native hybrid search. The collections Lasagna creates have no
sparse_embeddingfield, so every hybrid search tried the native call,
failed and fell back (all 12 queries of astandardrun onsmoke). The
same answer skipped filling the BM25 index from the store, so wherever a
process had not indexed the chunks itself, the fallback found it empty and
returned dense results only. In the stack, the Celery worker ingests
uploads, so this was every REST and MCP hybrid search an API worker ran: in
a test of 22 searches on one upload, all 4 that ran (one per worker) fell
back to dense-only, and the search cache served the other 18 from them. In a
benchmark, it was a hybrid arm on a corpus ingested with hybrid off: underspeed
it scored exactly the control, ndcg@10 0.9464 onsmoke, where Milvus Lite
gives 0.9505. In the stack it now gives 0.9505, with map@10 and R-precision
as on Milvus Lite. The base adapter answers per collection from its
capabilities (pgvector, OpenSearch and Weaviate as before), Milvus reads the
collection's fields, and the pool asks one of its instances. The searcher
never answers yes without asking, and asks about the collection a session
searches, which the benchmark names before its first search. The legacy
Milvus searcher serves only the collection it was built for. On a collection
without the field,MilvusBackend.hybrid_searchreturns dense results and
reports a degradedhybridstage instead of failing. Numbers move only for
hybrid search on a Milvus server where it had run dense-only. - A stage that falls back while a benchmark's queries run is named in the
results and the bundle (Part of #50). Hybrid search, the graph expansion
and other stages catch their own failure, carry on with less and record it;
REST, MCP and the SDK reported those stages, and the benchmark harness
never collected them. In the Compose stack, every query of a hybrid arm
underspeedfailed the native call and then ran dense-only, scoring
exactly the control (#52), and its bundle carried no fallback. Each task now
recordssystem.degraded_stages: per track, the stage, on how many of its
queries, and up to three reasons with the queries each covered, so a
dense-only fallback is told apart from one that changed nothing. The bundle
raisesACTIVE_FALLBACKSwith one line per arm, track and stage
(hybrid fell back during the run: smoke, arm 'hybrid', retrieval track, 12 of 12 queries (12 queries: ...; 12 queries: BM25 index empty; dense results only)), whichlasagna results validateprints and--strict
refuses;lasagna benchmark run,benchmark experimentand
lasagna results viewshow the same lines. A task where nothing fell back
records nothing new, so its bundle is what it was. Reasons are redacted
like the rest of a bundle. Adegraded.collecting()block opened inside
another now hands its stages to the outer one. - A document turns
readyonly once a search finds it. On a Milvus server,
where searches read at the Bounded level, a search sent the moment a
document turned ready missed it: in the default Compose stack 27 of 30 such
searches came back without it, and each found it a second later. Before it
marks a document ready, an upload or a reindex now w...
Lasagna 0.2.1
A patch release: uploads to the default Docker Compose stack are indexed again.
Thank you to @tnalbant, for the CLI reference
that no longer changes with the installed Typer version, and to
@segevshlomovIBM, whose evidence runs on
Milvus Lite and in the Compose stack found seven harness bugs, filed as #46 to
#52 (still open).
Added
- A dev container (
.devcontainer/devcontainer.json): a GitHub Codespace or
the Dev Containers extension sets up Python 3.12, the full development
install (with CPU-only torch, under the release's constraints) and Docker on
first start, so the Compose stack a pull request is tested in runs inside
it.
Fixed
- Uploads to the default Docker Compose stack (Milvus server, Celery worker)
are indexed again. Before indexing, the worker clears what an earlier attempt
at the document left, and a Milvus server refuses a filtered delete on a
collection that is not loaded, so before any search had loaded it every upload
ended inerror.MilvusBackend.deleteanddelete_rawnow load the
collection first, as its reads already did. - An AUTOINDEX index is accepted by a Milvus server, whether it comes from an
IndexConfig(which carried its default HNSW params) or from the collection
manager withindex_paramsset. A server refuses any param but the metric
with AUTOINDEX; only Milvus Lite, which builds a FLAT index instead, took it. - The generated CLI reference labels an option's type the same way whichever
Typer version is installed, so a checkout installed under the release's
constraints file (Typer 0.26.8) no longer reports the page as out of date and
rewrites it. - The evaluation SDK's benchmark catalog lists
2wikimultihopqaand says 32
public benchmarks; it had kept 31 since 0.2.0. Every page that states how many
benchmarks, components or published loops there are is now checked against
the code, so a pull request that adds one updates each page that counts it.
Earlier changes: https://github.com/dolev31/lasagna/blob/v0.2.1/CHANGELOG.md
Lasagna 0.2.0
The first release with outside contributors. Thank you to
@ect2000, for the 2wikimultihopqa benchmark
with its distractor paragraphs and the agentic-pipeline checks in
lasagna component validate, and to @juliasadler,
for the fix to two services racing to create a collection, measured against
the stack's own Qdrant.
Deprecated
- The old names renamed in 0.1.0 (
lasagna benchmark compare,lasagna benchmark leaderboard,results interaction --bootstrap, and the names
dropped fromlasagna.__all__) and the unversioned/api/config/knowledge
routes stay for one more release, with their warnings, and are removed in
0.3.0. 0.1.0 said 0.2.0; their warnings now name 0.3.0.
Changed
- The PyPI page links the documentation site and the paper.
Added
- A pinned
2wikimultihopqabenchmark adapter with a validation corpus that
retains gold paragraphs and distractors, with version-specific IDs that deduplicate
formatting-only whitespace differences while keeping the first original text. - A "PR quality" check on every pull request (
scripts/check_pr_body.py). It
fails until the description fills in the template, a person names themselves
as having read the whole diff, every suppression in the added lines gives a
reason, every TODO links an issue, and a change tosrc/lasagna/comes with a
test and a line here. Theno-changelogandno-new-testlabels waive the
last two. CONTRIBUTING.md has the full bar under "Quality bar". - A pull request template, and
CODEOWNERS. - Issue forms for a benchmark request and a research proposal. The component
form now asks for the stage, the paper and the number it reports, and a
comparison with a built-in control. Blank issues are off; questions go to
Discussions. - A Wanted page: the
components, benchmarks, studies and deployment work the project would like to
have, each with what "done" means. - A pull request checklist in
docs/contributing-components.md. - An
IMAGE_EXTRASbuild argument for the Docker image (set in.env): extras
installed beyond the production set, empty by default.IMAGE_EXTRAS=eval
runslasagna benchmarkand sweeps inside the Compose stack. - Every pull request is tested in the production environment, the Compose
stack built from its branch (CONTRIBUTING.md, "Tested in the production
environment"). The "PR quality" check fails while "How it was tested" shows
nodocker composecommand, and the template has a box for it.
Fixed
- On a Milvus server, the benchmark's corpus census and pre-flight see the
rows written just before them. A server reads with the collection's Bounded
consistency, so a count right after ingestion came up short (0 of 24
documents), and every coldlasagna benchmark runin the Compose stack was
refused as "changed while this run used it". Backends gainread_fresh(),
a block whose filter reads see every earlier write (a no-op by default; on a
Milvus server it reads Strong, about 200 ms a query, so only these checks
use it and search keeps its speed).iter_allon a server first runs one
Strong query, since pymilvus's query iterator ignores the consistency level,
so a scan right after an insert could come up short.PooledVectorBackend,
which the Compose stack runs in front of a Milvus server, passes both on: it
entersread_freshon every pooled instance, anditer_alluses an
instance's own cursor instead of paging by offset, which read stale and
stopped at Milvus's 16,384-row limit. The pool also passes onis_lite, so
a pool over Milvus Lite (connection_pool_sizeabove 1) is no longer taken
for a server. - Two services starting together no longer fail when the other creates the
collection first:create_collectionraisesCollectionExistsError(a
ValueErrorsubclass) on an existing name, and collection initialization
continues as the "already exists" branch does instead of failing to start.
The same lost-race handling now covers the third check-then-create path,
CollectionManager.ensure_backend_collection, which the API process and a
Celery worker can reach at the same time. - Validate
BaseAgenticPipelinecomponents with an async contract check and
report incorrect results or input mutation instead of rejecting the stage.
Earlier changes: https://github.com/dolev31/lasagna/blob/v0.2.0/CHANGELOG.md
Lasagna 0.1.0
The first public release. Lasagna is a RAG framework built for comparing one
part of a pipeline with everything else held fixed: eight swappable stages and
55 built-in components (each one that implements a published method names its
paper), an evaluation harness that says whether a difference is real (paired
significance and interaction tests, power, a stage isolation check, checkable
result bundles), and the same pipeline served to agents over MCP, to programs
over REST and the Python SDK, and in a Docker Compose stack.
Its metric definitions are METRICS_VERSION 1.7
(metrics.md defines
each metric). The version is part of every comparability key, so results from
the internal 0.1.0a1 pre-release (1.1) or from development builds since (1.2 to
1.6) never share a leaderboard group with results from this release.
Users of 0.1.0a1: read Changed for users of 0.1.0a1
below. The fixes made since 0.1.0a1 are described in the git history.
Added
Components, by stage (lasagna component list [stage] lists them without
loading any):
- Generator, an eighth stage.
concise(the harness's own prompt),cited
(ALCE),chain_of_thought,chain_of_note, and the baselinesclosed_book
(the question alone) andlong_context(all ten retrieved passages). A
functionf(query, contexts) -> strbecomes one with@lasagna.generator.
An arm withno_rag: trueretrieves nothing and answers withclosed_book. - Retriever:
dense,bm25,splade,hybrid_rrfandhybrid_convex. - Reranker:
cross_encoder(any current Hugging Face reranker by name),
colbert,monot5,llm_listwise(RankGPT's sliding window),
llm_pointwise,flashrank,cohere,mmr,combinedandensemble. - Query enhancer:
hyde,query_expansion,query_decomposition
(least-to-most),step_back,query2doc,rewriteandmulti_query. - Agentic pipeline:
self_ask,ircot,flare,adaptiveand
self_rag, each with its paper's defaults as settings;loop_tracerecords
what each loop did. - Chunker:
semantic(with Kamradt's breakpoint methods),
structure_aware,hierarchical,fixed,recursive,sentence_window,
markdown_header,propositionandcontextual. - Embedder:
sentence_transformers,bge_m3,openai,cohereand
voyage, with each model family's query and document prefixes. - Document processor: built-in extraction, and
docling,markitdownand
pymupdf4llm(each an extra;pymupdf4llmis AGPL-3.0 and not inall). - Your own component in one function.
@lasagna.reranker(and a decorator
per stage) turns a function into a component a sweep names as
module:function;lasagna component validatechecks it keeps its stage's
contract.lasagna.from_langchainandlasagna.from_llamaindexregister
LangChain and LlamaIndex objects as stages (extra:llamaindex).
Experiments:
- Sweeps (
lasagna benchmark experiment sweep.yaml) over components and
settings, factorial designs, repeated runs and baseline arms; chunker and
embedder arms get corpora of their own. A key the harness does not read is
an error naming the likely intended one, and a value out of its setting's
range is an error naming the arm, before anything is ingested and in
--dry-runtoo. The ranges are the same for an arm, a session, a settings
file and the environment. - Your own data as a benchmark. A JSONL or CSV file, or a BEIR folder:
benchmarks: [{path: data/my.jsonl}]orlasagna benchmark run ./my.jsonl.
It registers aslocal/<stem>@<12 hex digits of its sha256>, so an edited
file is a different benchmark.lasagna benchmark validate-datanames each
bad row and field. See docs/own-data.md. lasagna benchmark synthesizewrites factoid, multi-hop and unanswerable
questions from your documents with the configured LLM, for development
rather than for reported results. The plan warns when the documents are too
few for the number of questions asked.- Benchmarks run beside
lasagna serve. While another process holds the
local Milvus Lite store,benchmark runandbenchmark experimentuse one
of their own in~/.lasagna/bench/dbsand say so
(configuration.md). --dry-runprints the arms, tasks, models and whether each is
downloaded, with token, time and (from a price table,prices:) dollar
estimates from a 3-sample probe.- A re-run replays its LLM responses. Requests at temperature 0 are
answered from~/.lasagna/cache/llm.sqliteinside an experiment; the hit
rate is inresults.jsonand the bundle. Off withllm_cache: falseor
--no-cache. - Metrics: NDCG, recall, precision, MRR, MAP, R-precision, complete
recall, hit rate and F1 atk; exact match, token F1, ROUGE-L, answer
containment and similarity, ALCE citation recall and precision, abstention
rate and SQuAD v2's answerable and unanswerable splits; the RAGAS judged
metrics withfactual_correctness; latency, tokens and LLM calls per query.
Results:
lasagna results significance(paired sign-flip test, Holm correction)
ends in a verdict naming each pair that survives and who won;
results powergives the minimum detectable effect;results interaction
tests a 2x2 design. A sweep ends with the command to run next on its
results: one line per benchmark, andresults interactionafter a factorial.lasagna results viewwrites a run as one offline HTML page:
leaderboard, significance, every metric, the isolation check and a per-query
diff with what each arm retrieved.- Every
lasagna resultscommand takes a run's folder (the one a run prints
after "Output:") or theresults.jsonin it, and reads--benchmarkas the
benchmark's id or thename:the sweep gave it, listing the run's
benchmarks when it matches none. - Bundles (
results bundle,results validate) carry what a reader needs
to decide whether two numbers may be compared, with one comparability key per
benchmark and configuration;results leaderboardgroups by it, as markdown
pages or LaTeX tables (--format latex), andresults compareprints it
for each side. Every results command refuses a bundle thatresults validaterejects;--allow-invalidshows one marked INVALID. - Stage isolation. Every run records a manifest of each stage's settings;
arms that differ outside the stages they name block the bundle, and an arm
whose named change never moved a ranking (on at least ten queries) is
flagged inert. - A reproduction: on the full SciFact and NFCorpus test sets the dense
models land within 0.002 NDCG@10 of MTEB, and BM25 with a cross-encoder
within 0.004 of the BEIR paper (examples/reproduction/). Its
results.csvrecords the commit, metrics version and date of the run that
produced the numbers, andresults_csv.pywrites the same file from a rerun
to compare.
Serving:
- MCP server (
lasagna-mcp, stdio by default, orlasagna serve --mcp)
for Claude Code, Claude Desktop and Cursor. The default tool set,safe,
serves no deletion tools (--tools alldoes).searchnames the stages that
fell back and why, with the fix, also when it finds nothing (for example,
when the session's documents are in another model's collection). An
argument of the wrong type, or one the tool does not take (sessionfor
session_id), is refused in one sentence naming the one it takes. - REST API (
lasagna serve, or the Compose stack) under/api/v1,
covered by the deprecation policy in GOVERNANCE.md; search
responses name the stages that fell back indegraded_stages(a reranker
not downloaded: retrieval order and the download, also when switching to it). lasagna uploadtakes files, globs and folders (a folder's readable
files, subfolders included, saying what it skipped).- Profiles
speed(92 MB, no LLM),standard,balancedand
max_quality, applied per session from any surface. A switch that moves a
session's documents to another model (lasagna configure --reindex,
reindex=trueover REST and MCP) checks the new models load, reindexes every
document first and switches only when all are reindexed. - Deployment: a Compose stack (Postgres, Redis, a Milvus server or Qdrant,
Celery, MCP) with per-service healthchecks, Kubernetes manifests, Prometheus
metrics, JSON logs with request ids, and readiness that pings each store. - Six vector stores behind one interface: Milvus (and Milvus Lite),
pgvector, Qdrant, Weaviate, Pinecone and OpenSearch.
For contributors: the test suite keeps the Hugging Face hub offline unless
HF_HUB_OFFLINE=0, and tests that load real weights skip when their model is
not cached.
Documentation and examples: the guide,
configuration (which setting wins, profiles, where
data lives), the generated settings reference,
model licenses, a Colab notebook comparing a
one-function reranker with the built-in cross-encoder, the plugin template,
and an offline acceptance demo (examples/acceptance/offline_demo.py).
Changed for users of 0.1.0a1
Metric definitions (METRICS_VERSION 1.1 to 1.7)
Each step changes what a number means, so each is part of the comparability
key; `results ...