Skip to content

Releases

Lasagna 0.3.0

Choose a tag to compare

@dolev31 dolev31 released this 07 Nov 20:48
1295bfc

Chat with a corpus from any OpenAI client, a knowledge graph every process
shares, and benchmarks that say when a stage fell back. Thank you to
@segevshlomovIBM, for the stage isolation
check's move to judging each setting an arm names, on each track that ran
(#25), and for the evidence runs in the Compose stack that found hybrid search
failing over with no trace in the bundle (#50) and, under speed, running
dense-only (#52), both fixed here.

Upgrading:

  • New runs get new stage digests, and a run that writes no answers no longer
    records a judge, so their comparability keys differ from 0.2.x results; and
    seven settings under system.stages.<stage>.settings are recorded per
    track.
  • A benchmark graph arm's corpus is ingested once more: its key now holds the
    extraction model.
  • Documents uploaded with enable_graph_retrieval before 0.3.0 have no stored
    graph (0.2.x kept it in one process's memory), so graph expansion finds
    nothing for them until they are reindexed:
    POST /api/v1/documents/{document_id}/reindex for one, or
    POST /api/knowledge/reindex for every document of a session. The reindex
    reads the stored file, and needs an LLM to extract with.
  • The new graph tables are created on their own, on SQLite and PostgreSQL;
    there is nothing to migrate.

Deprecated

  • The old names renamed in 0.1.0 (lasagna benchmark compare, lasagna benchmark leaderboard, results interaction --bootstrap, and the names
    dropped from lasagna.__all__) and the unversioned /api/config/knowledge
    routes stay for one more release, with their warnings: the old names are
    removed in 0.4.0, and the routes move under /api/v1 then, once the
    configurator calls them there. 0.2.0 said 0.3.0.
  • lasagna.retrieval.graph.get_or_create_session_graph: it builds a graph no
    other process sees, and retrieval no longer reads it. Use graph_for and
    index_document_graph.

Added

  • A benchmark's run record says how each graph arm's corpus got its knowledge
    graph: system.graph.state is built, reused, or rebuilt when a reused
    corpus had none (ingested before the graph was stored in the metadata
    database), whose documents are then extracted again from the chunks
    already stored (#55), as are documents another extraction model did.
    incomplete, with how many documents are missing, when some could not be
    extracted (no LLM, a failed call); every query of that arm counts as a
    graphrag fallback, so the bundle says so. A document that names no entity,
    or whose replies held no readable JSON (unparsed_chunks), counts as
    extracted. The benchmark ingest stores a
    document's graph only once the document is in, so a corpus whose ingest
    failed leaves none. Two graph arms that differ in graph_extraction_model
    or graph_entity_types get a corpus each, and an arm that names no model is
    keyed by the default model. That key is new: a graph arm's corpus ingested
    before it is ingested once more.
  • POST /api/v1/chat/completions and GET /api/v1/models: chat with a
    session's documents from any OpenAI client library or chat front end (#13).
    The route searches with the session's pipeline, as /search/citations does,
    under the same session header, auth, deadline and quota, and writes the
    answer with the cited generator from the configured LLM. It answers in the
    OpenAI format, at once or streamed as server-sent events ending with
    data: [DONE], with the passages as citations ([n] in the answer is
    citations[n-1]) and degraded_stages. The answer runs under the new
    RAG_CHAT_TIMEOUT (120 s); the LLM is checked before the search, and an
    answer that fails is not retried by the client (x-should-retry: false), so
    nothing is searched for an answer that cannot be written. The endpoint reference, the REST
    guide and the README show how; every stated endpoint count is now checked
    against the API's OpenAPI schema, and every route against the reference.

Fixed

  • A run that writes no answers names no judge and no answer-similarity
    model, since only answers are judged. A live retrieval sweep recorded the
    judge from RAG_EVAL_JUDGE_MODEL or judge_model:, and the similarity
    model from RAG_EVAL_SIMILARITY_MODEL or its default: with the judge set to
    an alias (llama3.2) it failed lasagna results validate --strict with
    UNRESOLVED_JUDGE_ALIAS, and two identical retrieval runs split on a
    leaderboard by which judge the environment named. A new run's key differs
    from an older identical one's; an older results file keeps its key however
    it is bundled, and bundled again it loses the warning. A benchmark whose
    mode resolves to generation_only (TriviaQA under auto) answers and is
    judged whatever the tracks, so it keeps its judge, and now records the
    default judge when none is named, which changes its key too.
  • The knowledge graph enable_graph_retrieval builds is kept in the metadata
    database every process shares, so a search finds the graph whichever process
    built it (Part of #55). It lived in the memory of the process that ingested:
    in the Compose stack the Celery worker built it and every API worker searched
    an empty one, silently, and a restart lost it. It is now stored per session
    and collection (SQLite, or PostgreSQL in the stack), each mention and edge
    with its document, after the document's chunks are stored, and each process
    loads it again once the session's documents change (an upload, a delete, a
    benchmark's ingest). Storing a document again replaces its part, so a
    reindex leaves no old chunk ids behind, and an upload that fails keeps none.
    A search no longer creates an empty graph. Concurrent writers lock entities
    in one order, so PostgreSQL does not deadlock them. In
    lasagna.retrieval.graph, get_or_create_session_graph is deprecated (it
    builds a graph no other process sees, and retrieval no longer reads it); use
    graph_for and index_document_graph. GraphRetriever.expand takes
    collection_name, defaulting to the session's.
  • Deleting a document removes its part of the session's knowledge graphs, on
    every delete path, and deleting a session drops what is left; a reset
    forgets the graphs a process had loaded, and a stale benchmark corpus drops
    its graph with its vectors (#55).
  • Hybrid search asks about the collection it searches, and a connection pool
    passes the question on (#50, #52). The Compose stack wraps its Milvus server
    in a pool (RAG_CONNECTION_POOL_SIZE=10) that did not pass on
    collection_supports_native_hybrid, so every collection counted as able to
    run native hybrid search. The collections Lasagna creates have no
    sparse_embedding field, so every hybrid search tried the native call,
    failed and fell back (all 12 queries of a standard run on smoke). The
    same answer skipped filling the BM25 index from the store, so wherever a
    process had not indexed the chunks itself, the fallback found it empty and
    returned dense results only. In the stack, the Celery worker ingests
    uploads, so this was every REST and MCP hybrid search an API worker ran: in
    a test of 22 searches on one upload, all 4 that ran (one per worker) fell
    back to dense-only, and the search cache served the other 18 from them. In a
    benchmark, it was a hybrid arm on a corpus ingested with hybrid off: under speed
    it scored exactly the control, ndcg@10 0.9464 on smoke, where Milvus Lite
    gives 0.9505. In the stack it now gives 0.9505, with map@10 and R-precision
    as on Milvus Lite. The base adapter answers per collection from its
    capabilities (pgvector, OpenSearch and Weaviate as before), Milvus reads the
    collection's fields, and the pool asks one of its instances. The searcher
    never answers yes without asking, and asks about the collection a session
    searches, which the benchmark names before its first search. The legacy
    Milvus searcher serves only the collection it was built for. On a collection
    without the field, MilvusBackend.hybrid_search returns dense results and
    reports a degraded hybrid stage instead of failing. Numbers move only for
    hybrid search on a Milvus server where it had run dense-only.
  • A stage that falls back while a benchmark's queries run is named in the
    results and the bundle (Part of #50). Hybrid search, the graph expansion
    and other stages catch their own failure, carry on with less and record it;
    REST, MCP and the SDK reported those stages, and the benchmark harness
    never collected them. In the Compose stack, every query of a hybrid arm
    under speed failed the native call and then ran dense-only, scoring
    exactly the control (#52), and its bundle carried no fallback. Each task now
    records system.degraded_stages: per track, the stage, on how many of its
    queries, and up to three reasons with the queries each covered, so a
    dense-only fallback is told apart from one that changed nothing. The bundle
    raises ACTIVE_FALLBACKS with one line per arm, track and stage
    (hybrid fell back during the run: smoke, arm 'hybrid', retrieval track, 12 of 12 queries (12 queries: ...; 12 queries: BM25 index empty; dense results only)), which lasagna results validate prints and --strict
    refuses; lasagna benchmark run, benchmark experiment and
    lasagna results view show the same lines. A task where nothing fell back
    records nothing new, so its bundle is what it was. Reasons are redacted
    like the rest of a bundle. A degraded.collecting() block opened inside
    another now hands its stages to the outer one.
  • A document turns ready only once a search finds it. On a Milvus server,
    where searches read at the Bounded level, a search sent the moment a
    document turned ready missed it: in the default Compose stack 27 of 30 such
    searches came back without it, and each found it a second later. Before it
    marks a document ready, an upload or a reindex now w...
Read more

Lasagna 0.2.1

Choose a tag to compare

@dolev31 dolev31 released this 04 Nov 08:45
5df85c7

A patch release: uploads to the default Docker Compose stack are indexed again.
Thank you to @tnalbant, for the CLI reference
that no longer changes with the installed Typer version, and to
@segevshlomovIBM, whose evidence runs on
Milvus Lite and in the Compose stack found seven harness bugs, filed as #46 to
#52 (still open).

Added

  • A dev container (.devcontainer/devcontainer.json): a GitHub Codespace or
    the Dev Containers extension sets up Python 3.12, the full development
    install (with CPU-only torch, under the release's constraints) and Docker on
    first start, so the Compose stack a pull request is tested in runs inside
    it.

Fixed

  • Uploads to the default Docker Compose stack (Milvus server, Celery worker)
    are indexed again. Before indexing, the worker clears what an earlier attempt
    at the document left, and a Milvus server refuses a filtered delete on a
    collection that is not loaded, so before any search had loaded it every upload
    ended in error. MilvusBackend.delete and delete_raw now load the
    collection first, as its reads already did.
  • An AUTOINDEX index is accepted by a Milvus server, whether it comes from an
    IndexConfig (which carried its default HNSW params) or from the collection
    manager with index_params set. A server refuses any param but the metric
    with AUTOINDEX; only Milvus Lite, which builds a FLAT index instead, took it.
  • The generated CLI reference labels an option's type the same way whichever
    Typer version is installed, so a checkout installed under the release's
    constraints file (Typer 0.26.8) no longer reports the page as out of date and
    rewrites it.
  • The evaluation SDK's benchmark catalog lists 2wikimultihopqa and says 32
    public benchmarks; it had kept 31 since 0.2.0. Every page that states how many
    benchmarks, components or published loops there are is now checked against
    the code, so a pull request that adds one updates each page that counts it.

Earlier changes: https://github.com/dolev31/lasagna/blob/v0.2.1/CHANGELOG.md

Lasagna 0.2.0

Choose a tag to compare

@dolev31 dolev31 released this 01 Nov 11:20
cf6d313

The first release with outside contributors. Thank you to
@ect2000, for the 2wikimultihopqa benchmark
with its distractor paragraphs and the agentic-pipeline checks in
lasagna component validate, and to @juliasadler,
for the fix to two services racing to create a collection, measured against
the stack's own Qdrant.

Deprecated

  • The old names renamed in 0.1.0 (lasagna benchmark compare, lasagna benchmark leaderboard, results interaction --bootstrap, and the names
    dropped from lasagna.__all__) and the unversioned /api/config/knowledge
    routes stay for one more release, with their warnings, and are removed in
    0.3.0. 0.1.0 said 0.2.0; their warnings now name 0.3.0.

Changed

  • The PyPI page links the documentation site and the paper.

Added

  • A pinned 2wikimultihopqa benchmark adapter with a validation corpus that
    retains gold paragraphs and distractors, with version-specific IDs that deduplicate
    formatting-only whitespace differences while keeping the first original text.
  • A "PR quality" check on every pull request (scripts/check_pr_body.py). It
    fails until the description fills in the template, a person names themselves
    as having read the whole diff, every suppression in the added lines gives a
    reason, every TODO links an issue, and a change to src/lasagna/ comes with a
    test and a line here. The no-changelog and no-new-test labels waive the
    last two. CONTRIBUTING.md has the full bar under "Quality bar".
  • A pull request template, and CODEOWNERS.
  • Issue forms for a benchmark request and a research proposal. The component
    form now asks for the stage, the paper and the number it reports, and a
    comparison with a built-in control. Blank issues are off; questions go to
    Discussions.
  • A Wanted page: the
    components, benchmarks, studies and deployment work the project would like to
    have, each with what "done" means.
  • A pull request checklist in docs/contributing-components.md.
  • An IMAGE_EXTRAS build argument for the Docker image (set in .env): extras
    installed beyond the production set, empty by default. IMAGE_EXTRAS=eval
    runs lasagna benchmark and sweeps inside the Compose stack.
  • Every pull request is tested in the production environment, the Compose
    stack built from its branch (CONTRIBUTING.md, "Tested in the production
    environment"). The "PR quality" check fails while "How it was tested" shows
    no docker compose command, and the template has a box for it.

Fixed

  • On a Milvus server, the benchmark's corpus census and pre-flight see the
    rows written just before them. A server reads with the collection's Bounded
    consistency, so a count right after ingestion came up short (0 of 24
    documents), and every cold lasagna benchmark run in the Compose stack was
    refused as "changed while this run used it". Backends gain read_fresh(),
    a block whose filter reads see every earlier write (a no-op by default; on a
    Milvus server it reads Strong, about 200 ms a query, so only these checks
    use it and search keeps its speed). iter_all on a server first runs one
    Strong query, since pymilvus's query iterator ignores the consistency level,
    so a scan right after an insert could come up short. PooledVectorBackend,
    which the Compose stack runs in front of a Milvus server, passes both on: it
    enters read_fresh on every pooled instance, and iter_all uses an
    instance's own cursor instead of paging by offset, which read stale and
    stopped at Milvus's 16,384-row limit. The pool also passes on is_lite, so
    a pool over Milvus Lite (connection_pool_size above 1) is no longer taken
    for a server.
  • Two services starting together no longer fail when the other creates the
    collection first: create_collection raises CollectionExistsError (a
    ValueError subclass) on an existing name, and collection initialization
    continues as the "already exists" branch does instead of failing to start.
    The same lost-race handling now covers the third check-then-create path,
    CollectionManager.ensure_backend_collection, which the API process and a
    Celery worker can reach at the same time.
  • Validate BaseAgenticPipeline components with an async contract check and
    report incorrect results or input mutation instead of rejecting the stage.

Earlier changes: https://github.com/dolev31/lasagna/blob/v0.2.0/CHANGELOG.md

Lasagna 0.1.0

Choose a tag to compare

@dolev31 dolev31 released this 21 Oct 22:08

The first public release. Lasagna is a RAG framework built for comparing one
part of a pipeline with everything else held fixed: eight swappable stages and
55 built-in components (each one that implements a published method names its
paper), an evaluation harness that says whether a difference is real (paired
significance and interaction tests, power, a stage isolation check, checkable
result bundles), and the same pipeline served to agents over MCP, to programs
over REST and the Python SDK, and in a Docker Compose stack.

Its metric definitions are METRICS_VERSION 1.7
(metrics.md defines
each metric). The version is part of every comparability key, so results from
the internal 0.1.0a1 pre-release (1.1) or from development builds since (1.2 to
1.6) never share a leaderboard group with results from this release.

Users of 0.1.0a1: read Changed for users of 0.1.0a1
below. The fixes made since 0.1.0a1 are described in the git history.

Added

Components, by stage (lasagna component list [stage] lists them without
loading any):

  • Generator, an eighth stage. concise (the harness's own prompt), cited
    (ALCE), chain_of_thought, chain_of_note, and the baselines closed_book
    (the question alone) and long_context (all ten retrieved passages). A
    function f(query, contexts) -> str becomes one with @lasagna.generator.
    An arm with no_rag: true retrieves nothing and answers with closed_book.
  • Retriever: dense, bm25, splade, hybrid_rrf and hybrid_convex.
  • Reranker: cross_encoder (any current Hugging Face reranker by name),
    colbert, monot5, llm_listwise (RankGPT's sliding window),
    llm_pointwise, flashrank, cohere, mmr, combined and ensemble.
  • Query enhancer: hyde, query_expansion, query_decomposition
    (least-to-most), step_back, query2doc, rewrite and multi_query.
  • Agentic pipeline: self_ask, ircot, flare, adaptive and
    self_rag, each with its paper's defaults as settings; loop_trace records
    what each loop did.
  • Chunker: semantic (with Kamradt's breakpoint methods),
    structure_aware, hierarchical, fixed, recursive, sentence_window,
    markdown_header, proposition and contextual.
  • Embedder: sentence_transformers, bge_m3, openai, cohere and
    voyage, with each model family's query and document prefixes.
  • Document processor: built-in extraction, and docling, markitdown and
    pymupdf4llm (each an extra; pymupdf4llm is AGPL-3.0 and not in all).
  • Your own component in one function. @lasagna.reranker (and a decorator
    per stage) turns a function into a component a sweep names as
    module:function; lasagna component validate checks it keeps its stage's
    contract. lasagna.from_langchain and lasagna.from_llamaindex register
    LangChain and LlamaIndex objects as stages (extra: llamaindex).

Experiments:

  • Sweeps (lasagna benchmark experiment sweep.yaml) over components and
    settings, factorial designs, repeated runs and baseline arms; chunker and
    embedder arms get corpora of their own. A key the harness does not read is
    an error naming the likely intended one, and a value out of its setting's
    range is an error naming the arm, before anything is ingested and in
    --dry-run too. The ranges are the same for an arm, a session, a settings
    file and the environment.
  • Your own data as a benchmark. A JSONL or CSV file, or a BEIR folder:
    benchmarks: [{path: data/my.jsonl}] or lasagna benchmark run ./my.jsonl.
    It registers as local/<stem>@<12 hex digits of its sha256>, so an edited
    file is a different benchmark. lasagna benchmark validate-data names each
    bad row and field. See docs/own-data.md.
  • lasagna benchmark synthesize writes factoid, multi-hop and unanswerable
    questions from your documents with the configured LLM, for development
    rather than for reported results. The plan warns when the documents are too
    few for the number of questions asked.
  • Benchmarks run beside lasagna serve. While another process holds the
    local Milvus Lite store, benchmark run and benchmark experiment use one
    of their own in ~/.lasagna/bench/dbs and say so
    (configuration.md).
  • --dry-run prints the arms, tasks, models and whether each is
    downloaded, with token, time and (from a price table, prices:) dollar
    estimates from a 3-sample probe.
  • A re-run replays its LLM responses. Requests at temperature 0 are
    answered from ~/.lasagna/cache/llm.sqlite inside an experiment; the hit
    rate is in results.json and the bundle. Off with llm_cache: false or
    --no-cache.
  • Metrics: NDCG, recall, precision, MRR, MAP, R-precision, complete
    recall, hit rate and F1 at k; exact match, token F1, ROUGE-L, answer
    containment and similarity, ALCE citation recall and precision, abstention
    rate and SQuAD v2's answerable and unanswerable splits; the RAGAS judged
    metrics with factual_correctness; latency, tokens and LLM calls per query.

Results:

  • lasagna results significance (paired sign-flip test, Holm correction)
    ends in a verdict naming each pair that survives and who won;
    results power gives the minimum detectable effect; results interaction
    tests a 2x2 design. A sweep ends with the command to run next on its
    results: one line per benchmark, and results interaction after a factorial.
  • lasagna results view writes a run as one offline HTML page:
    leaderboard, significance, every metric, the isolation check and a per-query
    diff with what each arm retrieved.
  • Every lasagna results command takes a run's folder (the one a run prints
    after "Output:") or the results.json in it, and reads --benchmark as the
    benchmark's id or the name: the sweep gave it, listing the run's
    benchmarks when it matches none.
  • Bundles (results bundle, results validate) carry what a reader needs
    to decide whether two numbers may be compared, with one comparability key per
    benchmark and configuration; results leaderboard groups by it, as markdown
    pages or LaTeX tables (--format latex), and results compare prints it
    for each side. Every results command refuses a bundle that results validate rejects; --allow-invalid shows one marked INVALID.
  • Stage isolation. Every run records a manifest of each stage's settings;
    arms that differ outside the stages they name block the bundle, and an arm
    whose named change never moved a ranking (on at least ten queries) is
    flagged inert.
  • A reproduction: on the full SciFact and NFCorpus test sets the dense
    models land within 0.002 NDCG@10 of MTEB, and BM25 with a cross-encoder
    within 0.004 of the BEIR paper (examples/reproduction/). Its
    results.csv records the commit, metrics version and date of the run that
    produced the numbers, and results_csv.py writes the same file from a rerun
    to compare.

Serving:

  • MCP server (lasagna-mcp, stdio by default, or lasagna serve --mcp)
    for Claude Code, Claude Desktop and Cursor. The default tool set, safe,
    serves no deletion tools (--tools all does). search names the stages that
    fell back and why, with the fix, also when it finds nothing (for example,
    when the session's documents are in another model's collection). An
    argument of the wrong type, or one the tool does not take (session for
    session_id), is refused in one sentence naming the one it takes.
  • REST API (lasagna serve, or the Compose stack) under /api/v1,
    covered by the deprecation policy in GOVERNANCE.md; search
    responses name the stages that fell back in degraded_stages (a reranker
    not downloaded: retrieval order and the download, also when switching to it).
  • lasagna upload takes files, globs and folders (a folder's readable
    files, subfolders included, saying what it skipped).
  • Profiles speed (92 MB, no LLM), standard, balanced and
    max_quality, applied per session from any surface. A switch that moves a
    session's documents to another model (lasagna configure --reindex,
    reindex=true over REST and MCP) checks the new models load, reindexes every
    document first and switches only when all are reindexed.
  • Deployment: a Compose stack (Postgres, Redis, a Milvus server or Qdrant,
    Celery, MCP) with per-service healthchecks, Kubernetes manifests, Prometheus
    metrics, JSON logs with request ids, and readiness that pings each store.
  • Six vector stores behind one interface: Milvus (and Milvus Lite),
    pgvector, Qdrant, Weaviate, Pinecone and OpenSearch.

For contributors: the test suite keeps the Hugging Face hub offline unless
HF_HUB_OFFLINE=0, and tests that load real weights skip when their model is
not cached.

Documentation and examples: the guide,
configuration (which setting wins, profiles, where
data lives), the generated settings reference,
model licenses, a Colab notebook comparing a
one-function reranker with the built-in cross-encoder, the plugin template,
and an offline acceptance demo (examples/acceptance/offline_demo.py).

Changed for users of 0.1.0a1

Metric definitions (METRICS_VERSION 1.1 to 1.7)

Each step changes what a number means, so each is part of the comparability
key; `results ...

Read more