fluree doc: --model and --entities, grounded entity and relation extraction; hybrid search - #1764
Conversation
…action `fluree doc ingest` can now say what a document is about, grounded on two graphs the user already has: - `--entities <ledger|file[#Class]>` (repeatable) is a gazetteer. Every label (skos:prefLabel/altLabel/hiddenLabel, rdfs:label, schema:name, schema:alternateName) goes into one Aho-Corasick automaton; each chunk is scanned for the longest whole-word match, case-folded, and again over Snowball stems. A match is a doc:Mention on the chunk (NIF offsets, anchor, source element) whose nif:entity is the subject's OWN IRI from the source, so a query across the entities ledger and the documents ledger joins with no mapping. Needs no model. - `--model <ledger|file>` is an ontology. Its classes and properties are rendered into the extraction system prompt (the same prompt Fluree AI's hosted extraction uses) and index the predicate gate. Per chunk the llm slot is asked about the text with the known entities found in it; its entities resolve to gazetteer IRIs where a label matches and are minted under a name-derived IRI otherwise, an entity whose excerpt is not in the chunk is dropped, and each relation predicate is judged valid, repaired (unambiguous label, local name, compact form or class-to-property) or rejected. Every relation is written reified with excerpt and verdict; `--relations direct` (default) also writes an edge for admitted ones, `reified` writes nodes only, `off` skips relations. Sources are loaded through the database itself: a file goes into a scratch in-memory ledger and is queried like any other, so there is no second RDF stack. The ledger's own minted doc:Entity nodes join the gazetteer on later runs so a corpus converges on one node per name. Re-runs: the document node carries an extraction fingerprint (ontology, entity sources, model, guidance, relation mode) and joins the unchanged check; mentions and relations are retracted with the document, minted entities are shared and kept, and an asserted edge is dropped only when no remaining relation supports it. A third cache keys each chunk's answer on the exact ask. The crop reader's wire code moves into a shared LlmClient (chat completions or the gateway's Responses route with `instructions` and the `extraction` intent); Chunk gains per-element spans so a mention names the paragraph or cell it sits in. Docs: a new Unstructured concept page, the CLI reference, vocabulary, tiers, caching and querying pages updated. Tests: unit coverage for the model gate, scanner, resolution and cache, and CLI integration tests against a stub model server covering known-IRI reuse, minting, hallucination drop, repair and rejection, unchanged skip, cache hit and edge cleanup on re-ingest.
…odels, fit oversize crops Three findings from the first runs over real documents through a Fluree AI account. The gateway's Responses route drops the `instructions` field, so the extraction system prompt (ontology, field names) never reached the model: every relation parsed with null subject and object, and entity types were free-form. A system-role message with string content does get through, so LlmClient sends the system prompt that way; OpenAI accepts both. The extraction cache key carries a wire version so answers cached before this are not reused. A Fluree model declares its properties through SHACL: node shapes with sh:targetClass, property shapes with sh:path, sh:class or sh:datatype, sh:name and sh:description. The model loader now reads those alongside rdf:Property and OWL declarations (magna: 16 classes, 119 properties). A dense drawing renders to a crop past the gateway's 6 MB request cap. Crops are now fitted under 4.2 MB the way the hosted extractor does it: Lanczos downscale of the long edge from 2048 px to a 1024 px floor, sent untouched when they already fit, cached on the original pixels. Also: schema:legalName joins the label predicates (supplier masters use it), and a `#Class` scope under an unknown prefix is refused with a hint to write the full IRI rather than failing inside the query.
… chunking joins the unchanged check Default chunks are now 1500 to 4000 characters, what Fluree AI's hosted embedder cuts for text-embedding-3-small, and embedding batches are 64. The heading-boundary cut and start-of-chunk labelling stay: they are the two places this chunker is deliberately better than the one it was ported from. The document node records the chunk sizes it was cut with, and a document is unchanged only when they match too. Before this a changed --min-chars or --max-chars over an unchanged folder did nothing.
A name the ontology has no class for is kept flagged for review, not trusted: fed back into the scan, "notes" tagged eighty-eight spans of one corpus. The target ledger's doc:Entity nodes still join the gazetteer, minus those marked doc:offModel.
`fluree doc search --mode hybrid` runs both indexes, each for three times the requested count, and fuses them. The two methods do not score alike: a cosine sits in 0..1 and 0.9 is very high, while a BM25 weight is an unbounded sum where 3 is weak and 35 is strong, so fusing by rank alone would credit a weak first place as much as a strong one. Each score is put on one confidence scale first, cosine as it is and BM25 as s / (s + 10), and a chunk's fused score is the mean over both methods with a missing method counting 0. Agreement wins; a strong single-method hit still outranks a weak agreed one. Each hit prints the raw score and rank from every method that returned it. `auto` now picks hybrid when both indexes exist and the query can be embedded, else whichever index there is. On the Fed statements the query "paramount lien on Reserve Bank assets" ranks the answering chunk tenth by vector and first by text; fused, it leads at 0.53 against 0.28. JSON output is now one object per hit (score, chunk, document, file, path, text, ranks) instead of a positional array.
…n the account Hybrid search ran the vector lane, embedding included, and then the BM25 lane, so it cost their sum. The two are independent and now run under try_join, so hybrid costs the slower lane: on the Fed ledger 4.3 s against 3.9 s for vector alone and 2.2 s for text. Every search also began by resolving the account remote, which meant an authenticated round trip to refresh the login before a BM25 query that needs no token. Whether the query could be embedded is now decided from the local config alone; the account is asked for its token only when a vector lane runs, and only when the stored login is within a minute of expiring, read from the JWT's exp without a server call.
A CLI process never runs the background indexer a server would, so a ledger written by `doc ingest` stayed at index t=0 and every later invocation replayed its commits into novelty before answering. On the Fed statements, six commits carrying 209 vectors of 1536 floats, that was 824 ms for a trivial query and 2.1 s for a BM25 search; indexed, 6 ms and 37 ms. `fluree index`'s build-and-publish is factored into `index_ledger` and ingest calls it after the documents, before the vector and full-text syncs. `--no-index` now skips all three.
bplatz
added
enhancement
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz this makes sense and I have no disagreement with any of the fixes, their design, or their implementation (particularly given how they build on #1760). Claude review below:
This is careful work: the part I care about most in an extraction pipeline is who gets to decide identity and predicates, and here it's never the model: the gazetteer's IRI wins over anything minted, a minted IRI is a hash of the normalized name (so it can't carry hostile characters and one name is one node across runs), an entity whose excerpt isn't in the chunk is dropped with its relations, off-model types are flagged rather than adopted (and no longer seed the next run's gazetteer), and every predicate goes through the same exact-one Valid/Repaired/Rejected gate solo's extractor uses, with rejected relations kept only as reified evidence. Hybrid search fusing calibrated scores rather than ranks is the right call and the reasoning in the description is convincing. The four "found by running real documents" fixes are each real — the instructions drop in particular matches what solo-proxy#285 fixes on the proxy side, and your client-side choice works with both.
What I verified locally, since nothing ran in CI for this head: cargo fmt --check clean, cargo test -p fluree-db-doc 51/51, and the docs table's flags and defaults match the clap definitions. What I could not run: clippy --all-targets on fluree-db-cli with default features and the CLI integration tests, both because the CLI's new default vector feature (from #1760) needs a C++ toolchain for usearch/cxx that this machine doesn't have — which is the practical reason to retarget to main after #1760 merges, so the Linux lane runs them before this lands. One thing I'd fold in now: delete_triples_update interpolates operator-supplied gazetteer/model IRIs into DELETE DATA without the sparql_iri_safe-style check solo's publisher uses; the document IRI path is already percent-encoded, so it's a one-liner to match. And a sentence in the concept page that direct edges are claims-with-evidence would set operators' expectations right.
Adherence to repo commitments:
- Patterns/abstractions: ✔ extends
fluree-db-docand thedoccommand; sources load through the database itself (scratch in-memory ledger) rather than a parallel loader; same gate semantics as solo's extractor. - Performance (speed first, memory second): ✔ no engine hot path; ingest now indexes the ledger (824 ms → 6 ms on the body's measurement); extraction bounded by concurrency, token and crop caps.
- Testing: ✔ 51 unit tests green locally; CLI integration tests against a stub model server (+334) exist and are wired;
⚠️ not executed here and not by CI on this head — retarget so they run. - Conventions: ✔ thorough multi-line commits; docs updated across the CLI reference, ingest/search, concepts, configuration, vocabulary;
fmt --checkclean;⚠️ stacked-PR CI gap.
Verified locally at branch HEAD 3ae651a33: cargo fmt -p fluree-db-doc -p fluree-db-cli -- --check → clean; cargo test -p fluree-db-doc → 51 passed; cargo clippy -p fluree-db-doc --all-targets -- -D warnings → clean; cargo clippy -p fluree-db-cli --no-default-features --features server,iceberg,shacl,aws → does not compile (vector API used with no cfg gate — inherited from #1760, filed there); extract.rs::resolve, minted_iri, document_iri, both SPARQL builders and their call sites read; docs-vs-clap spot-check on --relations, --concurrency, --mode.
Approving so you can merge once it's re-based and CI has seen it — and let's get the IRI gate in while it's one line.
| pub fn delete_triples_update(triples: &[(String, String, String)]) -> String { | ||
| let body: Vec<String> = triples | ||
| .iter() | ||
| .map(|(s, p, o)| format!("<{s}> <{p}> <{o}> .")) |
There was a problem hiding this comment.
Should-fix (fold in now). delete_triples_update interpolates every subject/predicate/object into <…> with format!. The triples come back from asserted_triples_query (doc.rs:471), so subjects and objects are minted hex IRIs or gazetteer IRIs and predicates are model property IRIs after the gate.
Minted ones are safe by construction; gazetteer and model IRIs are operator-supplied (--entities file, --model file), so a malformed IRI carrying > breaks out of the brackets and splices extra DELETE DATA triples on the target ledger. Low likelihood — the operator owns those files — but solo's publisher gates exactly this with a one-line sparql_iri_safe predicate (publish.rs:630 on solo#1118), and the same check here (refuse or percent-encode anything outside [^<>"{}|^\x00-\x20]) closes it. retract_update (:134) is already safe because document_iri` percent-encodes.
Minor and non-blocking — if you agree it's right, I'd rather see it here than in the backlog.
| }; | ||
| let names = | ||
| std::iter::once(name).chain(e.alternate_names.iter().flatten().map(String::as_str)); | ||
| // The gazetteer's IRI wins over anything minted, and over the |
There was a problem hiding this comment.
Praise, and the line to protect. "The gazetteer's IRI wins over anything minted" plus the zero-occurrence drop at :553 and the exact-one judge at :573 is what keeps a document from inventing identities or predicates; a rejected relation surviving only as a reified node with its verdict (:589) is the right evidence-not-edge posture. If the repair rules ever grow a "closest match", that's where the model starts writing the graph.
| @@ -0,0 +1,89 @@ | |||
| # Entities and relations | |||
There was a problem hiding this comment.
Optional (docs). Worth one plain sentence for operators: an untrusted document can attach admitted-predicate edges to any gazetteer entity it names by label — that is the feature — so a doc: ledger's direct edges are claims with evidence (the reified node carries excerpt and verdict), and downstream consumers should read them that way.
| // former and drops the latter, and OpenAI accepts both. | ||
| let mut input = Vec::new(); | ||
| if let Some(system) = req.system { | ||
| input.push(serde_json::json!({ "role": "system", "content": system })); |
There was a problem hiding this comment.
Note (cross-repo). Sending the system prompt as a system-role message because the Fluree AI gateway's proxy path drops instructions is the right client-side move, and it keeps working after solo-proxy#285's fix(responses) (which makes the proxy honor instructions) lands — both generations of client are then served. Worth cross-linking the two in the description so nobody "simplifies" the client back once the proxy is fixed.
#1760 landed on main, so the vector lane's HNSW index is gone. Conflicts resolved by keeping this branch's structure — two concurrent lanes, hybrid fusion, the post-ingest ledger index — and swapping what the vector lane runs. The lane now scores every chunk's `doc:embedding` with `cosineSimilarity`, ordered and cut by LIMIT, instead of searching a graph source. Fusion is untouched: it calibrates against a cosine in 0..1, which is exactly what flatrank produces, so the same scores arrive on the same scale. `search_hits` takes the where-clauses that bind `?c` and `?score` plus an optional query vector, rather than a single index pattern, so both lanes build their own head and share the chunk join. Mode selection asks whether the chunks carry embeddings (`has_embeddings`) rather than whether a vector graph source exists, and the "no vector index" error becomes one about embeddings, since that is now the real precondition.
`delete_triples_update` interpolated every subject, predicate and object
into `<…>` with `format!`. Minted IRIs are hex and document IRIs are
percent-encoded, but gazetteer and model IRIs come from operator-supplied
files (`--entities`, `--model`), and one carrying `>` closed the brackets
and spliced extra triples into a `DELETE DATA` on the target ledger.
`sparql_iri_safe` applies the RFC 3987 IRIREF exclusions. An unsafe triple
is skipped rather than escaped: it is stored under that exact IRI, so a
rewritten one would retract nothing and hide that it happened. The caller
filters first and reports what it left in place; the builder filters again
so it cannot be misused, and returns `None` when nothing is left.
The test asserts the injected IRI produces no update and that a safe
triple alongside it still retracts exactly one. Verified non-vacuous by
removing the filter: the update then reads
`DELETE DATA { <https://ex.org/x> <https://ex.org/p> <https://ex.org/o> .
<urn:victim> <https://ex.org/knows> <urn:fluree:doc:b> . }`.
Also documents that a `doc:` ledger's direct edges are claims with
evidence, not assertions: the gate decides which predicates may be
written, never which statements are true, so any ingested document can
attach an admitted-predicate edge to a gazetteer entity it names. The
reified node beside every edge carries excerpt, verdict and source, and
`--relations reified` writes the evidence without the edges.
The gateway's proxy path drops Responses `instructions`, so the prompt is sent as a system-role message instead. That stays correct once the proxy honors `instructions` — both shapes are then served — while reverting it breaks any client pointed at a proxy that has not been updated, and the failure is a prompt-less call that still returns 200 with null subjects and objects in every relation.
|
Thanks — all three folded in. IRI gate. Claims-with-evidence. New section on the concept page, in the terms you put it: the gate decides which predicates may be written, never which statements are true, so any ingested document can attach an admitted-predicate edge to a gazetteer entity it names by label. Every edge has a reified node beside it with excerpt, verdict and source, and System-role message. The "do not simplify this back" note now lives as a comment in #1760 merged, so this retargeted to The vector lane also changed with #1760 — it is flatrank now, not an HNSW search. Fusion is untouched, since it calibrates against a cosine in 0..1 and that is exactly what flatrank produces. One gap I would rather name than leave implied: hybrid has unit coverage for |
2 participants
Adds the extraction stage to
fluree doc ingest, hybrid search tofluree doc search, and the fixes found by running both over real documents through a Fluree AI account. #1760 has merged, so this now targetsmainand CI sees the whole diff.Extraction:
--modeland--entities--entities <ledger|file[#Class]>, repeatable, is a gazetteer. Every label (skos:prefLabel/altLabel/hiddenLabel,rdfs:label,schema:name/legalName/alternateName) goes into one Aho-Corasick automaton; each chunk is scanned for the longest whole-word match, case-folded, and again over Snowball stems. A match is adoc:Mentionon the chunk (NIF offsets, anchor, source element) whosenif:entityis the subject's own IRI from the source, so a dataset query over the entities ledger and the documents ledger joins with no mapping. Needs no model.--model <ledger|file>is the ontology:rdfs:Class/owl:Class,rdf:Property/OWL properties, and SHACL shapes (sh:targetClass→sh:property→sh:path/sh:class/sh:datatype), which is how a Fluree model usually declares its properties. It is rendered into the extraction system prompt (the same prompt the hosted extractor uses) and indexes the predicate gate. Per chunk thellmslot is asked about the text with the known entities found in it; entities resolve to gazetteer IRIs where a label matches and are minted under a name-derived IRI otherwise; an entity whose excerpt is not in the chunk is dropped; a new entity typed outside the ontology is kept flaggeddoc:offModel(or dropped with--drop-off-model); each relation predicate is judgedvalid,repaired(unambiguous label / local name / compact form / class-to-property) orrejected. Every relation is written reified with excerpt and verdict;--relations direct(default) also writes an edge for admitted ones,reifiedwrites nodes only,offskips relations. A relation whose object names no entity carries it as a literal.doc:Entitynodes join the gazetteer on later runs (off-model ones excluded), so a corpus converges on one node per name.[doc.extraction]config so every run over a corpus makes the same ask:guidancetext, andsystem_prompt/user_prompttemplate files keeping the{model}/{guidance}/{existing}/{document}slots.--concurrencyat a time (default 4). A chunk whose call fails keeps its gazetteer mentions; the document is written but not stamped as extracted, so the next run retries it while the cache answers for the rest.Search
--mode hybridruns BM25 and vector and fuses them on calibrated scores: cosine as it is, BM25 ass / (s + 10)(3 → 0.23, 35 → 0.78), averaged over both methods with a missing method counting 0. Rank-only fusion would credit a weak first place as much as a strong one. Each hit prints every method's own score and rank.autopicks hybrid when the chunks carry embeddings, the full-text index exists and the query can be embedded.fluree doc— Fluree Unstructured: documents into a searchable graph #1760: it scores every chunk'sdoc:embeddingwithcosineSimilarity, ordered and cut by LIMIT, rather than searching an HNSW graph source. Fusion is unchanged — it calibrates against a cosine in 0..1, which is exactly what flatrank produces, so the same scores arrive on the same scale.search_hitsnow takes the where-clauses that bind?cand?scoreplus an optional query vector, so both lanes build their own head and share the chunk join, and mode selection asks whether the chunks carry embeddings rather than whether a vector index exists.Found by running real documents
instructionsfield, so the extraction prompt never reached the model (every relation came back with null subject and object). The client sends the system prompt as a system-role message, which both the gateway and OpenAI accept; the extraction cache carries a wire version so prompt-less answers are not reused. The proxy-side fix is separate.doc ingestnow indexes the ledger after writing. The general case for CLI writers is Follow-up: Indexing strategy for CLI-only use: ledgers written by one-shot commands stay unindexed #1762.Verification
cargo fmt --check, workspace clippy with all features and targets under-D warnings.it_iceberg_directtests could not start LocalStack (Docker daemon wedged on the dev box) and the docs-embed test found two gitignored local design pages that no checkout has.main:cargo test -p fluree-db-doc53/53, the fullfluree-db-clisuite 357/357,fmtand clippy clean on both crates.PRT-292191 depicts COATED SHAFT, change orders with NOC numbers, dates and signer edges), embeddings through the real OpenAI endpoint, and a string query answered by the chunk carryingFederal Reserve notes governed by FRAwith its lien excerpt.Addressed from review
delete_triples_updateinterpolated subjects, predicates and objects into<…>; gazetteer and model IRIs come from--entities/--modelfiles, so one carrying>closed the brackets and spliced extra triples into aDELETE DATA.sparql_iri_safeapplies the RFC 3987 IRIREF exclusions. An unsafe triple is skipped, not escaped — it is stored under that exact IRI, so a rewritten one would retract nothing and hide that it happened; the caller reports what it left in place. Shown non-vacuous by removing the filter, which producesDELETE DATA { <https://ex.org/x> <https://ex.org/p> <https://ex.org/o> . <urn:victim> … }.--relations reifiedwrites the evidence without the edges — the right mode for a corpus you do not trust.instructions(both shapes are then served), while reverting it breaks any client pointed at a proxy that has not been updated — and the failure is a prompt-less call that still returns 200, with null subjects and objects in every relation.Docs
New Unstructured concept page (entities and relations, with the cross-ledger query), and updates to the CLI reference,
ingestandsearchpages (modes and how hybrid scores), configuration ([doc.extraction]), vocabulary, caching and the querying guide.Follow-up: #1762