fix: multi-ledger JSON-LD queries through /v1/fluree/query (dispatcher + formatter)#1267
Conversation
Two API/engine bugs in the connection-scoped formatting path (query_from().execute_formatted), both surfacing only across ledgers with divergent namespace vocabularies: Issue 2 — hydration emitted the wrong @id for cross-ledger subjects. Multi-ledger queries stamp subject/ref bindings as Binding::IriMatch carrying the IRI decoded against the source ledger. The flat formatters use that canonical IRI, but the hydration formatter discarded it and re-decoded the foreign SID against the single primary-snapshot dict, silently producing an IRI with the wrong prefix. Thread the root's canonical IRI through format_subject and emit the root @id via compact_iri (a pure context op, independent of the namespace dict) when present; nested refs keep compact_sid. The canonical IRI is added to CacheKey so two ledgers minting the same (namespace_code, name) SID for different IRIs cannot share a cache entry. Issue 3 (formatter layer) — FromQueryBuilder::execute_formatted errored "No default graph for formatting" on fromNamed-only queries. Fall back to the first named graph (default_graphs.first().or_else(named_graphs .first())), mirroring DataSetDb::primary()'s default-then-named precedence. Tests: it_multi_ledger_formatting.rs reproduces both bugs with divergent per-ledger prefixes; a flat-select case guards the already-correct IriMatch path. Cross-ledger hydration *expansion* under divergent vocab remains a deeper limitation (composite overlay is keyed by raw ledger-local SID) and is captured as an ignored follow-up test.
…her (#1259) The connection-scoped JSON-LD query route (`POST /v1/fluree/query`) extracted the ledger id via `get_ledger_id`, which only matched a bare string `from`. Array `from` (multi-default-graph union), object `from` (time travel / graph selectors), and `fromNamed`-only queries were rejected with `400 MissingLedger` before `requires_dataset_features` could route them to the dataset execution path — even though the engine and `parse_dataset_spec` accept all of them. Add `first_ledger_identifier`, which pulls a representative source from any supported `from` / `fromNamed` shape (string, array, `{@id}` object, or `fromNamed` alias map). It is used only for the conservative bearer scope check and span recording; the full multi-graph dataset and per-ledger policy are resolved downstream by `parse_dataset_spec`. The `@t:` / `#graph` suffix is stripped so auth scopes to the base ledger. A request with no path/header/`from`/`fromNamed` is still `400`. Path- and header-scoped routes are unaffected (path/header take priority before `from` is consulted). Tests: multi_ledger_query_dispatch.rs drives the router end to end for `from: [array]`, `fromNamed`-only, and the still-rejected no-ledger case. Docs: note the dataset `from`/`fromNamed` forms on the connection route.
Dataset-aware formatting, slice 1 of cross-ledger hydration expansion. The connection formatter dropped the DataSetDb and reloaded a single view, so hydration always scanned and decoded against the primary ledger. A hydration root that lives in a non-primary ledger therefore had its properties fetched from the wrong snapshot and its predicate / value IRIs decoded against the wrong namespace dict — silently renaming a foreign ledger's vocabulary to the primary's (e.g. p:fullName became cat:fullName when their codes collided). Keep the DataSetDb alive through execute_formatted (new query_connection_jsonld_returning_dataset returns it for the multi-ledger case) and add a dataset-aware formatter entry that builds one (GraphDbRef, IriCompactor, policy) view per ledger. A hydration column's root is routed to its home ledger via the IriMatch.ledger_alias provenance, so the subject's properties, predicate keys, and @id all decode against the ledger that stores them — under that ledger's own policy enforcer, never the primary's. The expansion cache key gains the active ledger so a view-local SID can't collide across ledgers. Single-ledger queries are unchanged: one view, every lookup falls back to it. Flat SELECT / CONSTRUCT / SPARQL already carry IriMatch.iri and stay on the primary view. Nested cross-ledger refs still expand within the root's view; the union-resolution slice (and the ignored divergent-vocab expansion test) follows next. Adds cross_graph_root_properties_decode_in_home_ledger.
Dataset-aware formatting, slice 2: nested-ref union resolution.
After slice 1 routed hydration roots to their home ledger, nested refs
that crossed into another ledger still expanded within the parent's
view, so they collapsed to a bare {"@id": ...} whenever the two ledgers
encoded the IRI under different namespace codes (the previously-ignored
divergent-vocab case). The single primary snapshot also meant a foreign
ref's target was unreachable once indexed, regardless of vocabulary.
Give the hydration formatter a dataset context holding every ledger's
view. When expansion follows a ref, decode it to its canonical IRI in
the current view, then re-encode (encode_iri_strict) and expand it in
the target ledger(s): a ref inside the default-graph union resolves
across all default views, merging each contributing ledger's view of the
subject under that ledger's own policy and de-duplicating identical
triples (RDF merge); a ref inside a named graph stays in that graph.
Views whose namespace dict can't encode the IRI contribute nothing and
are skipped without a scan.
Un-ignores cross_graph_hydration_expansion_divergent_vocab, now asserting
the full depth-3 chain (movie → book → author) hydrates across three
ledgers that share no entity vocabulary.
|
Awesome! Thanks for picking up this issue, Brian. Here's Claude's description of the remaining work with the failing test includedOne gap I want to flag before this merges. The fix routes per Repro#[tokio::test]
async fn dataset_nested_projection_cross_graph_iri_resolution_via_connection() {
let fluree = FlureeBuilder::memory().build_memory();
let catalog0 = genesis_ledger(&fluree, "catalog:main");
let _catalog = fluree.insert(catalog0, &json!({
"@context": {"@vocab": "http://example.org/catalog/"},
"@graph": [
{"@id": "http://example.org/items/q1", "@type": "Item",
"name": "Item One", "isbn": "0001"}
]
})).await.unwrap().ledger;
let lists0 = genesis_ledger(&fluree, "lists:main");
let _lists = fluree.insert(lists0, &json!({
"@context": {"@vocab": "http://example.org/lists/"},
"@graph": [
{"@id": "http://example.org/lists/summer", "@type": "List",
"name": "Summer",
"contains": [{"@id": "http://example.org/items/q1"}]}
]
})).await.unwrap().ledger;
let query = json!({
"from": "catalog:main",
"fromNamed": { "lists_g": { "@id": "lists:main" } },
"@context": {
"@vocab": "http://example.org/lists/",
"catalog": "http://example.org/catalog/"
},
"select": {"?list": ["@id", "name",
{"contains": ["@id", "catalog:name", "catalog:isbn"]}
]},
"where": [["graph", "lists_g", {"@id": "?list", "@type": "List"}]]
});
let result = fluree.query_from().jsonld(&query).execute_formatted().await
.expect("nested cross-graph projection should succeed");
let s = result.to_string();
assert!(
s.contains("http://example.org/lists/summer"),
"expected @id `http://example.org/lists/summer`; got: {s}"
);
}Observed on this branch: [{"@id": "http://example.org/items/summer"}]The Why this shape mattersThis isn't an edge case — it's the canonical multi-ledger pattern the repo's own docs document at
Where the gap sits
Two fix shapesI'm not sure which fits your architecture better:
(1) is what I prototyped in the work we'd been carrying for the same issue and it's the smaller change — engine produces consistent Either way — the substantive piece is that the originally-reported shape isn't covered by the new test fixtures (which all use |
bplatz
force-pushed
the
fix/multi-ledger-query-dispatch-and-formatting
branch
from
f5af776 to
44b82ea
Compare
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
Agree w/ Jack's assessment above and have a few additional notes.
A connection query with "from": ["A", "B"] only seems to bearer-auth check on the first ledger. If the check passes on A the response can include B without a check on B (also seems true for fromNamed). The valid / correct pattern does already seem to exist within the path for multi-query e.g. fluree-db-server/src/routes/query.rs:2847:
// Bearer ledger-scope enforcement — parity with single-query
// /query and /query/:ledger. Unsigned bearer tokens may carry a
// scope that limits which ledgers they can read; any envelope
// referencing a ledger outside that scope is rejected with 404
// (avoiding existence leak), matching the single-query response.
if let Some(principal) = bearer.0.as_ref() {
if !credential.is_signed() {
for ledger_id in &distinct_ledgers {
if !principal.can_read(ledger_id) {
set_span_error_code(&span, "error:Forbidden");
return Err(ServerError::not_found("Ledger not found"));
}
}
}
}Some additional perf-related comments line-by-line.
Approving so you can merge when ready, but maybe worth considering a few of these first.
| /// Expand a referenced subject, routing across ledgers when needed. | ||
| /// | ||
| /// In single-ledger mode this is just `format_subject` against the current | ||
| /// view. In dataset mode the ref's SID is decoded (via the current view) to | ||
| /// its canonical IRI, then re-encoded and expanded in the target ledger(s): | ||
| /// a ref inside a default-graph view resolves across the **whole | ||
| /// default-graph union** (merging each contributing ledger's view of the | ||
| /// subject under that ledger's own policy); a ref inside a named graph stays | ||
| /// in that graph. This is what makes a ref that points into another ledger | ||
| /// hydrate its properties instead of rendering as a bare `{"@id": ...}` | ||
| /// (issue #1259). | ||
| fn expand_ref<'b>( | ||
| &'b self, | ||
| ref_sid: &'b Sid, | ||
| level: &'b NestedSelectSpec, | ||
| depth: DepthBudget, | ||
| visited: &'b mut HashSet<Sid>, | ||
| cache: &'b mut HashMap<CacheKey, JsonValue>, | ||
| ) -> BoxFuture<'b, Result<JsonValue>> { | ||
| async move { | ||
| let Some(ctx) = self.dataset else { | ||
| // Single-ledger: stay in the current view. | ||
| return self | ||
| .format_subject(ref_sid, None, level, depth, visited, cache) | ||
| .await; | ||
| }; | ||
|
|
||
| // Decode against the CURRENT view (where the flake lives) to recover | ||
| // the canonical IRI, then resolve it in the target ledger(s). | ||
| let iri: Arc<str> = Arc::from(self.compactor.decode_sid(ref_sid)?.as_str()); | ||
|
|
||
| // Scope: default-graph refs resolve across the default-graph union; | ||
| // a named-graph ref stays within its graph. | ||
| let targets: Vec<usize> = if ctx.default_indices.contains(&self.active_idx) { | ||
| ctx.default_indices.clone() | ||
| } else { | ||
| vec![self.active_idx] | ||
| }; | ||
|
|
||
| let mut merged: Option<JsonValue> = None; | ||
| for tidx in targets { | ||
| let tview = &ctx.views[tidx]; | ||
| // Only a view whose namespace dict can encode the IRI can hold | ||
| // the subject — skip the rest (no scan, no error). | ||
| let Some(tsid) = tview.db.snapshot.encode_iri_strict(&iri) else { | ||
| continue; | ||
| }; | ||
| let tfmt = ctx.formatter_for(tidx); | ||
| let obj = tfmt | ||
| .format_subject(&tsid, Some(Arc::clone(&iri)), level, depth, visited, cache) | ||
| .await?; | ||
| merged = Some(match merged { | ||
| None => obj, | ||
| Some(acc) => merge_subject_objects(acc, obj), | ||
| }); | ||
| } | ||
|
|
||
| // No view could resolve the subject → bare canonical @id. | ||
| match merged { | ||
| Some(v) => Ok(v), | ||
| None => Ok(json!({ "@id": self.compactor.compact_iri(&iri)? })), | ||
| } | ||
| } | ||
| .boxed() | ||
| } |
There was a problem hiding this comment.
Even though expand_ref() tries to short-circuit for the most common single-ledger graph-crawl paths, the fact that all nested ref expansion now always paths through expand_ref() AND the fact that this short-circuit returns the .boxed() heap alloc means that the short-circuit for single-ledger graph crawls now always allocates 2 boxed futures where it previously only allocated one. I think the best solution is, at the call sites where self.expand_ref() currently gets called, we should be checking THERE whether self.dataset.is_some(). If true, then expand_ref() is fine. Where false, we should continue straight to self.format_subject()
| fn test_cache_key_different_depths() { | ||
| let sid = Sid::new(100, "alice"); | ||
| let spec_hash = 12345u64; | ||
|
|
||
| let key1: CacheKey = (sid.clone(), spec_hash, 0); | ||
| let key2: CacheKey = (sid.clone(), spec_hash, 1); | ||
| let key3: CacheKey = (sid, spec_hash, 0); | ||
| let ledger: Arc<str> = Arc::from("a:main"); | ||
|
|
||
| let key1: CacheKey = (Arc::clone(&ledger), sid.clone(), spec_hash, 0, None); | ||
| let key2: CacheKey = (Arc::clone(&ledger), sid.clone(), spec_hash, 1, None); | ||
| let key3: CacheKey = (Arc::clone(&ledger), sid.clone(), spec_hash, 0, None); | ||
| // Same SID + spec + depth but a different canonical IRI (cross-ledger | ||
| // collision) must NOT share a cache entry. | ||
| let key4: CacheKey = ( | ||
| Arc::clone(&ledger), | ||
| sid.clone(), | ||
| spec_hash, | ||
| 0, | ||
| Some(Arc::from("http://other.example/x")), | ||
| ); | ||
| // Same SID + spec + depth in a DIFFERENT ledger must NOT share an entry. | ||
| let key5: CacheKey = (Arc::from("b:main"), sid, spec_hash, 0, None); | ||
|
|
||
| // Different depths should produce different keys | ||
| assert_ne!(key1, key2); | ||
| // Same Sid + spec + depth should be equal | ||
| assert_eq!(key1, key3); | ||
| // Different canonical IRI → different key | ||
| assert_ne!(key1, key4); | ||
| // Different active ledger → different key | ||
| assert_ne!(key1, key5); | ||
| } |
There was a problem hiding this comment.
The change to CacheKey to move from (Sid,u64,usize) to (Arc<str>, Sid, u64, usize, Option<Arc<str>>) seems right from a correctness perspective, but I'm just thinking about the performance impact of (1) hashing active_ledger str and also Arc cloning it on every format_subject() call. Not sure exactly how to solve for this though. If we had a u32 ledger index to key on instead of Arc<str> this would be cheaper to the point of being effectively free
Inside a GRAPH <iri> {..} scope only one graph is active, so
DatasetOperator's multi-ledger check never fires and inner bindings
come out as plain Binding::Sid encoded against the named graph's
namespace. The formatter then decodes them against the primary view,
silently mis-decoding @id for fromNamed:{alias:{@id}} + [graph,alias,..]
queries.
GraphOperator now stamps inner batches with the named graph's home
ledger when the dataset spans multiple ledgers, forcing eager
materialization so SIDs resolve before stamping.
|
Good catch on the GRAPH-scope gap, @Jackamus29 — thanks for the detailed repro. Fixed in c5c45e0: Separately, your repro surfaced a non-conformant |
Two hot-path allocations flagged in review: - Ref expansion always routed through expand_ref, whose single-ledger short-circuit returns a boxed future and then awaits format_subject's boxed future — two heap allocations per ref for the common single-ledger crawl. Add expand_or_format_ref, a plain async fn that dispatches to expand_ref only when a dataset is present and otherwise calls format_subject directly (one boxed future). - CacheKey led with the active-ledger Arc<str>, hashed and Arc-cloned on every format_subject call. Replace it with the view index (active_idx), which uniquely identifies the ledger within a hydration's shared cache. Drops the per-call string hash and Arc clone; removes the now-unused active_ledger and LedgerView::ledger_id fields.
The single-query /query handler derived one representative ledger from from/fromNamed and only scope-checked that one. A from: ["a","b"] union (or a fromNamed alias) therefore let a token scoped to the first ledger read the others. Add enforce_bearer_dataset_scope, which collects every ledger a query's from/fromNamed references and rejects the request with 404 (existence-leak avoiding) if any is outside an unsigned bearer's read scope — parity with the multi-query envelope path. Wired into both the connection and ledger-scoped JSON-LD handlers.
|
@aaj3f addressed your perf + auth notes:
Tests added for all three. |
Apply rustfmt to the multi-ledger formatter/auth changes, and regenerate testsuite-sparql/Cargo.lock, which was stale relative to fluree-db-api's flate2/zstd/indexmap deps (added with gz/zst import support) — the separate testsuite-sparql lockfile is not updated by workspace builds.
3 participants
Closes #1259.
Summary
Multi-ledger JSON-LD queries did not work correctly through the connection-scoped
/v1/fluree/queryroute. The reported issue covered three bugs; investigating itsurfaced a fourth, deeper one (cross-ledger hydration expansion). This PR fixes
all four, end to end, and adds regression coverage that reproduces each against
ledgers with divergent namespace vocabularies (the condition the existing tests
never exercised).
The work is split into four reviewable commits, smallest → largest:
correct multi-ledger JSON-LD query formattingfluree-db-apiformatteraccept array/object from and fromNamed-only in query dispatcherfluree-db-serverrouteroute hydration roots to their home ledgerfluree-db-apiformatterexpand cross-ledger refs against their home ledgerfluree-db-apiformatterBackground — why it broke
The connection path was built around single-ledger queries with a string
from.Multi-ledger support (
fromarray,fromNamed, GRAPH alias) was wired into theengine and
parse_dataset_spec— where it works — but neither the HTTP route'sget_ledger_idnor the result formatter were updated to match. The formatter,in particular, threw away the multi-view
DataSetDbafter execution and reloaded asingle ledger view, so every result IRI was decoded against one ledger's
namespace dictionary. Because each ledger numbers its namespaces independently, a
SID from another ledger silently decoded to the wrong prefix — or its properties
didn't resolve at all. The existing multi-ledger tests only passed because they
shared
schema.org/wikidata.orgacross every ledger, so the colliding codeshappened to line up.
What changed
1. Formatter — wrong cross-graph
@id(Issue 2, silent data corruption)Multi-ledger queries already stamp result bindings as
Binding::IriMatch, whichcarries the canonical IRI decoded against the source ledger. The flat SELECT /
CONSTRUCT formatters use it; the hydration formatter discarded it and re-decoded
the foreign SID against the primary dict. Hydration now emits the root
@idfromthe canonical IRI (
compact_iri, which is namespace-dict-independent), and theexpansion cache key gained the active ledger so a view-local SID can't collide
across ledgers.
2. Formatter —
fromNamed-only formatting error (Issue 3-layer-2)FromQueryBuilder::execute_formattederrored"No default graph for formatting"when a query had only named graphs. It now falls back to the first named graph
(
default_graphs.first().or_else(|| named_graphs.first())), matching thedefault-then-named precedence the sibling
DatasetQueryBuilderalready used.3. HTTP dispatcher — array/object
fromandfromNamed-only rejected (Issues 1 & 3-layer-1)get_ledger_idonly matched a bare stringfrom, so arrayfrom(multi-default-graphunion), object
from(time travel / graph selectors), andfromNamed-only querieswere rejected with
400 MissingLedgerbefore the dataset path could run. A newfirst_ledger_identifierextracts a representative source from any supported shape;it is used only for the conservative bearer scope check and span recording, while
per-ledger policy and routing are resolved downstream by
parse_dataset_spec. Arequest with no path/header/
from/fromNamedis still400.4. Formatter — cross-ledger hydration expansion (the deeper bug)
Fixing #2 made the cross-graph
@idcorrect but did not make a foreignsubject's nested properties expand: hydration scanned only the primary snapshot
(plus a composite novelty overlay keyed by raw ledger-local SID), so a ref into
another ledger collapsed to a bare
{"@id": ...}— and indexed foreign data wasunreachable entirely. The formatter is now dataset-aware:
DataSetDbalive (query_connection_jsonld_returning_dataset) andbuilds one
(GraphDbRef, IriCompactor, policy)view per ledger.IriMatch.ledger_aliasprovenance, so its properties, predicate keys, and
@iddecode against the ledgerthat stores them — under that ledger's own policy enforcer, never the primary's.
re-encoded (
encode_iri_strict) and expanded in the target ledger(s). A ref insidethe default-graph union resolves across all default views, merging each
contributing ledger's view of the subject (RDF merge, de-duplicating identical
triples) under that ledger's own policy; a ref inside a named graph stays in that
graph. Views whose namespace dict can't encode the IRI contribute nothing and are
skipped without a scan.
Single-ledger queries are unchanged: one view, every lookup falls back to it. Flat
SELECT / CONSTRUCT / SPARQL already carry
IriMatch.iriand stay on the primaryview.
Design decisions
policy enforcer — reusing the primary's would be both semantically wrong and a
plausible leak path (policy checks and class caches are SID-local).
from: [A, B]default graph is a union, so asubject visible in both is merged from both (each under its own policy), rather
than taking the first match (order-dependent, hides data) or guessing ownership
from vocabulary.
Sidis view-local; cache and routing keysincorporate the active ledger / canonical IRI so one ledger's rendering can never
be served for another's.
Testing
New
fluree-db-api/tests/it_multi_ledger_formatting.rsreproduces every issueagainst ledgers with divergent entity/predicate vocabularies:
cross_graph_projection_iri_decode— foreign-ledger root@id(Issue 2)flat_select_cross_graph_iri_decode— guards the already-correct flat pathfrom_named_only_formats—fromNamed-only formats (Issue 3-layer-2)cross_graph_root_properties_decode_in_home_ledger— foreign root's divergentpredicate decodes in its home ledger
cross_graph_hydration_expansion_divergent_vocab— full depth-3movie → book → authorchain across three ledgers that share no entityvocabulary
New
fluree-db-server/tests/multi_ledger_query_dispatch.rsdrives the Axum routerend to end for
from: [array],fromNamed-only, and the still-rejected no-ledgercase.
No regressions across
it_query_connection(including the shared-vocab depth-3hydration test),
it_query_dataset,it_named_graphs, allit_query_jsonld*/hydration / reverse / construct / typed / sparql suites, the format unit tests, and
server
integration/cross_ledger_http_integration/data_auth_integration.Compatibility & risk
before
fromis consulted).QueryResultschema change, no core/index changes; the dataset-aware work iscontained in
fluree-db-api's formatter + connection builder, plus theserver dispatcher relaxation.
docs/api/endpoints.mdnow documents the datasetfrom/fromNamedformson the connection route.
Out of scope / follow-ups
the cross-ledger routing path is identical for indexed vs novelty data (both go
through
encode_iri_strict+ the target view'sdb.range), so the divergent-vocabtest already exercises the logic; a dedicated indexed test is belt-and-suspenders.
cannot loop infinitely, but a pathological cross-ledger cycle within budget could
re-expand rather than collapse. Tightening
visitedto canonical-IRI keying is apossible hardening follow-up.