Skip to content

fix: stop hiding user f:-namespace data from variable-predicate scans - #1597

Merged
bplatz merged 1 commit into
mainfrom
fix/wildcard-fluree-ns-visibility
Aug 10, 2026
Merged

fix: stop hiding user f:-namespace data from variable-predicate scans#1597
bplatz merged 1 commit into
mainfrom
fix/wildcard-fluree-ns-visibility

Conversation

@bplatz

@bplatz bplatz commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1593 — contains that branch's commit; diff shown here is just this change.

Problem

A ?s ?p ?o scan of a stored f:AccessPolicy node returned only its rdf:type triple — f:action, f:allow, f:exMessage were invisible, while explicit-predicate lookups of the same triples returned them. Variable-predicate scans hid the entire https://ns.flur.ee/db# namespace in the default graph.

That hide is a fossil from before named graphs existed, when commit metadata was stored in the main graph. Today commit metadata routes to the txn-meta graph (g_id=1) on every path — novelty-side stamping (stamp_graph_on_commit_flakes, all three call sites) and both indexer emitters hard-code g_id=1 — and an empirical dump of a current ledger's default graph with includeSystemFacts: true contains zero system-written f: triples. The only data the hide still caught was user-authored f: vocabulary, policies being the prime case: a wildcard inspection of a policy write looks like the write silently dropped everything but @type, and an export-by-crawl silently loses policy definitions.

Worse, the hide only ran on the indexed scan path — the novelty path never applied it — so the same query returned all triples right after the write and dropped them once the background indexer caught up.

Fix

Narrow the filter to the seven f:reifies* predicates, hidden in every graph as before. Those stay hidden deliberately: they are the system-written storage encoding of edge annotations (user transactions cannot assert them, so surfacing them would break export/re-import round-trips), they are redundant with the edge and annotation content already visible in results, and they live in the same graph as the edge they reify so there is no system graph to banish them to. opts.includeSystemFacts: true still reveals them for inspection.

Applied at all three mirrors so scan and fast-path semantics agree:

  • BinaryScanOperator::is_internal_predicate (binary_scan.rs)
  • the whole-graph fold's overlay-flake walk (fast_whole_graph_agg.rs)
  • the fold's exactness preflight graph_has_scan_hidden_predicates (fast_whole_graph_agg.rs)

Other FLUREE_DB checks in the query crate (planner selectivity, annotation probes, wildcard hydration, fulltext datatype) were audited and are reifies-specific or unrelated — untouched.

Compatibility note

A ledger whose index predates txn-meta routing and has never been rebuilt could re-expose old commit metadata in default-graph wildcard dumps; reindexing re-emits commit metadata into the txn-meta graph.

Testing

  • New it_query_wildcard_system_facts.rs: user f: data visible to wildcards pre-index and post-index (pinning the novelty/indexed flip), and a full default-graph dump stays free of commit-metadata predicates.
  • Existing it_edge_annotations.rs reifies-hide and includeSystemFacts tests pass unchanged.
  • Full fluree-db-api suite green (30 targets, 3,041 tests); fluree-db-query full tests green; clippy --all-features clean on both crates.
  • Docs: variable-predicate visibility rule, the f:reifies* exception, and includeSystemFacts documented in docs/query/jsonld-query.md (the flag was previously undocumented).

Variable-predicate scans hid the entire https://ns.flur.ee/db# namespace
in the default graph — a fossil from before named graphs existed, when
commit metadata was stored in the main graph. Today commit metadata is
routed to the txn-meta graph on every path (novelty stamping and both
indexer emitters), so the only data the hide caught was user-authored
f: vocabulary — stored f:AccessPolicy definitions being the prime case:
a ?p ?o dump of a policy node returned only rdf:type, making the write
look truncated. Worse, the hide only ran on the indexed scan path, so
the same query returned all triples pre-index and dropped them once the
background indexer caught up.

Narrow the filter to the seven f:reifies* predicates, which remain
hidden in every graph: they are the system-written encoding of edge
annotations, unwritable by user transactions, and redundant with the
edge and annotation content already visible. opts.includeSystemFacts
still reveals them for inspection.

Applied at all three mirrors: BinaryScanOperator::is_internal_predicate,
the whole-graph fold's overlay walk, and its exactness preflight
(graph_has_scan_hidden_predicates). Adds novelty-vs-indexed regression
tests and documents the visibility rule in the JSON-LD query guide.
@bplatz
bplatz requested review from aaj3f and zonotope August 6, 2026 11:35

@zonotope zonotope left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🖲️

Base automatically changed from fix/jsonld-filter-fail-open to main August 10, 2026 16:55
@bplatz
bplatz merged commit 9c770af into main Aug 10, 2026
9 checks passed
@bplatz
bplatz deleted the fix/wildcard-fluree-ns-visibility branch August 10, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants