fix: stop hiding user f:-namespace data from variable-predicate scans - #1597
Merged
Conversation
Variable-predicate scans hid the entire https://ns.flur.ee/db# namespace in the default graph — a fossil from before named graphs existed, when commit metadata was stored in the main graph. Today commit metadata is routed to the txn-meta graph on every path (novelty stamping and both indexer emitters), so the only data the hide caught was user-authored f: vocabulary — stored f:AccessPolicy definitions being the prime case: a ?p ?o dump of a policy node returned only rdf:type, making the write look truncated. Worse, the hide only ran on the indexed scan path, so the same query returned all triples pre-index and dropped them once the background indexer caught up. Narrow the filter to the seven f:reifies* predicates, which remain hidden in every graph: they are the system-written encoding of edge annotations, unwritable by user transactions, and redundant with the edge and annotation content already visible. opts.includeSystemFacts still reveals them for inspection. Applied at all three mirrors: BinaryScanOperator::is_internal_predicate, the whole-graph fold's overlay walk, and its exactness preflight (graph_has_scan_hidden_predicates). Adds novelty-vs-indexed regression tests and documents the visibility rule in the JSON-LD query guide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
?s ?p ?oscan of a storedf:AccessPolicynode returned only itsrdf:typetriple —f:action,f:allow,f:exMessagewere invisible, while explicit-predicate lookups of the same triples returned them. Variable-predicate scans hid the entirehttps://ns.flur.ee/db#namespace in the default graph.That hide is a fossil from before named graphs existed, when commit metadata was stored in the main graph. Today commit metadata routes to the txn-meta graph (
g_id=1) on every path — novelty-side stamping (stamp_graph_on_commit_flakes, all three call sites) and both indexer emitters hard-codeg_id=1— and an empirical dump of a current ledger's default graph withincludeSystemFacts: truecontains zero system-writtenf:triples. The only data the hide still caught was user-authoredf:vocabulary, policies being the prime case: a wildcard inspection of a policy write looks like the write silently dropped everything but@type, and an export-by-crawl silently loses policy definitions.Worse, the hide only ran on the indexed scan path — the novelty path never applied it — so the same query returned all triples right after the write and dropped them once the background indexer caught up.
Fix
Narrow the filter to the seven
f:reifies*predicates, hidden in every graph as before. Those stay hidden deliberately: they are the system-written storage encoding of edge annotations (user transactions cannot assert them, so surfacing them would break export/re-import round-trips), they are redundant with the edge and annotation content already visible in results, and they live in the same graph as the edge they reify so there is no system graph to banish them to.opts.includeSystemFacts: truestill reveals them for inspection.Applied at all three mirrors so scan and fast-path semantics agree:
BinaryScanOperator::is_internal_predicate(binary_scan.rs)fast_whole_graph_agg.rs)graph_has_scan_hidden_predicates(fast_whole_graph_agg.rs)Other
FLUREE_DBchecks in the query crate (planner selectivity, annotation probes, wildcard hydration, fulltext datatype) were audited and are reifies-specific or unrelated — untouched.Compatibility note
A ledger whose index predates txn-meta routing and has never been rebuilt could re-expose old commit metadata in default-graph wildcard dumps; reindexing re-emits commit metadata into the txn-meta graph.
Testing
it_query_wildcard_system_facts.rs: userf:data visible to wildcards pre-index and post-index (pinning the novelty/indexed flip), and a full default-graph dump stays free of commit-metadata predicates.it_edge_annotations.rsreifies-hide andincludeSystemFactstests pass unchanged.fluree-db-apisuite green (30 targets, 3,041 tests);fluree-db-queryfull tests green; clippy--all-featuresclean on both crates.f:reifies*exception, andincludeSystemFactsdocumented indocs/query/jsonld-query.md(the flag was previously undocumented).