fix(dfra): concept ownership + property links + RDF dedup + notifications idle - #596
Merged
Conversation
This was referenced Jul 10, 2026
…ions idle
Batch of fixes from customer report ("dfra"):
- Owners on concept/term/property didn't render: business_owners `by-object` /
`history` and metadata `/entities/{type}/{id}/{rich-texts,links,documents,
attachments}` and semantic-links `/iri/…` used path params for IRI-valued
ids, which the Databricks Apps proxy mangles by collapsing `%2F%2F` in path
segments (see #536). Switched to `?iri=` / `?object_id=` / `?entity_id=`
query params — inert to that transformation — matching the pattern #536
established for the concepts routes. Frontend callers (`ownership-panel`,
`entity-metadata-panel`, `linked-objects-panel`) updated accordingly.
- Business concepts/properties disappeared from contract columns because
`data-contract-details.tsx` declared `propertyLinks` without a setter and
never populated it. Added `fetchPropertySemanticLinks` (schema-level, one
call per schema via new `/semantic-links/entity-prefix/{type}/{prefix}`
endpoint) that replaces the slice per schema so removed assignments no
longer linger.
- RDF triples duplicated unboundedly on re-import. `_skolemize_bnode`
embedded rdflib's random per-parse bnode id, so re-importing an ontology
with blank nodes (OWL restrictions, SHACL shapes) minted brand-new URIs
that never hit the `uq_rdf_triple` constraint. Now canonicalise the graph
(RGDA1 `to_canonical_graph`) before persisting, so identical content
produces identical rows and re-imports are true no-ops.
- Added threshold-gated bloat diagnostics to `RdfTriplesRepository`: when the
table exceeds `RDF_TRIPLES_DIAGNOSTIC_THRESHOLD` (default 30000) rows, log a
one-shot forensic snapshot (blank-node count, constraint-bypass check,
source_type + context breakdown, per-(subject,predicate) churn) — needed
because the customer's 64k-with-no-file case is not locally reproducible.
- `_register_sources_as_collections` was registering the dynamic recomputed
contexts `urn:app-entities` / `urn:semantic-links` as KnowledgeCollections
(spurious timestamped metadata, polluted Collections list). Added them to
the skip set.
- `list_for_iri` returned explicit DB links + inferred graph links with no
dedup ("assignment shown twice" report). Now deduped on `(entity_type,
entity_id)`, explicit wins over the synthetic inferred link.
- Data domains stuck loading in Home Discovery when no domain named "Core"
existed. Now falls back to first root domain, then first domain.
- Notifications polling kept Lakebase compute warm 24/7 because there was no
tab-visibility handling. Store now pauses the interval on
`visibilitychange` and resumes (with an immediate fetch) on visible.
- Made `vite.config.ts` proxy target env-configurable via `VITE_PROXY_TARGET`
(dev-only ergonomics; defaults to `http://localhost:8000`).
Tests: added `test_rdf_bnode_dedup.py` (re-import idempotency, stable bnode
ids, threshold-gated diagnostic firing + throttle + no-op paths) and a
`list_for_iri` dedup case in `test_semantic_links_manager.py`.
Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
Make frontend/backend dev ports overridable via env vars so multiple git worktrees can run isolated servers without colliding on 3000/8000: - vite.config.ts: VITE_PORT (default 3000) drives server.port - pyproject.toml: dev-backend honors BACKEND_PORT/BACKEND_HOST (default 8000/0.0.0.0) - CONTRIBUTING.md: "Running Multiple Worktrees Side-by-Side" section - .cursor/rules/08-testing-and-deployment.mdc + CLAUDE.md: LLM-agent guidance VITE_PROXY_TARGET (already present) points a worktree's frontend at its own backend. Worktrees share the local app_ontos DB; only ports differ. Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
…equest
Customer redeploy was still slow. Prod logs showed every request logging
"Persistent cache not found for {stats,taxonomies,concepts}, computing live"
against a 68,684-triple graph — the caches were destroyed and never rebuilt.
Two compounding defects:
1. on_models_changed() ran rebuild_graph_from_enabled() (which rebuilds BOTH
the in-memory snapshots and the persistent JSON files) and then immediately
called _invalidate_cache(), deleting exactly what it had just built. Any
concept/link mutation wiped every cache tier.
2. The read paths (get_taxonomies / get_taxonomy_stats / get_grouped_concepts)
recomputed live on a miss but never repopulated, so the miss recurred on
every subsequent request until the next full rebuild.
Fix:
- Drop the redundant _invalidate_cache() in on_models_changed; the rebuild is
already the authoritative clear-then-rebuild.
- Add _ensure_caches_warm(): on a cold read, recompute all three tiers from the
current singleton graph via the existing atomic writer, populating memory +
files. Read paths call it on miss and return the warmed value.
- File-cache-hit branches now also populate the in-memory snapshot to avoid
re-parsing JSON every request.
Coherence: warming recomputes from self._graph, which every mutation keeps
fresh (full rebuild, or incremental graph edit + invalidate), so a warmed cache
is never staler than the graph. Verified single-worker deploy + singleton read
manager, and audited all 12 graph mutators satisfy the invalidate-or-rebuild
invariant — so this does not reintroduce the historical "changes don't show"
staleness.
Adds test_semantic_cache_warm.py covering both defects + invalidate→rewarm.
Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
The background job-polling thread opened a fresh DB session every JOB_POLLING_INTERVAL_SECONDS (default 300s) and queried workflow_installation_repo.get_all_installed unconditionally — even with zero installed workflows (customer log: "Polling 0 installed workflows..."). On Databricks Apps that query every 5 minutes never lets Lakebase reach its idle window, so compute stayed permanently warm (the reported "compute always active"). This is server-side and independent of the client notification-poll visibility fix. Fix: cache an in-memory presence flag (_has_installations). Once a cycle confirms zero installations, subsequent cycles skip the DB session entirely and just wait the interval, letting Lakebase idle down. install_workflow() resets the flag to None so a newly installed workflow is re-detected on the next cycle. Authoritative because JobsManager is a single app-state singleton on a single-worker deployment. Adds test_jobs_polling_idle.py: skip-after-zero, None-forces-recheck, non-empty-keeps-polling. Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
…oint
The panel's fetch moved from /api/semantic-links/iri/{iri} to
/api/semantic-links/by-iri?iri= in this branch, but the test mock
still matched the old path, so the panel rendered the empty state.
Co-authored-by: Isaac
will-yuponce-db
force-pushed
the
dfra-fixes
branch
from
July 13, 2026 12:06
893f568 to
86902bf
Compare
…backoff Even with workflows installed, the background job poll kept Lakebase warm and busy: every cycle it re-fetched the last 7 days of runs per job and re-committed every row unconditionally. A job scheduled every 10 min = ~1000 historical runs re-written every 5 min, forever — so Lakebase never idled and did constant redundant work. Three changes so a quiet cycle performs zero Lakebase writes and a fully idle system stops waking frequently: - Incremental window: query runs since (last_polled_at - overlap) instead of a fixed 7-day lookback, capped by JOB_POLLING_BACKFILL_DAYS for cold-start / post-downtime catch-up. Steady state fetches a handful of recent runs, not the whole history. - Change-gated writes: upsert_run skips its commit when the run row is unchanged; update_last_polled gains only_if_changed=True so an unchanged job state is not re-persisted. (list_runs still hits the Databricks control plane, not Lakebase.) - Adaptive backoff: when a cycle sees no active (non-terminal) runs, the interval doubles toward a cap (4x base); any active run or state change snaps it back to base so live jobs are still tracked promptly. Builds on the earlier skip-when-zero-installations fix. Net: an app with idle or quiet workflows lets Lakebase scale down; an actively running job is still tracked at the base cadence. Full scale-to-zero while a scheduled job runs is inherently not possible without dropping proactive tracking — that trade-off (activity-gating / push-based) is left as a follow-up. Tests: upsert skip-unchanged/commit-on-change; update_last_polled skip-unchanged/write-on-change; existing polling-idle tests still pass. Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
The dev-backend hatch script used --port=${BACKEND_PORT:-8000}, but hatch
parses ${...}/{...} as its own template syntax and fails to launch with
"Unknown context field 'BACKEND_HOST'" — so BACKEND_PORT/BACKEND_HOST never
worked. Reverted the script to a hard-coded --port=8000.
For a secondary worktree, invoke uvicorn directly with the desired port;
updated CONTRIBUTING.md, .cursor/rules/08-testing-and-deployment.mdc, and
CLAUDE.md to show that command instead of the non-working BACKEND_PORT form.
Frontend VITE_PORT / VITE_PROXY_TARGET are real process.env and unaffected.
Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9
This was referenced Jul 31, 2026
Contributor
|
Reviewed this while triaging a batch of field reports from a customer building their semantic layer on Ontos. Solid PR — the cache-warming fix in particular checks out: Two clarifications so nearby customer-reported bugs don't get closed as covered here — both are out of scope for this PR and stand as separate issues:
One small consistency nit: the new Minor: |
mvkonchits-db
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch of fixes from customer report ("dfra"). All root causes reproduced locally against isolated servers (
:8100/:3100) and verified end-to-end.Fixes
Owners on concept/term/property didn't render.
business_owners /by-object,metadata /entities/{type}/{id}/…, andsemantic-links /iri/…used path params for IRI-valued ids, which the Databricks Apps proxy mangles by collapsing%2F%2F→/in path segments (see fix(semantic-models): switch concept_iri routes to ?iri= to survive %2F%2F path collapse #536). Switched to?iri=/?object_id=/?entity_id=query params — inert to that transformation — matching the pattern fix(semantic-models): switch concept_iri routes to ?iri= to survive %2F%2F path collapse #536 established. Frontend callers (ownership-panel,entity-metadata-panel,linked-objects-panel) updated.Business concepts/properties disappeared from contract columns.
data-contract-details.tsxdeclaredpropertyLinkswithout a setter and never populated it. AddedfetchPropertySemanticLinks(schema-level, one call per schema via new/semantic-links/entity-prefix/{type}/{prefix}endpoint) that replaces the slice per schema so removed assignments don't linger.RDF triples duplicated unboundedly on re-import.
_skolemize_bnodeembedded rdflib's random per-parse bnode id, so re-importing an ontology with blank nodes (OWL restrictions, SHACL shapes) never hit theuq_rdf_tripleconstraint. Now canonicalise the graph (RGDA1to_canonical_graph) before persisting, so identical content produces identical rows and re-imports are true no-ops.Threshold-gated bloat diagnostics on
RdfTriplesRepository. When the table exceedsRDF_TRIPLES_DIAGNOSTIC_THRESHOLD(default 30000), log a one-shot forensic snapshot: blank-node count, constraint-bypass check, source_type + context breakdown, per-(subject,predicate) churn. Needed because the customer's 64k-with-no-file case is not locally reproducible._register_sources_as_collectionspolluted the Collections list by registering the dynamic recomputedurn:app-entities/urn:semantic-linkscontexts as KnowledgeCollections. Added them to the skip set."Assignment shown twice."
list_for_irireturned explicit DB links + inferred graph links with no dedup. Now deduped on(entity_type, entity_id); explicit wins over the synthetic inferred link.Data domains stuck loading in Home Discovery when no domain named "Core" existed. Now falls back to first root domain, then first domain.
Notifications polling kept Lakebase compute warm 24/7 — no tab-visibility handling. Store now pauses the interval on
visibilitychangeand resumes (with an immediate fetch) on visible.Dev ergonomics — parallel worktrees: frontend/backend dev ports are now fully env-configurable so multiple git worktrees can run isolated servers without colliding on
3000/8000.vite.config.tsreadsVITE_PORT(server port) andVITE_PROXY_TARGET(proxy target); thedev-backendhatch script honorsBACKEND_PORT/BACKEND_HOST. Documented inCONTRIBUTING.md("Running Multiple Worktrees Side-by-Side") and mirrored into the LLM-agent instructions (.cursor/rules/08-testing-and-deployment.mdc,CLAUDE.md) so Cursor/Claude sessions in secondary worktrees don't restart or rebind the primary:8000/:3000servers.Semantic caches were destroyed on every mutation and never repopulated (perf — the "still slow after redeploy" report). Prod logs showed every dashboard request logging
Persistent cache not found for {stats,taxonomies,concepts}, computing liveagainst a 68,684-triple graph. Two defects: (1)on_models_changedranrebuild_graph_from_enabled(which rebuilds both cache tiers) and then immediately_invalidate_cache(), deleting what it just built; (2) read paths recomputed live on a miss but never repopulated, so the miss recurred every request. Fix: drop the redundant post-rebuild invalidation, and add_ensure_caches_warm()so a cold read recomputes all tiers once from the (always-fresh) singleton graph via the existing atomic writer. Audited all 12 graph mutators for the invalidate-or-rebuild invariant so this does not reintroduce the historical "changes don't show" staleness (single-worker deploy, singleton read manager).Tests
src/backend/src/tests/unit/test_rdf_bnode_dedup.py— re-import idempotency, stable bnode ids across parses, threshold-gated diagnostic firing + throttle + no-op-below-threshold + disabled-when-zero.test_semantic_links_manager.py— addedtest_list_for_iri_dedups_explicit_and_inferred_same_entity.src/backend/src/tests/unit/test_semantic_cache_warm.py— cold-read warms both cache tiers, warm reads don't recompute,on_models_changedleaves caches warm, invalidate→read rewarms.Test plan
hatch -e dev run pytest src/tests/unit/test_rdf_bnode_dedup.py src/tests/unit/test_semantic_links_manager.py— new tests pass (2 pre-existing failures intest_semantic_links_managerare unrelated)./concepts/browser/urn%3Aglossary%3Atest-1%2Ftest-1.1-prop) — assign an owner via Ownership panel, reload → owner persists.00400001-0000-4000-8000-000000000001,customersschema) — column concept badges render (customer_id → customerId/Unique Identifier, email → emailAddress/PII).triples_imported: 0; total row count stable acrossPOST /api/semantic-models/refresh-graph./marketplace— domain graph auto-renders (no "Loading domains…" stuck state)./api/notificationshits every 60s; hide the tab → hits stop; return → immediate fetch then interval resumes.Non-code answers (for customer reply)
Follow-up (open)
The customer's 64k rdf_triples with no ontology file loaded is not locally reproducible —
uq_rdf_tripleholds perfectly on the current schema (all 6 cols NOT NULL perz8_fix_rdf_triple_nulls, zero constraint bypass). Most likely cause: an older build predatinguq_rdf_triple/z8's NULL-coalesce, and/or the random-bnode vector now fixed. The diagnostics in this PR will identify the vector the next time it happens; ask the customer forSELECT source_type, count(*) FROM rdf_triples GROUP BY 1 ORDER BY 2 DESC+alembic currentoutput.This pull request and its description were written by Isaac.