fix(superset): bridge DataModel <-> Dashboard lineage through the Chart node (1.12.9)#8
Open
joaopamaral wants to merge 2 commits into
Open
fix(superset): bridge DataModel <-> Dashboard lineage through the Chart node (1.12.9)#8joaopamaral wants to merge 2 commits into
joaopamaral wants to merge 2 commits into
Conversation
…rt node OpenMetadata's lineage graph traverses through explicit AddLineageRequest edges. For Superset the current behaviour was: - `Dashboard.dataModels` was never set, so the structural link from the dashboard to its datamodels was missing in the dashboard's Data Models panel. - `DashboardServiceSource.yield_datamodel_dashboard_lineage()` emitted a direct `DataModel -> Dashboard` lineage edge, so the lineage graph showed datamodels connected to the dashboard but skipped the chart entirely. Net result in production: the dashboard's lineage view showed `Table -> DataModel -> Dashboard` and the chart was invisible in the graph, even though it was a member of the dashboard. This change rewires the relationship so the chart is the bridge node: 1. `SupersetSourceMixin.yield_dashboard_lineage_details` now also emits `DataModel -> Chart` and `Chart -> Dashboard` edges for each chart on the dashboard. Combined with the existing `Table -> DataModel` edge the full chain in the lineage view becomes `Table -> DataModel -> Chart -> Dashboard`. 2. `SupersetSourceMixin.yield_datamodel_dashboard_lineage` is overridden to yield nothing, suppressing the base class' direct `DataModel -> Dashboard` edge so the datamodel doesn't render alongside the chart as two parallel paths into the dashboard. 3. `db_source.yield_dashboard` and `api_source.yield_dashboard` now send `Dashboard.dataModels=[]` (an empty list, not absent) on every PUT so any datamodel entries persisted from prior runs are deleted by the server. The relationship is fully represented by the lineage chain. Two helpers — `_get_chart_entity` and `_get_dashboard_entity` — resolve the Chart and Dashboard entities by FQN via `metadata.get_by_name`. If either isn't yet visible on the server (first run, before the entity POST has been committed), the bridge edges are skipped without crashing and the next run picks them up. Tests ----- Adds `tests/unit/topology/dashboard/test_superset_chart_lineage.py` with five Mockito-style unit tests: - `test_db_source_yields_empty_data_models` / `test_api_source_yields_empty_data_models`: Both sources produce `CreateDashboardRequest` with `dataModels=[]`. - `test_override_yields_no_edges`: the `yield_datamodel_dashboard_lineage` override produces zero edges. - `test_chart_bridge_edges_emitted`: when the chart and dashboard entities resolve, `yield_dashboard_lineage_details` emits both `DataModel -> Chart` and `Chart -> Dashboard` edges. - `test_no_chart_entity_skips_bridge_edges`: when the chart isn't yet resolvable on the server, both bridge edges are skipped gracefully. Three of the five tests fail on the unpatched code, exercising each of the three behaviour changes above. Manually verified in production (Superset metadata-DB connection mode) against a real Meltano dashboard: after this fix the dashboard's lineage graph renders `Table -> DataModel -> Chart -> Dashboard` with the chart visible as the bridge node, and the API confirms `Dashboard.dataModels` is cleared while all `DataModel -> Chart` and `Chart -> Dashboard` edges are present. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…exing in test Two review comments from gitar-bot on open-metadata#28240: - mixin.yield_dashboard_lineage_details was calling self._get_dashboard_entity(dashboard_details) inside the per-chart loop, doing N identical metadata.get_by_name lookups for a dashboard with N charts. Hoist the call out of the loop and reuse the result. - test_superset_chart_lineage.py was wiring SupersetSourceMixin.__mro__[1]._get_add_lineage_request onto a mock, which silently breaks if a new intermediate base class is inserted. Reference DashboardServiceSource._get_add_lineage_request directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Backports upstream open-metadata/OpenMetadata#28240 onto our
release-1.12.9line.Superset lineage now bridges
DataModel <-> Dashboardthrough the Chart node instead of emitting a directDataModel -> Dashboardedge:mixin.py: suppress the directDataModel -> Dashboardedge (yield_datamodel_dashboard_lineageemits nothing); emitDataModel -> ChartandChart -> Dashboardper chart. Adds_get_chart_entity/_get_dashboard_entityhelpers resolving entities by FQN viametadata.get_by_name.db_source.py/api_source.py: sendDashboard.dataModels=[]on every PUT to clear prior server-side entries.tests/unit/topology/dashboard/test_superset_chart_lineage.py.Why
Our 1.12.9 Superset/Looker ingestion produced
dashboardDataModel -> chartlineage edges whose target 404s, hitting the client's unguardedcreated_lineage.get(...)and failing the workflow. Routing lineage through the Chart node yields the correct graph shape and avoids the broken direct edge.Scope
Ingestion-only (4 files). No server schema/API changes -> runs against our 1.12.7 server (same major.minor;
match_versionspasses).Base
release-1.12.9(cut from upstream1.12.9-releasetag). Diff is exactly the open-metadata#28240 change.Testing
test_superset_chart_lineage.py).🤖 Generated with Claude Code