ENG-3767: Land DSR traversal visualizer (initial merge from PoC)#8168
Merged
Conversation
Co-Authored-By: mfbrown <michael.brown@ethyca.com> Co-Authored-By: Lucano Vera <lucanovera@live.com.ar> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Apply post-cherry-pick code-quality pass on the DSR traversal visualizer: - Replace inline-styled `<div>` containers with `<Flex>`; extract chrome styling (sticky header, gating-task pill) into SCSS modules using fidesui palette tokens instead of hardcoded hex colors. - Convert template-string classnames to `classNames()`. - Introduce enums for `ActionType`, `Reachability`, `ActionStatus`, and `LaneId` (moved from `constants.ts` to `types.ts`); update all consumers + tests. `REACHABILITY_COLOR` now typed against `CUSTOM_TAG_COLOR`. - Replace `as any` test casts with a typed `buildIntegration()` fixture helper. Fix pre-existing fixture bug (`property.key` -> `property.id`). - Switch component default exports to named exports across the feature; Next.js page wrappers retain default export but use named imports. - Replace `propertyKey!` non-null assertion with RTK Query `skipToken`. - Fix antd v6 deprecation: `Select` `optionFilterProp` -> `showSearch` object config. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a self-contained fixture and handlers so the DSR traversal visualizer can be exercised via `npm run dev:mock` without seeding the fidesplus DB. Fixture covers all card lanes (stage-1 + stage-2 reachable, gated, not-touched), an inter-stage dependency edge with dep_count > 1, and a manual review task that gates one integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dependency edge legend swatch was a solid blue line, but the actual edges render dark dashed (react-flow's animated style). Use minos for the stroke + a dashed pattern in the legend swatch so the key matches what users see on the canvas. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace canvas height calc with flex:1 so the visualizer fills the remaining viewport height naturally. - Add a PageHeader matching the "Request workflows" nav item; move controls (property picker, action toggle, show-not-touched, summary) below it. - Drop the sticky positioning, horizontal padding, background, and bottom border from the header; consolidate the header styles into Tailwind utility classes and remove the now-unused SCSS module. - Wrap the canvas content in an antd Card so it gets a uniform border + rounded corners on all four sides without hand-rolling CSS. - Replace the text "Regenerate" button with an icon-only button using Icons.Renew + a tooltip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mary Reserve the summary row's vertical space so the page doesn't jump when the first preview loads, and swap the comma-separated string for the codebase's standard `Statistic + Text` stat pattern (as used in DSRStatusCard and AstralisPanel). Numbers now read with proper visual weight instead of inline bold inside secondary-colored text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hide the "Request workflows" nav item (and the routes it points to) behind a new beta feature flag. Defaults to on in dev/test and off in production, matching the rollout pattern used by dataCatalog, webMonitor, policies, and other recently-shipped beta features. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…kens LaneChrome and ManualTaskNode SCSS modules used raw hex colors throughout, violating the styling priority that requires palette tokens, antd CSS vars, or fidesui global vars. Swap every literal for the appropriate token: success-bg/border + success-text-active for the reach lane and stage labels; warning-bg/border for the gated lane and manual-task chrome; brand-bg-sandstone for the identity lane; fill-quaternary + neutral border for the skipped lane; color-text-tertiary for rail/chevron strokes; bg-container for white surfaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace manual filter+join class-string construction with the classnames package's object syntax, matching the convention already used in ManualTaskNode and LegendPanel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The traversal-preview slice keyed its cache on the full args including `refresh`, so the regenerate button (which passes `refresh: true`) populated a parallel cache entry instead of replacing the existing one. Strip `refresh` from the serialized key via `serializeQueryArgs` and force the network round-trip with `forceRefetch` when the caller opts in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ydrate The lane-collapse hook starts with hardcoded DEFAULTS for SSR safety, then swaps in the localStorage value in a mount effect. When stored prefs differ from defaults, the first ReactFlow paint uses the wrong lane state and the FitViewOnLayoutChange animation visibly snaps to the correct layout on the next tick. Have the hook expose a `hydrated` flag that flips true once the mount effect runs. TraversalCanvas renders a Spin placeholder until then. First paint of ReactFlow now matches localStorage so no re-layout animation fires on load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eight NODE_HEIGHT was a fiction: its comment claimed 160 was the rendered card height, but the IntegrationNode SCSS set min-height to 210 and CARD_PITCH was derived from NODE_HEIGHT + 70. Three hand-tuned numbers that had to harmonize, plus dead dagre-based layout code (layout-utils.ts) that nothing imported. Replace with: - INTEGRATION_CARD_MIN_HEIGHT (210) -- the actual minimum. - INTER_CARD_GAP (20) -- vertical gap between cards. - CARD_PITCH = INTEGRATION_CARD_MIN_HEIGHT + INTER_CARD_GAP -- derived. IntegrationNode applies min-height inline from the TS constant so the SCSS can't drift from the layout math. layout-utils.ts deleted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace `_refresh` destructure (rejected by naming-convention rule) with a shallow-copy + delete in serializeQueryArgs, and apply prettier formatting on the header summary row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
…query CodeQL flagged the hand-built route string in TraversalVisualizerPage's `goTo` as DOM text reinterpreted as HTML -- the property key from the picker was being interpolated directly into the URL, which would let a malformed key smuggle path separators or fragments. Use the existing `DSR_TRAVERSAL_PROPERTY_ACTION_ROUTE` constant with router.replace's pathname+query form so Next.js does the substitution and encoding. Also addresses code-review item #7 from the earlier session pass (use the route constant instead of hand-building paths). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gilluminate
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8168 +/- ##
==========================================
- Coverage 85.66% 85.10% -0.57%
==========================================
Files 665 669 +4
Lines 43086 43370 +284
Branches 5041 5080 +39
==========================================
Hits 36911 36911
- Misses 5067 5351 +284
Partials 1108 1108 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…st coverage - Fix changelog: remove incorrect db-migration label - Move local import to top level (reachability → builder) - Modernize typing: Dict→dict, List→list, Optional→X|None (all 4 files) - Document skipped field contract on CollectionDetail schema - Surface traversal warnings instead of swallowing them silently - Deduplicate: _dataset_to_integration computed once, _edges_from_counts extracted, graph.identity_keys replaces manual field walking - Scale: _static_dataset_detail O(N×M)→O(1) via _dataset_nodes and _integration_datasets indexes; _static_edges identity loop replaced with set comprehension over graph.identity_keys - Fix custom category matching: dot-prefix fallback for categories not in DEFAULT_TAXONOMY so preview matches real DSR runner behavior - Add test_policy_filter.py (10 tests including custom category coverage) - Add 6 builder tests (fallback edges, static detail, warnings, empty graph, target categories, skip_processing rename) - Add 2 reachability tests (REQUIRES_MANUAL_IDENTITY, multi-dataset wins) - Rename misleading test to test_graph_excludes_skip_processing_collections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…eachable nodes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drops the IntegrationDetailPanel SCSS module entirely: the ``Manual review required`` blocks now render as warning Alerts instead of hand-rolled div + badge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the custom centered secondary-text empty state with Ant's ``<Empty>`` component, which renders the standard illustration plus the prompt to pick a property. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the ``via {stage_via}`` line into the header label column so it
shares the logo's left offset, top-align the logo with the first text
row, and drop SCSS rules now covered by ``<Text type=\"secondary\">``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned Files
|
Regenerated TS types now that the fidesplus traversal-preview endpoint returns typed Pydantic models. Adds the 15 transitive generated schemas plus barrel re-exports, and reduces the feature's types.ts to thin wrappers: re-exports for 1:1 shapes, Omit-extends for the React Flow ``Data`` wrappers and for list fields that Pydantic ``default_factory=list`` emits as optional in OpenAPI but always populates at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Update the schema, generated types, mock data, and panel component to support multiple data uses per system in the DSR traversal preview. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket ENG-3767
Description Of Changes
Companion PR: fidesplus#3572
Lands the DSR traversal visualizer in
mainfor the first time.About the diff size. The PoC originated as a fast-moving, largely AI-generated branch (#8099) that wasn't given a substantive code review before it landed on its source branch. The branch on this PR includes both the cherry-pick of that PoC and a polish pass that applies our frontend conventions and addresses a first review pass before merge. The intent is that what you're reviewing is the version of the feature we'd actually want to ship — not the PoC's raw state.
Diffing this PR against the PoC will show substantive differences: string-union types replaced with enums, default exports flipped to named, hex literals swapped for palette tokens, antd v6 deprecations fixed, RTK Query cache key bug fixed, hydration-snap on first paint fixed, CodeQL finding on hand-built routes fixed, and more. Please review the final state rather than trying to diff-the-diff. The polish commit history is intact if you want to see what was changed and why.
Feature summary. A property-scoped, action-typed preview of how a DSR will fan out through configured integrations. Renders an interactive four-lane graph (identity input → systems queried → manual review gates → not touched) with per-card metadata, dataset breakdowns, and dependency edges. Gated behind the new
dsrTraversalVisualizerbeta flag (dev + test on, production off) and reachable at/dsr-traversal/[propertyKey]/[[...actionType]]from the "Request workflows" item in the Privacy requests nav group.Code Changes
Backend (
src/fides/api/graph/preview/)builder.py—TraversalPreviewBuilderbuilds the structured preview, falling back to FK-derived edges when full traversal failsreachability.py— per-integration reachability classificationpolicy_filter.py— policy-aware data-category filtering with descendant matchingschemas.py— Pydantic response shapestests/ops/graph/preview/— builder + reachability tests with sharedconftest.pyfixturesFrontend (
clients/admin-ui/src/features/dsr-traversal-visualizer/)layout/compute-lane-layout.ts,compute-stages.ts,compute-column-count.ts) — pure functions with unit testsserializeQueryArgsthat stripsrefreshso regenerate updates the same cache entryuseLaneCollapseState(persists collapse prefs tolocalStorage, exposes ahydratedflag to gate first-paint and avoid layout snap),useNodeSelection,useTraversalGraphclients/admin-ui/src/pages/dsr-traversal/ActionType,Reachability,ActionStatus,LaneId;REACHABILITY_COLORtyped againstCUSTOM_TAG_COLORMSW handlers (
clients/admin-ui/src/mocks/dsr-traversal/)npm run dev:mockwithout seeding the fidesplus DBFlag + nav wiring
dsrTraversalVisualizerflag inflags.json(dev + test on, production off)nav-config.tsxentry under "Privacy requests → Request workflows", gated by the flag +requiresPlusSteps to Confirm
The visualizer is gated behind the
dsrTraversalVisualizerflag (on by default in dev/test, off in production).traversal-preview?refresh=truenetwork call fires and writes back into the same cache entry (no parallel cache entries).localStoragekeyfides:dsr-traversal:lane-collapse:v1); on reload the canvas renders with the persisted layout from the first paint (no visible re-layout snap).GET /api/v1/plus/properties/{property_id}/traversal-preview?action_type=access&include_unreachable=trueand confirm the response matches theTraversalPreviewschema; passrefresh=trueto bypass cache.Known Follow-ups (deferred)
From an in-session
/code-reviewpass, these were judged worth deferring rather than blocking merge:transformResponsein a follow-up — same pattern affects other features in the repo.compute-stagesBFS depth-tracking deserves a diamond-with-extra-hop regression test. Practical case appears to be correct; a test would lock it in.Pre-Merge Checklist
CHANGELOG.mdupdatedmaindowngrade()migration is correct and works🤖 Generated with Claude Code