Add catalog graph plugin #7185

Fox32 · 2021-09-15T08:59:19Z

Welcome to the catalog graph plugin! The catalog graph visualizes the relations
between entities, like ownership, grouping or API relationships.

The plugin comes with these features:

EntityCatalogGraphCard:
A card that displays the directly related entities to the current entity.
This card is for use on the entity page.
The card can be customized, for example filtering for specific relations.
CatalogGraphPage:
A standalone page that can be added to your application providing a viewer for your entities and their relations.
The viewer can be used to navigate through the entities and filter for specific relations.
You can access it from the EntityCatalogGraphCard.
EntityRelationsGraph:
A react component that can be used to build own customized entity relation graphs.

It's similar to the system diagram and requests like #1204 and is based on suggestions from here.

Quick Demo

Entity Card:

CatalogGraphCard.mov

Standalone Viewer:

CatalogGraphStandaloneViewerV2.mov

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

changeset-bot · 2021-09-15T08:59:22Z

⚠️ No Changeset found

Latest commit: 0758089

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Fox32 · 2021-09-15T09:00:57Z

plugins/catalog-graph/package.json

+    "@backstage/theme": "^0.2.10",
+    "@material-ui/core": "^4.12.2",
+    "@material-ui/icons": "^4.9.1",
+    "@material-ui/lab": "4.0.0-alpha.47",


This version is different from what we use elsewhere. I had some trouble with the autocomplete that is resolved here. We could update to this version everywhere, I don't think that this bump included any breaking changes. On the other side we could just sit it out and wait for #7094 that includes the autocomplete.

Actual, the build seems to fail because my version is to new #7196

freben · 2021-09-15T09:17:13Z

freben · 2021-09-15T09:53:17Z

The ownedBy graphs showing in the video tell me that there's room for a future addition, that lets you toggle making relations into nodes too. Maybe with a different color. (A) -> (ownedBy / ownerOf) -> (B) and (C) and (D)

gets the fanout a bit under control and clarifies things when there are 8 relations of one type and three of another, etc

Fox32 · 2021-09-15T10:58:12Z

The ownedBy graphs showing in the video tell me that there's room for a future addition, that lets you toggle making relations into nodes too. Maybe with a different color. (A) -> (ownedBy / ownerOf) -> (B) and (C) and (D)

Yeah, the idea is to have more rendering modes over time that everyone can choose how he want the display it. I already played around with toggling. The bad thing about it is that the graph can quite drastically change when you "open" a relation. That felt quite hard to use and keep track of the context then. The idea of grouping multiple entities into a single node is neat too 🤔

freben · 2021-09-15T11:20:35Z

Ah but, even without the opening and closing. Even as a static thing, replacing arrows with arrow-node-arrow

freben · 2021-09-15T11:21:42Z

And I didn't mean to collapse entities actually, even though that sounds interesting :) I was just not gonna make a nicer ascii graph, I meant three arrows to B, C, D as three separate nodes :)

Fox32 · 2021-09-15T15:00:36Z

Ah but, even without the opening and closing. Even as a static thing, replacing arrows with arrow-node-arrow

Ah, yeah. Also played around with that idea. Can't show a screenshot (to much internal stuff on it), but I had two issues:

The Dependency Graph component has some trouble layouting everything with the additional nodes. But that could be solved with another renderer
I wanted to collapse multiple relations into one relation node, but there you need an intelligent heuristic to decide which to merge and which not (because they can come from both sides…) The problem is easier to see than to explain 😆

adamdmharvey · 2021-09-16T01:11:55Z

Just wanted to stop in to say: well done, @Fox32 ! Beautiful rendering, and wonderful feature to drive up discoverability of the engineering output. This drives such value. Kudos!!!

andrewthauer · 2021-09-16T10:41:55Z

🎉 Amazing! Can't wait to try this out!

julioz · 2021-09-16T12:53:33Z

I like this a lot, we have implemented something similar internally. I would love to give it a try in our backstage instance 👍

What has been very valuable for us is to also expose "metrics" out of the dependency graph to spot "non-standard component relationships", for example, suggesting to break down a component's responsibility into multiple sub-components when we detect > X dependsOn (where X is an arbitrary number).
On top of that, the filtering capability has been useful (when paired with component annotations) for us to also identify "component groups", for example to answer the question "show me all components that expose at least one route to the public internet" or "show me all components that connect to this specific database" (and thus identify integration via DB instead of via API), etc.

If I can propose (yet another 😆) feature idea that we also have internally, is the capability to "simulate edition" of the graph. An engineer can then more easily sketch ideas of how the "system landscape" should be. At the end of the process, we allow them to exporting sketches to common graph description languages (like dot or mermaid).

regicsolutions · 2021-09-16T15:56:04Z

Love it!

freben · 2021-09-16T16:06:48Z

Also, can't wait to combine this with like symbology or similar to give status overviews at a glance. For example, looking at your local "neighborhood" of nodes and seeing that one of your dependencies has a big warning triangle on it because it isn't updating properly or has some kind of alert going

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

Fox32 · 2021-09-17T13:21:01Z

CI is green now 🙂

freben

NICE!

.changeset/wicked-pugs-speak.md

freben · 2021-09-17T12:04:07Z

packages/app/src/App.tsx

+              RELATION_DEPENDS_ON,
+              RELATION_DEPENDENCY_OF,
+            ],
+          }}


Would it make sense to have a defaults mechanism (kinda like the EntityTable columns or something like that)? This is a bit much to have to slap into the example app :) So if you don't give a selectedKinds it'll be CatalogGraphPage.defaultSelectedKinds or so

Hmm, it's a bit complicated here. Not providing selectedKinds means "all selected kind" or "no kinds filter". So I can't set a default for undefined (same for relations). On the other hand, setting these is just a matter of taste, we could also go with "no filter" in the example app and readme.

freben · 2021-09-17T12:06:33Z

packages/app/src/components/catalog/EntityPage.tsx

      <EntityAboutCard variant="gridItem" />
    </Grid>

+    <Grid item md={6} xs={12}>
+      <EntityCatalogGraphCard variant="gridItem" maxHeight={400} />


i saw some vertical jumping in the demo video; i still think these might be best to have with a fixed height, and possibly below the fold (under some other stuff) so it can load in peace and maybe even does its first render out of sight. That way it has a bit more vertical and horizontal space to play with too

Yeah, fixed height is better. About the jumping inside the diagram: I wonder if that is something we can fix in the dependency graph. There is a debounce that where the timeout could be reduced to make it more responsive. But maybe we have to switch to another library under the hood anyway, as dagre seems to be kind of dead.

@Fox32 do we have an issue for the switch?

packages/app/src/components/catalog/EntityPage.tsx

freben · 2021-09-17T12:30:50Z

plugins/catalog-graph/api-report.md

+  kinds,
+  relations,
+  direction,
+  maxHeight,


is it worth having something like cardStyles or whatnot, that has the proper css type or similar? Or even, taking it farther, essentially cardProps: InfoCardProps? This is one of those components that's such a thin wrapper, and immediately somebody comes along and asks "what if i want to change the title of the card". So, not to rant too much, but sometimes I'm not sure what the right "abstraction level" is when we make components, and perhaps especially so for cards

Yeah, I think a lot of people might want to customize the card. For now I hoped making the card a very tiny wrapper would be sufficient and people can just copy it. On the other hand, where do you stop. Would it be better to just pass all card properties through? What about the ones that I modify in my card, like deepLink or noPadding?

Yeah it's a complicated subject. I hope we'll get the time soon to work on clarifying the app-plugin contract so to speak

I worked with mui 5 lately for another project, maybe they have some patterns we can take over. Like having the xs property on all components.

plugins/catalog-graph/src/components/CatalogGraphPage/useCatalogGraphPage.test.ts

freben · 2021-09-17T13:07:18Z

plugins/catalog-graph/src/components/EntityRelationsGraph/types.ts

+  /**
+   * Optional kind of the entity.
+   */
+  kind?: string;


would type?: string fit well in here too?

especially if that together with kind could be used to draw custom icons in the future

It's not used right now, but yeah, could be added later.

We could also pass the full entity in and make the nodes more customizable. But that is maybe also something for later.

plugins/catalog-graph/src/components/EntityRelationsGraph/useEntityStore.ts

plugins/catalog-graph/src/plugin.ts

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

freben

Let's ship this!

dudarau · 2022-01-05T13:49:08Z

@freben @Fox32 do we need to get rid of dagre? Do we have a ticket for this? If you don't mind I can work on it.

Fox32 commented Sep 15, 2021

View reviewed changes

Fox32 marked this pull request as ready for review September 15, 2021 09:01

Fox32 requested a review from a team as a code owner September 15, 2021 09:01

Fox32 force-pushed the feat/catalog-graph branch 2 times, most recently from 624893d to d979a1a Compare September 15, 2021 09:26

Fox32 force-pushed the feat/catalog-graph branch from cd713b4 to 9cba3ea Compare September 15, 2021 15:05

Fox32 added 2 commits September 17, 2021 14:06

Add catalog graph plugin

8c50da6

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

Update api-report

3742f41

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

Fox32 force-pushed the feat/catalog-graph branch from 9cba3ea to 3742f41 Compare September 17, 2021 12:06

Upgrade to @material-ui/lab@4.0.0-alpha.57

aec9385

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

Fox32 force-pushed the feat/catalog-graph branch from c0e509f to aec9385 Compare September 17, 2021 12:58

freben reviewed Sep 17, 2021

View reviewed changes

Fox32 force-pushed the feat/catalog-graph branch 2 times, most recently from a8ab0d3 to ae0446c Compare September 20, 2021 11:19

Review comments

0758089

Signed-off-by: Oliver Sand <oliver.sand@sda-se.com>

Fox32 force-pushed the feat/catalog-graph branch from ae0446c to 0758089 Compare September 20, 2021 14:12

freben approved these changes Sep 21, 2021

View reviewed changes

Fox32 merged commit f4004cd into backstage:master Sep 21, 2021

Fox32 deleted the feat/catalog-graph branch September 21, 2021 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add catalog graph plugin #7185

Add catalog graph plugin #7185

Fox32 commented Sep 15, 2021

changeset-bot bot commented Sep 15, 2021 •

edited

Loading

Fox32 Sep 15, 2021

Fox32 Sep 15, 2021

freben commented Sep 15, 2021

freben commented Sep 15, 2021

Fox32 commented Sep 15, 2021

freben commented Sep 15, 2021

freben commented Sep 15, 2021

Fox32 commented Sep 15, 2021 •

edited

Loading

adamdmharvey commented Sep 16, 2021

andrewthauer commented Sep 16, 2021

julioz commented Sep 16, 2021

regicsolutions commented Sep 16, 2021

freben commented Sep 16, 2021

Fox32 commented Sep 17, 2021

freben left a comment

freben Sep 17, 2021

Fox32 Sep 20, 2021

freben Sep 17, 2021

Fox32 Sep 20, 2021 •

edited

Loading

dudarau Dec 30, 2021

freben Sep 17, 2021

Fox32 Sep 20, 2021

freben Sep 21, 2021

Fox32 Sep 21, 2021

freben Sep 17, 2021

freben Sep 17, 2021

Fox32 Sep 20, 2021

Fox32 Sep 20, 2021

freben left a comment

dudarau commented Jan 5, 2022

Add catalog graph plugin #7185

Add catalog graph plugin #7185

Conversation

Fox32 commented Sep 15, 2021

Quick Demo

✔️ Checklist

changeset-bot bot commented Sep 15, 2021 • edited Loading

⚠️ No Changeset found

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freben commented Sep 15, 2021

freben commented Sep 15, 2021

Fox32 commented Sep 15, 2021

freben commented Sep 15, 2021

freben commented Sep 15, 2021

Fox32 commented Sep 15, 2021 • edited Loading

adamdmharvey commented Sep 16, 2021

andrewthauer commented Sep 16, 2021

julioz commented Sep 16, 2021

regicsolutions commented Sep 16, 2021

freben commented Sep 16, 2021

Fox32 commented Sep 17, 2021

freben left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fox32 Sep 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freben left a comment

Choose a reason for hiding this comment

dudarau commented Jan 5, 2022

changeset-bot bot commented Sep 15, 2021 •

edited

Loading

Fox32 commented Sep 15, 2021 •

edited

Loading

Fox32 Sep 20, 2021 •

edited

Loading