New experimental asset graph improvements

[braunjj]

We're working on a few new enhancements to the asset graph that we're aiming to ship in the next major release. You can enable these from the Experimental Features toggle in your user settings in Dagster 1.4.15 or later.

Faster DAG rendering:

We're experimenting with a new layout algorithm that renders large graphs more quickly.

New Asset sidebar explorer

The new sidebar for the lineage graph renders a flattened view of the entire asset lineage tree. You can use this new sidebar to quickly jump to a specific asset, navigate to different parts of the graph, and even filter the graph by selecting "show downstream" or "show upstream" from the dropdown menu.

Asset groups are now collapsable

You can now expand and collapse any group in the graph

graph-demo.mp4

Give it a try and let us know you think!

[malow106]

Hey, I love the idea ! I think I will use it often :)
On my side the two buttons for selecting up/dowstream dependencies don't work. The generated expression "T024D"* seems to not consider key prefix. In my case the correct expression should be : "ETL/T024D"*

[jasonicarter]

Love it! And much easier to navigate for our dbt project than the normal docs, plus being able to select/query up and down stream dependencies and then materialize them is perfect.

[stevenmurphy12]

It's looking good, I noticed the changes being referred to in the 1.5.9 release. A couple of thoughts:

With the experimental experience being enabled, I see roughly the same thing being displayed twice: code location, which can then be expanded to show asset groups.

A 'folder' icon represents a code location, which is helpful for navigation and organisational purposes. A 'group of groups' if you will.

My code is split into multiple code locations in order to get this organisation, but it introduces more overhead in the background as a result. I'm having to create many code locations when I don't really need it, apart from the UI organisation it offers. Would it be possible to add a feature by which we define our own 'group of groups', as opposed to relying on using code locations to achieve this?

(more detail can be found here - #17949)

[braunjj]

Thanks for the feedback @stevenmurphy12! We are actively exploring improvements to our grouping concepts and we are planning to make asset groups hierarchical so you will be able to create groups of groups. This will ship as a separate workstream from these asset graph improvements though.

We are also planning some changes to the information architecture and nav to eliminate the double sidebar you show in your screenshot.

[stevenmurphy12]

Appreciate the info, thank you 👍

[EtienneT]

I like the faster rendering speed, but what I liked about the older graph was that you could see the whole graph all at once, viewing the state of the materializations. Now you have to open the groups one by one to view it. A simple expand/collapse all would solve this problem. Also it would be nice to retain the collapsed/expanded state somehow (cookie, localStorage or indexedDb in the browser or in the dagster db) so that the next time you open the asset graph, you don't have to adjust those.

[braunjj]

Thanks for the feedback @EtienneT. There is actually an expand/collapse all button down in the bottom right, but we can do more to make this more clear. Storing the group state in localstorage is a great idea too. Expect to see that added soon.

Lastly we're planning to add status summaries to each group node. So in the future, you'll be able to see the overall status of each asset without having to click into each group (see the screenshot below).

Hope this helps!

[EtienneT]

Excellent! I did not see the expand all button. Works great. One quick comment, when clicking expand all it would be nice to adjust the zoom level automatically to show the whole graph.

[tacastillo]

Hello! Just here to provide an anecdote. TIL about the "collapse/expand all" button.

[EtienneT]

The scroll wheel of the mouse should control the zoom level just like in google maps.

[salazarm]

Hi EtienneT, this used to be the default behavior but we found that zooming is not as common as panning around so to zoom you should hold shift while scrolling the wheel of the mouse.