Visualize task graphs using ipycytoscape

[bnaul]

Inspired by dask/dask#9091; the same reasons given there make sense to me here as well:

1. The ability to run in pyodide without compiling graphviz to WASM (cf. Dask in pyodide dask/dask#9053)
2. More interactivity, especially with different coloring schemes and layouts.
3. The possibility to add richer tooltips/popups (cf. Clearer colourmap / labels when showing task order in visualize dask/dask#9069)

I actually don't know if Orion has a .visualize implementation yet, could be easier to add it there if starting from scratch?

[zanieb]

This looks really cool!

We don't have any visualize implementation in v2 yet. Since the flow is discovered dynamically it'll be quite the undertaking :) We should be able to do local visualizations after a run easily though.

It's very unlikely that we'd take this on for v1.

[bnaul]

Yeah it seems like a whole lot of work unless you can hijack the existing dask visualize functionality somehow, which seems doable to me...?

Since the flow is discovered dynamically

I know this is true in the general case but it seems like there are still cases where the flow is all declared statically, right? I know at least I can't easily break out of the DAG mindset so even my Orion flows still have all their tasks declared up front. So maybe "visualize whatever you already know about" would at least partly cover both cases?

[zanieb]

Yeah it seems like a whole lot of work unless you can hijack the existing dask visualize functionality somehow, which seems doable to me...?

For v2? That could be interesting. We are planning on dropping dask as a top-level dependency though.

I know this is true in the general case but it seems like there are still cases where the flow is all declared statically, right?

Yep, that's the case we intend to address first. We'll have to run the flow function with all of the actual execution mocked to build the graph. We're also planning to explore walking the AST to capture dynamic flows.

[zareami10]

I would also find this very useful.

Also see the Kedro case for a very well done example of pipeline visualization with interactive graphs.

[felix-ht]

I love this idea as well.

It would even be better to have it integrated into orion (with https://js.cytoscape.org/) - i really dislike the current radar view.

[github-actions]

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

[github-actions]

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.