Visualize task graphs using ipycytoscape

[ian-r-rose]

This adds a new graph visualization engine based on ipycytoscape. Right now it's pretty much a re-implementation of the graphviz visualizer, but with zooming and panning. However, there are a few things that this could be more useful for in the near future.

1. The ability to run in pyodide without compiling graphviz to WASM (cf. Dask in pyodide #9053)
2. More interactivity, especially with different coloring schemes and layouts.
3. The possibility to add richer tooltips/popups (cf. Clearer colourmap / labels when showing task order in visualize #9069)

• Closes Interactive alternatives to graphviz for .visualize() #7301
• Tests added / passed
• Passes pre-commit run --all-files

[jakirkham]

This is really cool! 🤩 Thanks for working on this Ian 🙏

[jakirkham]

Given dot is the CLI for GraphViz, wonder if we should put this in a different module for organizational clarity

[ian-r-rose]

Yeah, I waffled on whether to create a new module for this, but decided to keep them in the same module since they share some of the naming utilities.

What about changing this to dask/visualize.py (with perhaps a dot.py backwards compatibility shim)?

[jrbourbeau]

Thanks @ian-r-rose! This is cool to see. I'm not very familiar with the interactive-Jupyter-graph-visualization space -- are there other packages we should be considering?

Another benefit I of adding a cytoscape engine is it helps avoid some of the pain around installing the system graphviz package

[ian-r-rose]

are there other packages we should be considering?

Perhaps there are. I chose cytoscape for this primarily because (1) it is a well-supported JS library, (2) python bindings for it already exist through ipycytoscape, and (3) it already works in pyodide. But I wouldn't be averse to investigating others if they had additional benefits.

There is some discussion of alternatives in #7301, but cytoscape still seems like a reasonably safe choice.

[jakirkham]

The only alternative that stood out to me from reading that issue was ipyelk. Did we evaluate that one? If so, what conclusions did we draw?

Martin also mentioned a Google doc with other alternatives, but the link appears to be broken. Asked if there was a more up-to-date link

[ian-r-rose]

The only alternative that stood out to me from reading that issue was ipyelk. Did we evaluate that one? If so, what conclusions did we draw?

I didn't really evaluate it -- I looked at the screenshots and they looked less like the final result I wanted than the cytoscape ones.

[jakirkham]

That was my impression as well, but didn't know if that was just a perception gotten from those screenshots or if that is really representative of all ipyelk visualizations

[jacobtomlinson]

This is awesome. I love the implementation because it doesn't close us off from alternatives in the future. I also like that the default is configurable, really neat!

The only thing I would say this PR needs is some docs.

[ian-r-rose]

The only thing I would say this PR needs is some docs.

Good point, thanks @jacobtomlinson. I've added some docs.

[ian-r-rose]

That was my impression as well, but didn't know if that was just a perception gotten from those screenshots or if that is really representative of all ipyelk visualizations

I took a look at ipyelk for a few minutes and had a bit of a hard time getting it working (it has also not seen much activity recently). I feel comfortable moving forward with cytoscape for the time being. I also think that the plumbing work done to make the visualization engine configurable should make adding new visualizers easier in the future.

[jrbourbeau]

One last comment -- apologies for not noticing before

[jrbourbeau]

Thanks @ian-r-rose

[freyam]

Wow, this is incredibly awesome! @ian-r-rose 🚀