Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

static graph visualization #82

Open
hjoliver opened this issue May 27, 2019 · 9 comments

Comments

@hjoliver
Copy link
Contributor

commented May 27, 2019

We probably want to replace cylc graph with web technology too, not just the control GUI graph view.

@hjoliver

This comment has been minimized.

Copy link
Contributor Author

commented May 27, 2019

This graph demo (found by @MartinRyan ) allows manual node repositioning (as many do, I think): https://cytoscape.org/cytoscape.js-euler/. Could be useful when automatic layout fails us.

@MartinRyan

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

Cytoscape looks great, it has a dagre extension, and a navigator 'bird's eye view and pan control' extension (mini-map). It consumes JSON though there are convertors for graph objects.

@hjoliver

This comment has been minimized.

Copy link
Contributor Author

commented May 27, 2019

Also from @MartinRyan py2cytoscape might be useful for static vis. (Cytoscape itself might be useful for the live control GUI as well)

@oliver-sanders oliver-sanders referenced this issue May 28, 2019
0 of 14 tasks complete
@oliver-sanders

This comment has been minimized.

Copy link
Member

commented May 28, 2019

This is an interesting challenge because:

  • Users typically run cylc graph to visualise a suite whilst developing the working copy.
  • The UI-Server is concerned with the installed copy (in the cylc-run directory).

The UI-Server is only concerned with the installed copy and is not necessarily able to see the working copy.

It is OK to require users to register a suite for cylc graph functionality (as the suite wouldn't appear in the web UI otherwise) but we wouldn't want users to have to repeatedly re-install a suite in order to visualise it.

@hjoliver

This comment has been minimized.

Copy link
Contributor Author

commented May 28, 2019

Well, yes and no.

If you're working on the working copy then you are able to install it repeatedly pretty easily, and if the payoff is kick-ass web base viz that seems a small price to pay, And besides you currently have to do that anyway if your suite.rc takes Jinja2 inputs from rose-suite.conf (it won't validate or graph without being installed first).

Also what's the alternative? We don't want to keep a GTK-based static graph GUI. An electron app might allow us to kill both birds with one stone, I guess.

@matthewrmshin

This comment has been minimized.

Copy link
Member

commented May 28, 2019

Assuming that we have done enough logical migration from rose suite-run to Cylc in the coming months, then we should have enough logic in Cylc to deal with the rose-suite.conf stuffs. The web UI can have an local file system browser so users should be able to open a suite from its source location. Otherwise, we can also build support for the UI to search for named suites from various configurable locations on the file system. (E.g. If a suite cannot be found under ~/cylc-run/SUITE/ then look at ~/roses/SUITE/.)

@oliver-sanders

This comment has been minimized.

Copy link
Member

commented May 28, 2019

If you're working on the working copy then you are able to install it repeatedly pretty easily

But what if the suite is running?

Also this is the situation we have at the moment, I advise people to do this:

$ rose suite-run -l; cylc graph mysuite

Unfortunately this causes confusion and is a barrier to usage. As a result very few people here use essential Cylc utilities such as:

  • cylc graph
  • cylc get-config
  • cylc view --jinja2

I wrote this page to try and break down the barrier but sadly it has made little difference.

I still regularly see users trying is running the entire suite to test changes.

Also what's the alternative?

No nice solution:

  • An electron app if it's quick and simple.
  • The UI Server could attempt to access the working copy (via ~/cylc-run/<suite>/.service/source)
    • If the working copy is not available it could fallback to the installed copy and present a warning.
  • Require the user repeatably re-installs the suite during development.
@hjoliver

This comment has been minimized.

Copy link
Contributor Author

commented May 28, 2019

Fair point on barriers to easy use. So let's do better than that if we can. We do need a non-PyGTK static vis tool with cylc-8 though, one way or another.

I still regularly see users trying is running the entire suite to test changes.

Same, and even worse, running an entire beast of a suite repeatedly to test how some cylc feature works, when a 5-line 1-task test suite would do. You can lead a horse to water, but you can't make it change its spots. Or something.

@kinow

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

See comments from (now closed) #81 about graphs in Cylc 8:

@hjoliver

A Vue component to draw directed acyclic graphs in the browser leveraging dagre-d3v4 (https://github.com/smbolton/dagre-d3v4) - https://github.com/vdidonato/vue-dagre-d3v4 - seems to be broken.

#81 (comment)

@oliver-sanders

I'm currently looking at Javascript libraries for graphing, I'll update this issue with conclusions as I investigate. I'll probably put up my experiments as a GitHub repo when more complete.

Graphing consists of two components:

The Layout Engine

The bit which arranges nodes / edges spatially. See the layout engines supported by cytoscape for examples.

  • Directed hierarchical graphs are hard
  • Layout is slow, there isn't a whole lot of choice when it comes to layout engines. Dagre is the standout but is a fairly stagnant project.
  • More arbitrary layouts are easy
  • We might want to automatically swap to a less ideal layout engine for larger graphs.

Requirements

  1. Directed & hierarchical
  2. Must handle edges spanning multiple levels
  3. Width of node considered in graph (i.e. nodes cannot overlap)
  4. Speed!

Results

Engine 1 2 3 4 Supported Engines
Dagre 🆗 🆗 🆗 4.1s cytoscape, d3
Klay 🆗 ⛅️ 🚫 6.4s cytoscape, d3
D3-DAG 🆗 🆗 🚫 [2] >> 60s [3] None? [4]

[1] Benchmark using the first cycle of MO global opp suite
[2] Possibly using the separation function though I can't get this to work
[3] Tested "Vert", "Greedy" and "Spread" layout methods
[4] Visualisation constructed by user provided functions, i.e. user provides the engine

The Graph Engine

The bit which renders the graph, provides an interface for manipulating and interacting with it. This is the component we will end up working with / extending to form the graph view.

Requirements

  1. Mobile friendliness
  2. Hackability of the graph:
    • Job status indicators (require more than just text labels)
    • Sub-workflows?
    • Advanced collapsing (e.g. parametrisations using an ellipsis symbol)?
  3. Not a massive RAM hog
  4. Support for the desired layout engine(s)
  5. Node / family collapsing

Results

Engine 1 2 3 4 5
cytoscape 🆗 Dagre, Klay
d3 Dagre, Klay
vis.js
arbor

Hybrids

Systems where the graph engine and layout engine are integrated:

Results

Engine L1 L2 L3 L4 G1 G2 G3 G4 G5
viz.js 🆗 🆗 🆗 🆗

#81 (comment)

@hjoliver

Interactive arbitrary grouping: select a bunch of nodes (individually, or by drawing a rectangle around them) and group them into a single node.
Layout algorithms for visualizing directed acyclic graphs - https://github.com/erikbrinkman/d3-dag - new-ish? demo https://jsfiddle.net/ye2xanf9/12/

#81 (comment)
#81 (comment)

@kinow kinow added this to the 0.2 milestone Sep 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.