Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
New GUI: scalable navigable matrix (quilt) view #52
It has become clear that many users run complex suites with an inordinate number of tasks for dependency visualisation, which is currently only manifest with our 'graph' (node-link) view & in such cases exhibits as an illegible muddle of intersecting edges.
Accordingly, I have been browsing the literature on visualisations for directed acyclic graphs to see if there are alternatives which are suited better to workflows containing (very) large numbers of tasks.  is a very good summary of different approaches.
In light of the above, I would like to tentatively propose a new 'view' i.e. visualisation mode for suites, to complement the 'graph', 'dot' & 'text' task views we will want to re-establish in improved form for our new GUI (https://github.com/cylc/cylc/issues/1873), either as part of its initial development, or as a later enhancement.
The underlying concept would be (dynamic) adjacency matrix depictions of (D)AGs, which are essentially boolean arrays for edge existence between nodes. Research shows that for layered graphs with more than ~20 nodes, matrix representations are consistently clearer, except in some cases for path-finding .
However, this can yet be improved on  with an adapted, condensed form denoted a 'quilt' . "Quilts
Obviously the details of this idea need fleshing out, however the matrix-based underlying nature of this view lends itself intuitively to NumPy functionality. I will conduct some investigation into spatial requirements, scalability with 'skip links' & complications/subtleties I may have neglected as yet.
changed the title
Future GUI: scalable navigable additional view using quilts
Aug 14, 2018
A good idea, I haven't come across adjacency matrices, they are a bit intimidating at first but pretty easy to read once you understand what they are representing.
Having a quick think about how this view might be used...
The current views are task-orientated, this view is more dependency orientated as it is effectively a grid of dependencies with tasks as the axis. This makes it a natural complement to our non-graph views which don't display any dependency information. Adjacency matrices for directed graphs seem to put prereqs in one colour and postreqs in another, we could extend this to show whether a dependency has been satisfied. Such a view could:
Ah, that's a lovely post for a quick overview! Good find @hjoliver.
Apologies for perhaps over-referencing & for referencing rather heavy papers instead of introductory material which was probably more appropriate for context in my opening comment. In reply to your comment @oliver-sanders:
I'll continue to probe this idea in spare moments. A good next step will be taking a highly complex, e.g. a Met Office operational, suite & seeing how it translates into a matrix.
Displaying the whole matrix might be difficult for suites with many tasks, for example the MO global operational suite has ~2000 tasks, the matrix would be massive. I'm not quite sure what users would gain from visualising the whole matrix, it would be far too big to represent?
What might be a good idea is to display a smaller matrix consisting of the selected task and all other tasks within "n" nodes from that task in the graph (similar to what we are planning on doing with the graph view).
This means that all of the tasks in the matrix are closely linked in the graph keeping the view focused to the users activity, e.g. assessing the impact to the workflow of a failed task (which is the most significant use case I have thought of so far).
The concept of an adjacency matrix in itself wouldn't especially help for 'jumbo' suites. This is where the quilt concept comes in. That is not to say the overall view will not have it's limits, of course it will, but I want to demonstrate that there is distinct value supplementary to existing views.
I have jotted down some diagrams & done a few back-of-the-envelope calculations (using your n=2000 base figure) to illustrate this. I'm attaching the diagram now for convenience but will leave the description until tomorrow (it's bedtime).
Sorry, forgot to provide the description to those diagrams; I will get round to it eventually i.e. once I have located the paper I did my calculations on