Display dependency graph using DagreD3 #1817

Martchus · 2018-10-02T15:33:08Z

See https://progress.opensuse.org/issues/40772
Not fancy yet but a start

codecov · 2018-10-02T15:55:03Z

Codecov Report

Merging #1817 into master will decrease coverage by 0.14%.
The diff coverage is 82.92%.

@@            Coverage Diff             @@
##           master    #1817      +/-   ##
==========================================
- Coverage   90.36%   90.22%   -0.15%     
==========================================
  Files         139      139              
  Lines        9937     9978      +41     
==========================================
+ Hits         8980     9003      +23     
- Misses        957      975      +18

Impacted Files	Coverage Δ
lib/OpenQA/Schema/Result/Jobs.pm	`92.54% <100%> (+0.03%)`	⬆️
lib/OpenQA/WebAPI.pm	`93.93% <100%> (+0.02%)`	⬆️
lib/OpenQA/WebAPI/Controller/Test.pm	`95.59% <80.55%> (-2.1%)`	⬇️
lib/OpenQA/Worker/Commands.pm	`85.89% <0%> (-2.57%)`	⬇️
lib/OpenQA/Scheduler/Scheduler.pm	`86.3% <0%> (-2.29%)`	⬇️
lib/OpenQA/WebSockets/Server.pm	`90.36% <0%> (-1.84%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0a05f9...693829b. Read the comment docs.

coolo · 2018-10-02T17:58:03Z

/tests/2068098#settings looks like it can be better :)

coolo · 2018-10-03T08:48:32Z

I know this WIP, but experimented a little with it:

it needs to be in a tab of its own. we have pretty large clusters. Appending this to the huge settings table is no longer fitting. I guess this tab could either disappear or be grayed out for dependencyless jobs
the current parents/children can disappear then
the NAME is too noisy, especially in large clusters
the parallel clusters need to be a little more obvious
job status/result needs to be part of the display
the dots need to link back to the job

coolo · 2018-10-03T09:47:22Z

This might be better than what we have now, but I consider it still suboptimal (this manually drafted javascript to experiment with vis.js options

coolo · 2018-10-03T09:55:00Z

For the interested: http://stephan.kulow.org/hpc.html

coolo · 2018-10-03T10:44:25Z

Possibly a network is just the wrong way to think about it. Random links:

https://mermaidjs.github.io/flowchart.html
https://resources.jointjs.com/demos/layout

coolo · 2018-10-03T11:07:14Z

Playing with mermaidjs:

Martchus · 2018-10-03T14:52:45Z

Possibly a network is just the wrong way to think about it.

I had the same thought when testing vis.js.

The mermaidjs example looks good. I'll change the PR to use that library. Shouldn't be much work if their API is similar. @coolo Do you have already code or is the example manually drafted?

coolo · 2018-10-03T15:01:11Z

Their API is completely different (and IMO sucks): http://stephan.kulow.org/mermaid.html

But if we render it like the above, I have the impression we can just as well do it on our own (and better suited to our problem).

So some more links:
https://www.codeproject.com/Articles/16192/Graphic-JavaScript-Tree-with-Layout
https://dagrejs.github.io/project/dagre-d3/latest/demo/clusters.html

coolo · 2018-10-03T15:09:28Z

DagreD3 is my favorite for now - e.g. it allows us to specify the parallel clusters as tables and then layout it within the graph: https://dagrejs.github.io/project/dagre-d3/latest/demo/dom.html

Martchus · 2018-10-04T14:16:43Z

@coolo I think I've got the algo to compute the clusters. At least the results look plausible :-)

coolo · 2018-10-04T14:49:05Z

And 2068098 ? :)

Martchus · 2018-10-04T14:55:18Z

I only have 2041334 because my database is a little bit older:

BTW: I really need to make the boxes equally sized.

coolo · 2018-10-04T16:41:41Z

This leaves "the current parents/children can disappear then" - the rest works great.

Martchus · 2018-10-04T18:25:40Z

And tests are missing or at least need to be adapted.

Martchus · 2018-10-05T11:20:13Z

The reason why UI test are failing:

Martchus · 2018-10-05T15:04:26Z

I added/fixed tests and they pass now. The old table has been removed, too.

coolo

just nitpicks

coolo · 2018-10-06T05:54:42Z

assets/assetpack.def

@@ -100,6 +100,13 @@
 << https://raw.githubusercontent.com/twbs/bootstrap/v4.1.1/scss/_print.scss
 < https://raw.githubusercontent.com/sorich87/bootstrap-tour/6a1028fb562f9aa68c451f0901f8cfeb43cad140/build/css/bootstrap-tour.css

+! d3.js


[nitpick] It's not just d3

changed to dagre-d3.js

coolo · 2018-10-06T05:56:43Z

assets/javascripts/test_result.js

  if (window.location.hash === '#live') {
    resumeLiveView();
  }
+  // setup dependency graph if it's the default tab


these blocks look pretty much repeated

coolo · 2018-10-06T05:59:51Z

assets/javascripts/test_result.js

+            tooltipText: node.tooltipText,
+            testResultId: testResultId,
+            testResultName: testResultName,
+            fill: "#afa",


I'm a little worried to have a colour defined outside of scss. Perhaps this can be inherited from some (otherwise invisible) css?

This fill is overridden in CSS anyways. I moved the other fill (which was actually used) to CSS.

coolo · 2018-10-06T06:01:56Z

lib/OpenQA/WebAPI.pm

@@ -130,6 +130,7 @@ sub startup {
    $test_r->get('/')->name('test')->to('test#show');
    $test_r->get('/ajax')->name('job_next_previous_ajax')->to('test#job_next_previous_ajax');
    $test_r->get('/modules/:moduleid/fails')->name('test_module_fails')->to('test#module_fails');
+    $test_r->get('dependencies')->name('test_dependencies')->to('test#dependencies');


for consistency either have all $test_r->get calls with / or none.

* See https://progress.opensuse.org/issues/40772 * Not fancy yet but a start

* Show result/status in dependency graph * Highlight current job * Make the labels links

* Make boxes equally sized * Do text eliding in CSS * Mark current job more visible

* Remove comment in SCSS which would be rendered into CSS * Don't include fonts twice

* Fix duplicated code * Define all colors in CSS * Rename asset (d3.js => dagre-d3.js)

Martchus force-pushed the dependency_graph branch from 3aa2960 to 2fcb63a Compare October 4, 2018 14:14

Martchus changed the title ~~WIP: Display dependency graph using vis.js~~ WIP: Display dependency graph using DagreD3 Oct 4, 2018

Martchus force-pushed the dependency_graph branch from f149818 to 1c5e7b4 Compare October 5, 2018 09:43

Martchus changed the title ~~WIP: Display dependency graph using DagreD3~~ Display dependency graph using DagreD3 Oct 5, 2018

coolo approved these changes Oct 6, 2018

View reviewed changes

Martchus added 8 commits October 8, 2018 10:17

Display dependency graph using dagre-d3

da2c32c

* See https://progress.opensuse.org/issues/40772 * Not fancy yet but a start

Make dependency graph a bit more fancy

d8223f2

* Show result/status in dependency graph * Highlight current job * Make the labels links

Fix clipping and alignment of dependency graph

5432df1

Compute cluster for dependency graph

3802afa

Improve appearance of dependency graph

def4d37

* Make boxes equally sized * Do text eliding in CSS * Mark current job more visible

Test dependency graph

d03be31

Remove old dependency table

9f6e4b2

Rename part_of_cluster to has_dependencies

d1d89fe

Martchus added 3 commits October 8, 2018 10:17

Add help text to dependency graph

1520805

Ensure filter icon and last tab button don't overlap

b9cda9b

Clean CSS

91ace9e

* Remove comment in SCSS which would be rendered into CSS * Don't include fonts twice

Martchus force-pushed the dependency_graph branch from 314afec to 91ace9e Compare October 8, 2018 08:17

Polish code for dependency graph

693829b

* Fix duplicated code * Define all colors in CSS * Rename asset (d3.js => dagre-d3.js)

Martchus force-pushed the dependency_graph branch from 1c82219 to 693829b Compare October 8, 2018 08:42

coolo merged commit f526e2b into os-autoinst:master Oct 8, 2018

Martchus deleted the dependency_graph branch October 8, 2018 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Display dependency graph using DagreD3 #1817

Display dependency graph using DagreD3 #1817

Martchus commented Oct 2, 2018

codecov bot commented Oct 2, 2018 •

edited

Loading

coolo commented Oct 2, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

Martchus commented Oct 3, 2018 •

edited

Loading

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

Martchus commented Oct 4, 2018

coolo commented Oct 4, 2018

Martchus commented Oct 4, 2018 •

edited

Loading

coolo commented Oct 4, 2018

Martchus commented Oct 4, 2018

Martchus commented Oct 5, 2018 •

edited

Loading

Martchus commented Oct 5, 2018

coolo left a comment

coolo Oct 6, 2018

Martchus Oct 8, 2018

coolo Oct 6, 2018

Martchus Oct 8, 2018

coolo Oct 6, 2018

Martchus Oct 8, 2018

coolo Oct 6, 2018

Martchus Oct 8, 2018

Display dependency graph using DagreD3 #1817

Display dependency graph using DagreD3 #1817

Conversation

Martchus commented Oct 2, 2018

codecov bot commented Oct 2, 2018 • edited Loading

Codecov Report

coolo commented Oct 2, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

Martchus commented Oct 3, 2018 • edited Loading

coolo commented Oct 3, 2018

coolo commented Oct 3, 2018

Martchus commented Oct 4, 2018

coolo commented Oct 4, 2018

Martchus commented Oct 4, 2018 • edited Loading

coolo commented Oct 4, 2018

Martchus commented Oct 4, 2018

Martchus commented Oct 5, 2018 • edited Loading

Martchus commented Oct 5, 2018

coolo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 2, 2018 •

edited

Loading

Martchus commented Oct 3, 2018 •

edited

Loading

Martchus commented Oct 4, 2018 •

edited

Loading

Martchus commented Oct 5, 2018 •

edited

Loading