Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Pipeline Graph (DAG) Visualization #50
It is useful to have a visual representation of the Directed Acyclic Graph (DAG) that is produced during the execution of a pipeline.
In the graph,
The graph exists in the store under the path
A function jsonifyGraph is also exported. This is because the graph object from graph.js is not serializable. This creates a serializable JSON representation of the graph.
See here how the collection (aka DAG) is logged out during task resolution for debug.
A first implementation of this could be to write the JSON graph to disk during the pipeline execution, overwriting the previous file whenever a
Then it is a matter of parsing that JSON into a visualization using something like d3.
Suggestions to improve the way the graph is handled within watermill are welcome. Perhaps there is a better serializable format to use (e.g. graphml format).
referenced this issue
May 30, 2017
See the current JSON representation here. Note the duplicated logging of nodes if they are children of other nodes. This JSON graph structure is probably not ideal, or at least, should be created from another structure of
It could be useful to use a more standard graph format. The most ideal would be:
Even OBO could work: this could have nodes for a file, which might have an edge "created_by" and "used_by" to different task nodes