Ability to Configure Flows / DAG using NoFlo based JSON graph definition

[sirinath]

Master task is #11

Ability to configure the DAG using http://noflojs.org/documentation/json/ would be great addition. Then you can use the NoFlo UI Editor to define you program.

[sirinath]

Some similar projects:

• https://github.com/lmco/streamflow
• https://github.com/kuujo/vertigo

Which use UI builders.

[clockfly]

Thanks, @sirinath

We are looking into providing some ability to do visual programming.
What we want is a simple, and lightweight solution, which can be easliy embeded into this project, as we need to integrate the Graph with performance metrics in single UI(graph).

The look and feel of streamFlow UI(https://github.com/lmco/streamflow/wiki) is what'd like to have, but I am not sure whether it is too heavy to be used.

What is your opinion on these three? Do you know other alternatives that is much more lightweight?

[sirinath]

Have a look at https://github.com/noflo/noflo-ui and https://github.com/the-grid/the-graph

But please make it so you can embed the editor component in other application.

There could be better editors also.

A simple JSON configuration is very much like in NoFlo is desirable.

Also have a look at: http://www.yworks.com/ (Not open source through)

[sirinath]

Ideally this should be broken into parts:

1. Simple configuration object based set up using API
2. JSON based configuration to generate above object through the API (E.g. input from Web IDE)
3. GUI to generate the configuration to generate JSON

[kkasravi]

Thanks @sirinath. This is issue is also similar to #53. One will be marked a duplicate at some point.

Noflo's JSON schema consists of inports,outports,processes,connections. Processes is an interesting part since it describes what is required to deploy and run the TaskActor. It looks like connections is used to define graph edges and {inports, outports} are used to define graph nodes. It may be challenging to just include the UI editor since this is strongly associated with a flow based DSL. In general noFlo is based on FBP and is specific to a frontend development environment where their transpiler generates coffeescript. I'm a bit dubious we can include noFlo's UI editor without also buying into the FBP paradigm which we would need to evaluate separately. For example we would need the ability to take changes in the UI and generate messages to the AppMaster to modify the DAG dynamically.

theGrid may be a better candidate than noFlo since this is what noFlo uses to edit the graph UI.

StreamFlow may be too specific to Storm although it does take on remote code provisioning (as we've done) where its topology engine will remotely transmit and load jars via a separate class loader.

Veritgo may also be of limited value within the context of a UI since its built on Vert.x. The concept of Vert.x is interesting, where any language can be placed on their event bus as a graph vertex and the auto-redploy feature with instantly materialize this node.

Thanks for the interesting links. As Sean mentioned we would like to integrate this feature (UI editor for DAG) into the GearPump dashboard. We have looked at DataTorrent's dashboard and angular-dashboard-framework as candidate dashboards. I'll followup on this issue after looking at these frameworks more closely.

[sirinath]

Looked at DataTorrent. Thanks for the pointer. This has a table component perhaps can be used to create something like: http://www.ankhor.com/

[clockfly]

Will solve this in #491

The DAG concept in Gearpump is not like with other graph tool, so we'd like to use our own json definition to define a DAG. We already wrote a REST interface for this.

About the frontend graph composing UI, we tend to do a built-in solution in Gearpump dashboard, instead of relying on other third party service. javascript library visjs has a nice Graph editor, we can use that.

[clockfly]

Duplicate of #491