DATAVIEW Release 3.1

We have integrated metamask into the DATAVIEW to provide the login functionality, bringing the usability of DATAVIEW for Web3 users. A profile section displaying the address id, balance information and customized avatar is also provided.

DATAVIEW Release 3.0.1

Minor patch on NNWorkflowExecutor.java.

DATAVIEW Release 3.0

DATAVIEW Release 3.0

DATAVIEW documentation: https://github.com/shiyonglu/DATAVIEW

In this release of DATAVIEW 3.0, we introduce the infrastructure-level support of GPU-enabled Deep Learning via Deep-Learning-as-a-Workflow (DLaaW) in DATAVIEW. In this git release, we include all JAVA and javascript implementations which support NNWorkflows, our backend in-house developed CUDA APIs are compiled as either the Dynamic link libraries (.dll libraries) or Static link libraries (.o libraries) and saved in the WebContent/workflowTaskDir/TrainerDLLs and WebContent/workflowTaskDir/ExecutorDLLs folders, which are ready to be called by the NNWorkflow JAVA code. We have implemented the following JAVA side new features:

1. NNTask model: NNTask class added, which extends the Task class in DATAVIEW and acts as a blueprint for all a-Layer-as-a-Task NNTasks (e.g. Linear, ReLu, Sigmoid), with necessary constructor overloading to accept additional parameters (e.g. how many input/output neurons for a particular layer).

2. NNWorkflow model: NNWorkflow class added, which extends the Workflow class in DATAVIEW and acts a blueprint for all NNWorkflows (e.g. NNWorkflow1, NNWorkflow2), which can be constructed by an array of NNTasks.

3. GenericNNWorkflow model: GenericNNWorkflow.java class added, which is the class that extends the Workflow class and used to create a NNWorkflow instance from the web bench (which is instantiated in Mediator.java).

4. CallNativeJNI model: callNativeJNI.java class added, which serves as a utility that accepts fileLocation and DllRepo for specifying one of JNIs to be used in GPUResourceManagement Models.

5. GPUResourceManagement models:

   GPUResourceManagement.java, which is the base class as an interface for variety GPU resources management, it provides two basic methods for child classes to use: CallLocalGPUService() and CallGPUClusterService(). All child classes will implement its execute() method and call respective GPU services. Currently the ResourceManagement are managing the NNWorkflow/Workflow executions including: 1) regular Train and Test on LocalGPU, singleNano SoM (System-on-Module) and singleXavier SoM; 2) K-fold cross validation on LocalGPU, singleNano SoM, singleXavier SoM and heterogeneous GPU cluster.

6. NNTrainer models:

   NNWorkflowTrainer.java, which is the base class that will parse the NNWorkflow Specs to GPU recognizable specs in JSON format, these specs will be used in one of its particular child classes, in which each child class will specify and aggregate the infrastructural settings in JSON specs:

   • NNWorkflowTrainer_LocalGPU.java, which conducts regular NNWorkflow train and test on the Local NVIDIA GPU on the host PC.
   • NNWorkflowTrainer_SingleNano.java, which conducts regular NNWorkflow train and test on the single NVIDIA Nano SoM (System-on-Module).
   • NNWorkflowTrainer_SingleXavier.java, which conducts regular NNWorkflow train and test on the single NVIDIA Xavier SoM.
   • NNWorkflowTrainer_CrossValOnLocalGPU.java, which conducts K-fold cross validation on the Local NVIDIA GPU of the host PC.
   • NNWorkflowTrainer_CrossValOnSingleNano.java, which conducts K-fold cross validation on the single NVIDIA Nano SoM.
   • NNWorkflowTrainer_CrossValOnSingleXavier.java, which conducts K-fold cross validation on the single NVIDIA Xavier SoM.
   • NNWorkflowTrainer_CrossValOnGPUCluster.java, which conducts K-fold cross validation on the heterogeneous GPU cluster in distributed manner.

7. NNExecutor models:

   NNWorkflowExecutor_base.java, which is the base class that will read the trained models' (in json format) architectural design, hyper-parameters, and new input dataset and construct them into GPU recognizable specs. Its child class will aggregate the GPU infrastructural settings into the specs:

   • NNWorkflowExecutor_LocalGPU.java, which conducts the reuse of any trained model for prediction on new datasets on the Local GPU of a PC.
   • NNWorkflowExecutor.java, which retrieves the user-specified type of GPU infrastructure from config.json configuration file.

8. NNWorkflow web GUIs programming:

   The web GUIs are developed and introduced to support the design, construct, execute and reuse of NNWorkflows or pre-trained NNWorkflow models. The major changes for enabling NNWorkflow GUIs programming are including:

   • mxGraphRelatedCode.js, implemented the mechanism to distinguish the regular workflows and NNWorkflows, in order to parse them differently and show them in GUIs.
   • useWebbench.js, added a new execution type - Run NNWorkflow on local NVIDIA GPU in workflowRunConfigureDetail class, the new execution type will trigger an action in mediator.java and dispatch necessary parameters/specifications via AJAX call.

9. Mediator.java model:

   Introduced the localGPURunWorkflows() method to 1) get the GUIs' tree specification (mxGraph) from the web browser; 2) parsed specification and correspondingly construct an NNWorkflow by instantiating a GenericNNWorkflow class; 3) Instantiate a NNWorkflowTrainer_LocalGPU class and run the target NNWorkflow on the Local NVIDIA GPU on the PC via triggering the train() method. The execution result (in JSON format) will be saved in a text file in local temp folder as well as uploaded to users' dropbox folder.

DATAVIEW Release 2.2.1

In this release of DATAVIEW 2.2.1, we replace the previous invalid AMI.

DATAVIEW Release 2.2

DATAVIEW documentation: https://github.com/shiyonglu/DATAVIEW

In this release of DATAVIEW 2.2, we have implemented the following features:

(1) Task model: No change.

(2) Workflow model

1. For workflow.java, changed the return object of getExecutionTime() method from Map<Integer, Map<String, Double >> to Map<String, Map<String, Double >> and getTransferTime() method from Map<Integer, Map<Integer, Double>> to Map<String, Map<String, Double>>. The key of the Map is changed from TaskID to TaskName.
2. DATAVIEW_Table was proposed to support relational algebra operator tasks.

(3) WorkflowPlanner model
1) We improved the LPOD planner in accordance with the LPOD journal paper, which reduces the time complexity to polynomial time using dynamic programming.

(4) WorkflowExecutor model
1) We supported the workflow configuration functionality, such as the LocalStorage properties, Amazon EC2 parameters and confidential task names during the workflow execution process, which is located in DATAVIEW/WebContent/config.json, which can be referred from https://github.com/shiyonglu/DATAVIEW/blob/master/DATAVIEW/WebContent/config.json.

(5): A Provenance model
1) We added the provenance collector component so that provenance is captured during the execution of a workflow, which is used by the LPOD planner for the estimate of task execution times and data movement times.

(6) Visual programming
1) A local treeView functionality is integrated in the current version, eliminating the dependency on an external PHP script.

(7): DATAVIEW website
1) A user can now access the official DATAVIEW website: www.dataview.org or run their own custom workflow on the DATAVIEW server or on Amazon EC2 cloud. This project can be executed with any JAVA version.

(8): Relational algebra
1) This release supports relational algebra in terms of 13 Relational Algebra tasks, so that a user can connect these tasks into a RA expression and run it.

DATAVIEW Release 2.1

DATAVIEW documentation: https://github.com/shiyonglu/DATAVIEW

In this release of DATAVIEW 2.1, we have implemented the following features:

(1): A task model: No change.

(2): A workflow model:
1): We introduce workflow inputs and workflow outputs so that a workflow can take an arbitrary number of objects (in the type of DATAVIEW_int, DATAVIEW_double,..., DATAVIEW_Table) as the workflow inputs, and can take an arbitrary number of objects (in the type of DATAVIEW_int, DATAVIEW_double,..., DATAVIEW_Table) as workflow outputs.

(3): A WorkflowPlanner model:

1): We added the E2C2D planner.

2): We refactored the LPOD planner so that it is more readable in the context of the published LPOD conference paper.

(4): A WorkflowExecutor model:
1): We added the WorkflowExecutor_Local.

(5): A TaskExecutor model:

1): We parallelize the data movement of intermediate data products.
2): Refactoring the code to make it more readable

(6): A Provenance model:
1): We add the provenance collector component so that provenance is captured during the execution of a workflow.

(7): DATAVIEW website:
1): A user can now access the official DATAVIEW website: www.dataview.org or deploy their own custom DATAVIEW website.

We recommend deploying the DATAVIEW project with JAVA8.

DATAVIEW Release 2.0

In this release of DATAVIEW 2.0, we have implemented the following six features:

(1): A task model: Task.java, Port.Java, InputPort.java and OutputPort.java.

(2): A workflow model: Workflow.java, WorkflowEdge.java, TaskSchedule.java, LocalSchedule.java and GlobalSchedule.java

(3) A WorkflowPlanner model: WorkflowPlanner.java and several workflow planners:

1) Naive1: The naive1 workflow  planner will assign each task of a worklfow to a different VM.

2) Naive2: The naive2 workflow planner will assign all the tasks of a workflow  to one single VM.

3) T-cluster: The T_cluster workflow planner is based solely on the structure of the given workflow graph.

4) ICPCP: This ICPCP algorithm is implemented based on this paper "Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service Clouds".

5) LPOD: This algorithm is implemented on this paper: "A Local Path Based Optimized Scheduling Algorithm for Deadline-Constrained Big Data Workflows in the Cloud".

(4): A WorkflowExecutor model: WorkflowExecutor.java and several workflow executors:
1) WorkflowExecutor_Beta: WorkflowExecutor_Beta.java

(5): A TaskExecutor model: TaskExecutor.java.

(6) a Provenance model: ProvenanceGraph.java, ProvenanceEdge.java, ProvenanceNode.java