Skip to content

DATAVIEW Release 3.0

Choose a tag to compare

View all tags
@ChangxinBai ChangxinBai released this 11 Dec 21:32
· 29 commits to master since this release
3.0
4e4c22d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

DATAVIEW Release 3.0

DATAVIEW documentation: https://github.com/shiyonglu/DATAVIEW

In this release of DATAVIEW 3.0, we introduce the infrastructure-level support of GPU-enabled Deep Learning via Deep-Learning-as-a-Workflow (DLaaW) in DATAVIEW. In this git release, we include all JAVA and javascript implementations which support NNWorkflows, our backend in-house developed CUDA APIs are compiled as either the Dynamic link libraries (.dll libraries) or Static link libraries (.o libraries) and saved in the WebContent/workflowTaskDir/TrainerDLLs and WebContent/workflowTaskDir/ExecutorDLLs folders, which are ready to be called by the NNWorkflow JAVA code. We have implemented the following JAVA side new features:

  1. NNTask model: NNTask class added, which extends the Task class in DATAVIEW and acts as a blueprint for all a-Layer-as-a-Task NNTasks (e.g. Linear, ReLu, Sigmoid), with necessary constructor overloading to accept additional parameters (e.g. how many input/output neurons for a particular layer).

  2. NNWorkflow model: NNWorkflow class added, which extends the Workflow class in DATAVIEW and acts a blueprint for all NNWorkflows (e.g. NNWorkflow1, NNWorkflow2), which can be constructed by an array of NNTasks.

  3. GenericNNWorkflow model: GenericNNWorkflow.java class added, which is the class that extends the Workflow class and used to create a NNWorkflow instance from the web bench (which is instantiated in Mediator.java).

  4. CallNativeJNI model: callNativeJNI.java class added, which serves as a utility that accepts fileLocation and DllRepo for specifying one of JNIs to be used in GPUResourceManagement Models.

  5. GPUResourceManagement models:

    GPUResourceManagement.java, which is the base class as an interface for variety GPU resources management, it provides two basic methods for child classes to use: CallLocalGPUService() and CallGPUClusterService(). All child classes will implement its execute() method and call respective GPU services. Currently the ResourceManagement are managing the NNWorkflow/Workflow executions including: 1) regular Train and Test on LocalGPU, singleNano SoM (System-on-Module) and singleXavier SoM; 2) K-fold cross validation on LocalGPU, singleNano SoM, singleXavier SoM and heterogeneous GPU cluster.

  6. NNTrainer models:

    NNWorkflowTrainer.java, which is the base class that will parse the NNWorkflow Specs to GPU recognizable specs in JSON format, these specs will be used in one of its particular child classes, in which each child class will specify and aggregate the infrastructural settings in JSON specs:

    • NNWorkflowTrainer_LocalGPU.java, which conducts regular NNWorkflow train and test on the Local NVIDIA GPU on the host PC.
    • NNWorkflowTrainer_SingleNano.java, which conducts regular NNWorkflow train and test on the single NVIDIA Nano SoM (System-on-Module).
    • NNWorkflowTrainer_SingleXavier.java, which conducts regular NNWorkflow train and test on the single NVIDIA Xavier SoM.
    • NNWorkflowTrainer_CrossValOnLocalGPU.java, which conducts K-fold cross validation on the Local NVIDIA GPU of the host PC.
    • NNWorkflowTrainer_CrossValOnSingleNano.java, which conducts K-fold cross validation on the single NVIDIA Nano SoM.
    • NNWorkflowTrainer_CrossValOnSingleXavier.java, which conducts K-fold cross validation on the single NVIDIA Xavier SoM.
    • NNWorkflowTrainer_CrossValOnGPUCluster.java, which conducts K-fold cross validation on the heterogeneous GPU cluster in distributed manner.

  7. NNExecutor models:

    NNWorkflowExecutor_base.java, which is the base class that will read the trained models' (in json format) architectural design, hyper-parameters, and new input dataset and construct them into GPU recognizable specs. Its child class will aggregate the GPU infrastructural settings into the specs:

    • NNWorkflowExecutor_LocalGPU.java, which conducts the reuse of any trained model for prediction on new datasets on the Local GPU of a PC.
    • NNWorkflowExecutor.java, which retrieves the user-specified type of GPU infrastructure from config.json configuration file.

  8. NNWorkflow web GUIs programming:

    The web GUIs are developed and introduced to support the design, construct, execute and reuse of NNWorkflows or pre-trained NNWorkflow models. The major changes for enabling NNWorkflow GUIs programming are including:

    • mxGraphRelatedCode.js, implemented the mechanism to distinguish the regular workflows and NNWorkflows, in order to parse them differently and show them in GUIs.
    • useWebbench.js, added a new execution type - Run NNWorkflow on local NVIDIA GPU in workflowRunConfigureDetail class, the new execution type will trigger an action in mediator.java and dispatch necessary parameters/specifications via AJAX call.

  9. Mediator.java model:

    Introduced the localGPURunWorkflows() method to 1) get the GUIs' tree specification (mxGraph) from the web browser; 2) parsed specification and correspondingly construct an NNWorkflow by instantiating a GenericNNWorkflow class; 3) Instantiate a NNWorkflowTrainer_LocalGPU class and run the target NNWorkflow on the Local NVIDIA GPU on the PC via triggering the train() method. The execution result (in JSON format) will be saved in a text file in local temp folder as well as uploaded to users' dropbox folder.

Assets 2
Loading