Skip to content
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Java Scala CSS Shell TSQL
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assembly 1.Automated installation enhancements. Jan 13, 2020
bin Optimized the way to obtain the local IP address Jan 16, 2020
conf DSS workflow add shell node Feb 6, 2020
datachecker-appjoint Version number modification Dec 24, 2019
db DSS workflow add shell node Feb 6, 2020
docs Update DSS_LINKIS_Quick_Install.md Apr 2, 2020
dss-appjoint-auth 1.Automated installation enhancements. Jan 13, 2020
dss-appjoint-core Version number modification Dec 24, 2019
dss-appjoint-loader Version number modification Dec 24, 2019
dss-application Version number modification Dec 24, 2019
dss-azkaban-scheduler-appjoint Version number modification Dec 24, 2019
dss-common Version number modification Dec 24, 2019
dss-flow-execution-entrance 1.Automated installation enhancements. Jan 13, 2020
dss-linkis-node-execution 1.Automated installation enhancements. Jan 13, 2020
dss-scheduler-appjoint-core 1.Automated installation enhancements. Jan 13, 2020
dss-server 1.Automated installation enhancements. Jan 13, 2020
eventchecker-appjoint Version number modification Dec 24, 2019
images DSS Quick Installation Documentation Update Dec 2, 2019
plugins 1.Automated installation enhancements. Jan 13, 2020
qualitis-appjoint/appjoint Version number modification Dec 24, 2019
sendemail-appjoint/sendemail-core Version number modification Dec 24, 2019
visualis-appjoint/appjoint
web Merge pull request #102 from zwx-master/master Feb 11, 2020
.gitattributes initial commit for gitattributes. Dec 5, 2019
.gitignore initial commit for .gitignore Nov 30, 2019
LICENSE Initial commit Nov 24, 2019
README-ZH.md Update README-ZH.md Feb 12, 2020
README.md update README. Dec 5, 2019
pom.xml 1.Automated installation enhancements. Jan 13, 2020

README.md

DSS

License

English | 中文

Introduction

DataSphere Studio (DSS for short) is WeDataSphere, a big data platform of WeBank, a self-developed one-stop data application development management portal.

Based on Linkis computation middleware, DSS can easily integrate upper-level data application systems, making data application development simple and easy to use.

DataSphere Studio is positioned as a data application development portal, and the closed loop covers the entire process of data application development. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc.

With the connection, reusability, and simplification capabilities of Linkis, DSS is born with financial-grade capabilities of high concurrency, high availability, multi-tenant isolation, and resource management.

UI preview

Please be patient, it will take some time to load gif.

DSS-V1.0 GIF

Core features

1. One-stop, full-process application development management UI

       DSS is highly integrated. Currently integrated systems include:

       a. Scriptis - Data Development IDE Tool.

       b. Visualis - Data Visualization Tool(Based on the open source project Davinci contributed by CreditEase)

       c. Qualitis - Data Quality Management Tool

       d. Azkaban - Batch workflow job scheduler

DSS one-stop video

2. AppJoint, based on Linkis,defines a unique design concept

       AppJoint——application joint, defining unified front-end and back-end integration specifications, can quickly and easily integrate with external data application systems, making them as part of DSS data application development.

       DSS arranges multiple AppJoints in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.

       Since AppJoint is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppJoint also allows sharable context across system level and completely gets away from application silos.

3. Project, as the management unit

       With Project as the management unit, DSS organizes and manages the business applications of each data application system, and defines a set of common standards for collaborative development of projects across data application systems.

4. Integrated data application components

      a. Azkaban AppJoint —— Batch workflow job scheduler

         Many data applications developed by users usually require periodic scheduling capability.

         At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems.

         DSS implements Azkaban AppJoint, which allows users to publish DSS workflows to Azkaban for regular scheduling.

         DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS.

Azkaban

      b. Scriptis AppJoint —— Data Development IDE Tool

         What is Scriptis?

         Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

         Scriptis AppJoint integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.

         Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types.

Scriptis

      c. Visualis AppJoint —— Data Visualization Tool

         What is Visualis?

         Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase.

         Visualis AppJoint integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.

Visualis

      d. Qualitis AppJoint —— Data quality management Tool

         Qualitis AppJoint integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow

Qualitis

      e. Data Sender——Sender AppJoint

         Sender AppJoint provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.

         For example, the SendEmail node can directly send the screen shot of a display as an email.

      f. Signal AppJoint —— Signal Nodes

         Signal AppJoint is used to strengthen the correlation between business and process while keeping them decoupled.

         DataChecker Node:Checks whether a table or partition exists.

         EventSender Node: Messaging nodes across workflows and projects.

         EventReceiver: Receive nodes for messages across workflows and projects.

      g. Function node

         Empty nodes, sub workflow nodes.

Compared with similar systems

      DSS is an open source project leading the direction of data application development and management. The open source community currently does not have similar products.

Usage Scenarios

      DataSphere Studio is suitable for the following scenarios:

      1. Scenarios in which big data platform capability is being prepared or initialized but no data application tools are available.

      2. Scenarios in which users already have big data foundation platform capabilities but with only a few data application tools.

      3. Scenarios in which users have the ability of big data foundation platform and comprehensive data application tools, but suffers strong isolation and and high learning costs because those tools have not been integrated together.

      4. Scenarios in which users have the capabilities of big data foundation platform and comprehensive data application tools. but lacks unified and standardized specifications, while a part of these tools have been integrated.

Quick start

Click to Quick start

Architecture

DSS Architecture

Documents

Compiled documentation

User manual

Quick integration with DSS for external systems

Communication

communication

License

DSS is under the Apache 2.0 license. See the License file for details.

You can’t perform that action at this time.