Skip to content

Commit

Permalink
Add Data lifecycle image and minor changes (#887)
Browse files Browse the repository at this point in the history
* Add Data lifecycle image and minor changes

This change depends on the image upload PR #886
It adds the image to README, some minor readability changes, explanation of VDK in a separate sentence.

* Rename Data Lyfecycle to Data Journey

* Add the svg image github link

* Add data journey file to images

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename Data_journey.svg to versatile-data-kit-data-journey.svg

* Add the .svg file from the images folder

* Add data journey img for the dark mode

* Add dark mode image

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Replace an icon in the data journey img

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
zverulacis and pre-commit-ci[bot] committed Jul 4, 2022
1 parent b5978af commit df38aba
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 6 deletions.
19 changes: 13 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,25 @@

# Overview

Versatile Data Kit is a framework which enables Data Engineers to develop, deploy, run and manage Data Jobs. **A Data Job is a data processing workload** and can be written in Python, SQL, or both at the same time. A Data Job enables Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation (T in ELT) into a database or any type of data storage.
Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Versatile Data Kit enables Data Engineers to develop, deploy, run and manage Data Jobs. **A Data Job is a data processing workload** and can be written in Python, SQL, or both at the same time. A Data Job enables Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation (T in ELT) into a database or any type of data storage.

Versatile Data Kit consists of two main components:

* A **Data SDK**, which provides all tools for the automation of data extraction, transformation and loading, as well as a plugin framework which allows users to extend the framework according to their specific requirements.
* A **Control Service**, which allows users to create, deploy, manage and execute Data Jobs in Kubernetes runtime environment.
* A **Data SDK** provides all tools for the automation of data extraction, transformation, and loading, as well as a plugin framework that allows users to extend the framework according to their specific requirements.
* A **Control Service** allows users to create, deploy, manage and execute Data Jobs in Kubernetes runtime environment.

To help solve common data engineering problems Versatile Data Kit:
* allows ingestion of data from different sources including CSV files, JSON objects, data provided by REST API services, etc.;
* ensures data applications are packaged, versioned and deployed correctly, while dealing with credentials, retries, reconnects, etc.;
* allows ingestion of data from different sources, including CSV files, JSON objects, data provided by REST API services, etc.;
* ensures data applications are packaged, versioned, and deployed correctly while dealing with credentials, retries, reconnects, etc.;
* provides built-in monitoring and smart notification capabilities;
* tracks both code and data modifications and the relations between them enabling engineers to troubleshoot faster as well as providing an easy revert to a stable version.
* tracks both code and data modifications and the relations between them, enabling engineers to troubleshoot faster and providing an easy revert to a stable version.


#### Data Journey and where VDK fits in
![Data Journey](./support/images/versatile-data-kit-data-journey.svg#gh-light-mode-only)
![Data Journey](./support/images/versatile-data-kit-data-journey-dark-mode.svg#gh-dark-mode-only)

# Installation and Getting Started

Expand Down
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions support/images/versatile-data-kit-data-journey.svg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit df38aba

Please sign in to comment.