Installation

Install SDK

Versatile Data Kit SDK (vdk) is the go to tool for developing and running data jobs locally.

Prerequisites

Versatile Data Kit CLI (VDK) requires Python 3.7+. If you're new to Python, we recommend Anaconda.

Installation

pip install quickstart-vdk

This will install VDK with support for sqlite and trino databases and job lifecycle management operations.

See help to see what you can do:

vdk --help

Optional features

VDK comes with number plugins that can be installed to change or extend behaviour of vdk.

pip install vdk-plugin-name

See plugins page for more info.

Licenses

To generate licenses report for dependencies, build vdk-core & build vdk-control-cli, then run:

pip-licenses --format=html > dependency_license.html

Install control service server

Versatile Data Kit includes Control Service server enabling deploying, managing and monitoring data jobs. It is installed using helm.

Prerequisites

• Install helm
• Install kubernetes distribution. If you're new to kubernetes, we recommend minikube
• Install docker

Install Versatile Data Kit Control Service

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install bitnami/pipelines-control-service

Use

# see Help to see what you can do and start playing around
vdk --help 
# for example create a new job:
vdk create -u <URI-YOU-GOT-FROM-helm-install-OUTPUT>

Licenses

To generate licenses report for dependencies, found per module in build/reports/dependency-license:

./gradlew :base:generateLicenseReport
./gradlew :pipelines_control_service:generateLicenseReport
./model/gradlew generateLicenseReport