dictionary

Versatile Data Kit Terms Dictionary

Data Job

Data processing unit that allows Data Engineers to implement automated pull ingestion (E in ELT) or batch data transformation into Data Warehouse (T in ELT).

Data Job Step

A single unit of work for a data job. Usually this is SQL or python file. Steps are executed in alphanumerical order of their file names. Plugins can provide different type of steps or change execution order.

Data Job Source

All python, SQL files, and requirements.txt (and others) of the Data Job. Those are versioned and tracked per job.

Data Job Properties

Any saved state, configuration, and secrets of a Data Job. Those are tracked per deployment. In the future they should also be versioned.

Data Job Deployment

Deployment takes both the build/code and the deployment specific properties, build and packages them and is ready for immediate execution in the execution environment

Data Job Run

Data Job Attempt

Single run of data job. Data Job run is usually referred to when a data job is executed locally. A job may have multiple attempts (or runs) in a Job Execution. See Data Job Execution.

Data Job Execution

An instance of a running data job deployment is called an execution.

Data job execution can run a data job one or more times. If a run (attempt) fails due to a platform error, the job can be automatically re-run (this is configurable by Control Service operators)

This is applicable only for executions in the "Cloud"(Kubernetes). Local executions always comprise of a single attempt.

See Data Job Attempt.

Data Job Arguments

A Data Job Execution or Data Job Run can accept any number of arguments when started. Those are unique for each execution/run. They are usually used when Execution is triggered manually by user or when user is developing locally.

VDK

VDK is Versatile Data Kit SDK. It provides common functionality for data ingestion and processing and CLI for managing lifecycle of a data job (see Control Service)

Control Service

The "backend" part. It provides API for managing the lifecycle of data engineers' data jobs.