Skip to content
This repository was archived by the owner on Sep 1, 2026. It is now read-only.

airflow-provider-vdk: Initial Airflow provider structure - #772

Merged
gabrielgeorgiev1 merged 31 commits into
mainfrom
person/gageorgiev/aiflow-base
Apr 2, 2022
Merged

airflow-provider-vdk: Initial Airflow provider structure#772
gabrielgeorgiev1 merged 31 commits into
mainfrom
person/gageorgiev/aiflow-base

Conversation

@gabrielgeorgiev1

@gabrielgeorgiev1 gabrielgeorgiev1 commented Mar 18, 2022

Copy link
Copy Markdown
Contributor

Versatile Data Kit allows users to schedule job executions using
a cron-like interface, however there is no way to express job
dependencies through VDK. For example, a user might want to
run two different ingestion jobs, and when both pass successfully,
to run a transformation job on the ingested data.
Apache Airflow is an open-source workload scheduling framework
which allows exactly that sort of job dependency specification.
To take advantage of this, this PR sets the initial structure of the
VDK Airflow provider, which will integrate Airflow's scheduling
capabilities with VDK.

README is left short as much of the implementation and usage
is still somewhat up in the air AFAIK.
Interested in feedback on the way authentication is approached
in VDKHook.
Also interested in opinions on what would be good default values
for timeout and the retry params for VDKHook.

Testing done: package installs successfully

Signed-off-by: Gabriel Georgiev gageorgiev@vmware.com

Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gabrielgeorgiev1
gabrielgeorgiev1 marked this pull request as ready for review March 28, 2022 09:43
gabrielgeorgiev1 and others added 5 commits March 28, 2022 12:45
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gabrielgeorgiev1 gabrielgeorgiev1 changed the title versatile-data-kit: Initial Airflow provider structure airflow-provider-vdk: Initial Airflow provider structure Mar 28, 2022
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
…satile-data-kit into person/gageorgiev/aiflow-base
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@antoniivanov

Copy link
Copy Markdown
Contributor

Let's put it in the plugins directory. Technically it can be thought as of a plugin (extension) to vdk regardless if it uses vdk hooks or not.

Comment thread support/airflow-provider-vdk/setup.py Outdated
Comment thread support/airflow-provider-vdk/setup.py Outdated
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Comment thread projects/__init__.py Outdated
Comment thread projects/vdk-plugins/__init__.py Outdated
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gabrielgeorgiev1
gabrielgeorgiev1 merged commit d1a03da into main Apr 2, 2022
@gabrielgeorgiev1
gabrielgeorgiev1 deleted the person/gageorgiev/aiflow-base branch April 2, 2022 07:53
@gabrielgeorgiev1
gabrielgeorgiev1 restored the person/gageorgiev/aiflow-base branch April 2, 2022 08:14
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants