Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

airflow-provider-vdk: Start and cancel job execution methods #778

Merged
merged 52 commits into from
Apr 8, 2022
Merged

airflow-provider-vdk: Start and cancel job execution methods #778

merged 52 commits into from
Apr 8, 2022

Conversation

gageorgiev
Copy link
Contributor

@gageorgiev gageorgiev commented Apr 2, 2022 •
edited
Loading

This change allows job executions to be started and cancelled
through the VDK connection hook object.

Testing done: included two unit tests, tested locally against
deployed control-service

Signed-off-by: Gabriel Georgiev gageorgiev@vmware.com

gageorgiev and others added 30 commits March 18, 2022 15:46
@gageorgiev
versatile-data-kit: Initial Airflow provider structure
a80e93d 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
VDKHook interface, followed the airflow-provider-sample template
d696921 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
added auth, short readme
74bcdf1 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
added init files to test dirs
42f3f13 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fixing codacy warnings
18a0327 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
include cicd
b3ed3ee 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
change VDK to PROVIDER in CICD env var
cb7c66c 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
Merge branch 'main' into person/gageorgiev/aiflow-base
f584129
@gageorgiev
enable cicd jobs
ce16926 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
Merge branch 'person/gageorgiev/aiflow-base' of github.com:vmware/ver…
f391a59 
…satile-data-kit into person/gageorgiev/aiflow-base
@gageorgiev
amend cicd
4c0711f 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix cicd
fbec4a5 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix
f789072 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
testing ci problem
f7a3a43 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
testing ci issue
5702a22 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
testing ci issue
90f831d 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
this should work now
d422867 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
comment out hook method impls
e924977 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
final stuff
62593ca 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
commented out method impls again
4c1152d 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix entrypoint
0bfc0b0 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
include dummy test
8aeed2a 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix build script
ce2e730 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
move to vdk-plugins dir
7d42635 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix codacy warnings
78ef019 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix ci
f86efa6 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix import
16e2f1c 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
removed init files
baf78f4 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
dfsanjf
03a7594 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
gvhv
03ec452 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
restructure testing class
11a98cb 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix setup method
2aa08b6 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev gageorgiev mentioned this pull request Apr 3, 2022
Merged
doks5
doks5 reviewed Apr 4, 2022
View reviewed changes
Copy link
Contributor

@doks5 doks5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description of the PR and the diff look like they have diverged. Is this PR ready for review?

projects/vdk-plugins/airflow-provider-vdk/tests/hooks/test_vdkhook.py Show resolved Hide resolved
projects/vdk-plugins/airflow-provider-vdk/tests/hooks/test_vdkhook.py Show resolved Hide resolved
projects/vdk-plugins/airflow-provider-vdk/vdk_provider/hooks/vdk.py Outdated Show resolved Hide resolved
projects/vdk-plugins/airflow-provider-vdk/vdk_provider/hooks/vdk.py Outdated Show resolved Hide resolved
@gageorgiev
extra stuff
f333cd3 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
api call through taurus_datajob_api
6cc18ec 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
gageorgiev and others added 8 commits April 7, 2022 16:02
@gageorgiev
fixes, everything seems good now
8c3b8c0 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fixedb timeout passing
8dce11f 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
dropped requests-mock dep
afc316c 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
fix ci
2a24875 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
simplify tests
feae360 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
Merge branch 'main' into person/gageorgiev/aiflow-base
2b7f0f1
@gageorgiev
codacy fixes
faa3f02 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
@gageorgiev
Merge branch 'person/gageorgiev/aiflow-base' of github.com:vmware/ver…
5ee0d46 
…satile-data-kit into person/gageorgiev/aiflow-base
@gageorgiev
include todo
5a20cdd 
Signed-off-by: Gabriel Georgiev <gageorgiev@vmware.com>
gageorgiev and others added 2 commits April 8, 2022 14:07
@gageorgiev gageorgiev enabled auto-merge (squash) April 8, 2022 11:25
@gageorgiev gageorgiev merged commit 78016b0 into main Apr 8, 2022
@gageorgiev gageorgiev deleted the person/gageorgiev/aiflow-base branch April 8, 2022 11:27
antoniivanov
antoniivanov reviewed Apr 8, 2022
View reviewed changes
projects/vdk-plugins/airflow-provider-vdk/vdk_provider/hooks/vdk.py

:return: None
# setting these manually to avoid using VDKConfig
self.op_id = os.environ.get("VDK_OP_ID_OVERRIDE", f"{uuid.uuid4().hex}"[:16])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the airflow world , those would not come from environmental variables.

Sync with Andy or Miro (who may have more context) but generally this should be fixed in next PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@antoniivanov antoniivanov antoniivanov approved these changes

@mivanov1988 mivanov1988 mivanov1988 left review comments

@doks5 doks5 doks5 left review comments

Assignees
No one assigned
Labels
cla-not-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gageorgiev @antoniivanov @mivanov1988 @doks5 @vmwclabot