Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added job operator jinja templating #27

Merged
merged 8 commits into from
Nov 12, 2020
Merged

Added job operator jinja templating #27

merged 8 commits into from
Nov 12, 2020

Conversation

LamaAni
Copy link
Owner

@LamaAni LamaAni commented Nov 11, 2020

Fixes

  1. Added support for jinja templating.
  2. Added support for jinja templating in legacy operator.
  3. Changed operation execution schema, moved to create the job_runner only after the job execution starts. This allows for generated jinja template values.

@LamaAni LamaAni mentioned this pull request Nov 11, 2020
Closed
odesenfans
odesenfans requested changes Nov 12, 2020
View reviewed changes
Copy link
Collaborator

@odesenfans odesenfans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main points:

  • Missing the ability to specify custom template arguments
  • Too many fields seem to be template-ized, for all other fields beside the body it's possible to specify their values using Python strings/variables directly. Jinja is overkill for these use cases.

airflow_kubernetes_job_operator/kubernetes_job_operator.py Show resolved Hide resolved
airflow_kubernetes_job_operator/kubernetes_job_operator.py Outdated Show resolved Hide resolved
tests/dags/test_job_operator.py Show resolved Hide resolved
tests/dags/test_job_operator_jinja.py Show resolved Hide resolved
airflow_kubernetes_job_operator/kubernetes_job_operator.py Show resolved Hide resolved
airflow_kubernetes_job_operator/kubernetes_job_operator.py Outdated Show resolved Hide resolved
@odesenfans
Copy link
Collaborator

Definitely commented a bit early, I just read a bit more about Airflow's jinja support. Disregard my comment about having templates all over the place.

@LamaAni
Copy link
Owner Author

LamaAni commented Nov 12, 2020 •
edited
Loading

@odesenfans Can you please resolve your comments?

odesenfans
odesenfans approved these changes Nov 12, 2020
View reviewed changes
Copy link
Collaborator

@odesenfans odesenfans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood the templating mechanism better and all my other comments are fixed.

@LamaAni LamaAni merged commit 07ce84d into master Nov 12, 2020
@LamaAni LamaAni deleted the issue_23_jinja branch November 12, 2020 16:17
@odesenfans
Copy link
Collaborator

Note that this PR also improves the support for XCom. Technically you can now pull XCom results and pass them as image, arguments, etc. Good thing because that was my next feature request!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@odesenfans odesenfans odesenfans approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@LamaAni @odesenfans