Skip to content

Chart: Avoid git-sync sidecar on Websever when Airflow>=2.0.0 #15814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kaxil merged 1 commit into from
May 13, 2021
Merged

Chart: Avoid git-sync sidecar on Websever when Airflow>=2.0.0 #15814

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions chart/templates/webserver/webserver-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ spec:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
containers:
{{- if and (.Values.dags.gitSync.enabled) (not .Values.dags.persistence.enabled) (not .Values.dags.gitSync.excludeWebserver) }}
{{- if and (.Values.dags.gitSync.enabled) (not .Values.dags.persistence.enabled) (semverCompare "<2.0.0" .Values.airflowVersion) }}
{{- include "git_sync_container" . | indent 8 }}
{{- end }}
- name: webserver
Expand All @@ -134,7 +134,7 @@ spec:
subPath: airflow_local_settings.py
readOnly: true
{{- end }}
{{- if or (and .Values.dags.gitSync.enabled (not .Values.dags.gitSync.excludeWebserver)) .Values.dags.persistence.enabled }}
{{- if or (and .Values.dags.gitSync.enabled (semverCompare "<2.0.0" .Values.airflowVersion)) .Values.dags.persistence.enabled }}
- name: dags
mountPath: {{ template "airflow_dags_mount_path" . }}
{{- end }}
Expand Down Expand Up @@ -195,7 +195,7 @@ spec:
- name: dags
persistentVolumeClaim:
claimName: {{ template "airflow_dags_volume_claim" . }}
{{- else if and (.Values.dags.gitSync.enabled) (not .Values.dags.gitSync.excludeWebserver) }}
{{- else if and (.Values.dags.gitSync.enabled) (semverCompare "<2.0.0" .Values.airflowVersion) }}
- name: dags
emptyDir: {}
{{- if .Values.dags.gitSync.sshKeySecret }}
Expand Down
49 changes: 41 additions & 8 deletions chart/tests/test_git_sync_webserver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,21 @@
class GitSyncWebserverTest(unittest.TestCase):
def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_persistence_is_enabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}}},
values={
"airflowVersion": "1.10.14",
"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}},
},
show_only=["templates/webserver/webserver-deployment.yaml"],
)

assert "dags" == jmespath.search("spec.template.spec.volumes[1].name", docs[0])

def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_persistence_is_disabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}}},
values={
"airflowVersion": "1.10.14",
"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}},
},
show_only=["templates/webserver/webserver-deployment.yaml"],
)

Expand All @@ -43,10 +49,11 @@ def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_pers
def test_should_add_git_sync_container_to_webserver_if_persistence_is_not_enabled_but_git_sync_is(self):
docs = render_chart(
values={
"airflowVersion": "1.10.14",
"dags": {
"gitSync": {"enabled": True, "containerName": "git-sync"},
"persistence": {"enabled": False},
}
},
},
show_only=["templates/webserver/webserver-deployment.yaml"],
)
Expand All @@ -63,18 +70,44 @@ def test_should_have_service_account_defined(self):
"spec.template.spec.serviceAccountName", docs[0]
)

@parameterized.expand([(True,), (False,)])
def test_git_sync_with_exclude_webserver(self, exclude_webserver):
@parameterized.expand(
[
(
"2.0.0",
True,
),
(
"2.0.2",
True,
),
(
"1.10.14",
False,
),
(
"1.9.0",
False,
),
(
"2.1.0",
True,
),
],
)
def test_git_sync_with_different_airflow_versions(self, airflow_version, exclude_webserver):
"""
If that dags.gitSync.excludeWebserver=True - git sync related containers, volume mounts & volumes
If Airflow >= 2.0.0 - git sync related containers, volume mounts & volumes
are not created.
"""
docs = render_chart(
values={
"airflowVersion": airflow_version,
"dags": {
"gitSync": {"enabled": True, "excludeWebserver": exclude_webserver},
"gitSync": {
"enabled": True,
},
"persistence": {"enabled": False},
}
},
},
show_only=["templates/webserver/webserver-deployment.yaml"],
)
Expand Down
4 changes: 0 additions & 4 deletions chart/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1677,10 +1677,6 @@
"description": "Enable Git sync.",
"type": "boolean"
},
"excludeWebserver": {
"description": "Disable Git sync on webserver as it is not needed when DAG Serialization is enabled.",
"type": "boolean"
},
"repo": {
"description": "Git repository.",
"type": "string"
Expand Down
6 changes: 0 additions & 6 deletions chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -955,12 +955,6 @@ dags:
gitSync:
enabled: false

# Change it to true when DAG Serialization is turned on. This will exclude git sync containers
# from Webserver as DAGs are fetched from the DB. DAG Serialization was introduced in
# 1.10.7 and is optional for <2.0.0.
# https://airflow.apache.org/docs/apache-airflow/1.10.15/dag-serialization.html
excludeWebserver: false

# git repo clone url
# ssh examples ssh://git@github.com/apache/airflow.git
# git@github.com:apache/airflow.git
Expand Down
14 changes: 6 additions & 8 deletions docs/helm-chart/manage-dags-files.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Mounting DAGs using Git-Sync sidecar with Persistence enabled

This option will use a Persistent Volume Claim with an access mode of ``ReadWriteMany``.
The scheduler pod will sync DAGs from a git repository onto the PVC every configured number of
seconds. The other pods will read the synced DAGs. Not all volume plugins have support for
seconds. The other pods will read the synced DAGs. Not all volume plugins have support for
``ReadWriteMany`` access mode.
Refer `Persistent Volume Access Modes <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes>`__
for details.
Expand All @@ -96,25 +96,20 @@ for details.
# by setting the dags.persistence.* and dags.gitSync.* values
# Please refer to values.yaml for details

When using ``apache-airflow>=2.0.0``, :ref:`DAG Serialization <apache-airflow:dag-serialization>` is enabled by default,
hence Webserver does not need access to DAG files, so you can turn off ``git-sync`` for Webserver by setting
``dags.gitSync.excludeWebserver`` to ``true``.
This is also recommended when enabling DAG Serialization for ``apache-airflow>=1.10.11,<2``.

.. code-block:: bash

helm upgrade airflow . \
--set dags.persistence.enabled=true \
--set dags.gitSync.enabled=true \
--set dags.gitSync.excludeWebserver=true
# you can also override the other persistence or gitSync values
# by setting the dags.persistence.* and dags.gitSync.* values
# Please refer to values.yaml for details

Mounting DAGs using Git-Sync sidecar without Persistence
--------------------------------------------------------

This option will use an always running Git-Sync sidecar on every scheduler, webserver and worker pods.
This option will use an always running Git-Sync sidecar on every scheduler, webserver (if ``airflowVersion < 2.0.0``)
and worker pods.
The Git-Sync sidecar containers will sync DAGs from a git repository every configured number of
seconds. If you are using the ``KubernetesExecutor``, Git-sync will run as an init container on your worker pods.

Expand All @@ -127,6 +122,9 @@ seconds. If you are using the ``KubernetesExecutor``, Git-sync will run as an in
# by setting the dags.gitSync.* values
# Refer values.yaml for details

When using ``apache-airflow>=2.0.0``, :ref:`DAG Serialization <apache-airflow:dag-serialization>` is enabled by default,
hence Webserver does not need access to DAG files, so ``git-sync`` sidecar is not run on Webserver.

Mounting DAGs from an externally populated PVC
----------------------------------------------

Expand Down