Kubernetes worker pod doesn't use docker container entrypoint #12766

dimberman · 2020-12-02T22:17:40Z

Fixes issue on openshift caused by KubernetesExecutor pods not running
via the entrypoint script

Addresses #12602

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

chart/templates/webserver/webserver-deployment.yaml

scripts/in_container/prod/entrypoint_prod.sh

airflow/kubernetes/pod_generator.py

airflow/executors/kubernetes_executor.py

scripts/in_container/prod/entrypoint_prod.sh

github-actions · 2020-12-05T16:02:35Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-07T04:53:03Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

The change is backwards-compatible. It still allows to pass airflow command without "airflow" as first parameter, but you can now also pass "airflow" and the rest of the parameters will be treated as "airflow" command parameters. Documentation is updated to reflect the entrypoint behaviour including _CMD option in SQL connections. Part of apache#12762 and apache#12602 Partially extracted from apache#12766

ashb · 2020-12-07T13:08:22Z

Please rebase onto mainline once #12878 has been merged.

The change is backwards-compatible. It still allows to pass airflow command without "airflow" as first parameter, but you can now also pass "airflow" and the rest of the parameters will be treated as "airflow" command parameters. Documentation is updated to reflect the entrypoint behaviour including _CMD option in SQL connections. Part of #12762 and #12602 Partially extracted from #12766

Fixes issue on openshift caused by KubernetesExecutor pods not running via the entrypoint script

ashb · 2020-12-07T16:18:14Z

scripts/in_container/prod/entrypoint_prod.sh

-if [[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|flower)$ ]]; then
-    if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then
+if [[ ${AIRFLOW_COMMAND} =~ ^(scheduler|celery|worker|flower)$ ]]; then
+  if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then


Suggested change

if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then

if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then

ashb · 2020-12-07T16:18:50Z

scripts/in_container/prod/entrypoint_prod.sh

-# The Bash and python commands still should verify the basic connections so they are run after the
-# DB check but before the broker check
+


Could you add back/keep this comment.

UPGRADING_TO_2.0.md

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

The change is backwards-compatible. It still allows to pass airflow command without "airflow" as first parameter, but you can now also pass "airflow" and the rest of the parameters will be treated as "airflow" command parameters. Documentation is updated to reflect the entrypoint behaviour including _CMD option in SQL connections. Part of #12762 and #12602 Partially extracted from #12766 (cherry picked from commit 4d44faa)

@ashb

* Kubernetes worker pod doesn't use docker container entrypoint Fixes issue on openshift caused by KubernetesExecutor pods not running via the entrypoint script * fix * Update UPGRADING_TO_2.0.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * fix UPDGRADING * @ashb comments Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 190066c)

The change is backwards-compatible. It still allows to pass airflow command without "airflow" as first parameter, but you can now also pass "airflow" and the rest of the parameters will be treated as "airflow" command parameters. Documentation is updated to reflect the entrypoint behaviour including _CMD option in SQL connections. Part of #12762 and #12602 Partially extracted from #12766 (cherry picked from commit 4d44faa)

@ashb

* Kubernetes worker pod doesn't use docker container entrypoint Fixes issue on openshift caused by KubernetesExecutor pods not running via the entrypoint script * fix * Update UPGRADING_TO_2.0.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * fix UPDGRADING * @ashb comments Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 190066c)

The change is backwards-compatible. It still allows to pass airflow command without "airflow" as first parameter, but you can now also pass "airflow" and the rest of the parameters will be treated as "airflow" command parameters. Documentation is updated to reflect the entrypoint behaviour including _CMD option in SQL connections. Part of #12762 and #12602 Partially extracted from #12766 (cherry picked from commit 4d44faa)

@ashb

* Kubernetes worker pod doesn't use docker container entrypoint Fixes issue on openshift caused by KubernetesExecutor pods not running via the entrypoint script * fix * Update UPGRADING_TO_2.0.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * fix UPDGRADING * @ashb comments Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 190066c)

boring-cyborg bot added area:dev-tools area:helm-chart Airflow Helm Chart provider:cncf-kubernetes Kubernetes provider related issues labels Dec 2, 2020

dimberman requested a review from ashb December 2, 2020 22:18

turbaszek reviewed Dec 2, 2020

View reviewed changes

chart/templates/webserver/webserver-deployment.yaml Show resolved Hide resolved

dimberman linked an issue Dec 2, 2020 that may be closed by this pull request

Kubernetes worker pod doesn't use docker container entrypoint, removing OpenShift support #12602

Closed

ashb reviewed Dec 3, 2020

View reviewed changes

scripts/in_container/prod/entrypoint_prod.sh Show resolved Hide resolved

ashb reviewed Dec 3, 2020

View reviewed changes

airflow/kubernetes/pod_generator.py Show resolved Hide resolved

ashb reviewed Dec 3, 2020

View reviewed changes

airflow/executors/kubernetes_executor.py Show resolved Hide resolved

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 3, 2020

github-actions bot removed the full tests needed We need to run full set of tests for this PR to merge label Dec 3, 2020

ashb reviewed Dec 4, 2020

View reviewed changes

scripts/in_container/prod/entrypoint_prod.sh Outdated Show resolved Hide resolved

dimberman force-pushed the fix-entrypoint-for-openshift branch from 3c951ec to f36c07c Compare December 5, 2020 05:50

dimberman force-pushed the fix-entrypoint-for-openshift branch from b3917b4 to fe31372 Compare December 7, 2020 04:54

potiuk self-requested a review December 7, 2020 11:19

ashb mentioned this pull request Dec 7, 2020

Kubernetes worker pod doesn't use docker container entrypoint, removing OpenShift support #12602

Closed

potiuk mentioned this pull request Dec 7, 2020

Adds airflow as viable docker command in official image #12878

Merged

Kubernetes worker pod doesn't use docker container entrypoint

e866806

Fixes issue on openshift caused by KubernetesExecutor pods not running via the entrypoint script

dimberman force-pushed the fix-entrypoint-for-openshift branch from fe31372 to e866806 Compare December 7, 2020 14:29

fix

b64e3fd

ashb reviewed Dec 7, 2020

View reviewed changes

UPGRADING_TO_2.0.md Outdated Show resolved Hide resolved

ashb reviewed Dec 7, 2020

View reviewed changes

UPGRADING_TO_2.0.md Outdated Show resolved Hide resolved

dimberman and others added 3 commits December 7, 2020 10:02

Update UPGRADING_TO_2.0.md

69821a7

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

fix UPDGRADING

89f3244

@ashb comments

e642598

ashb approved these changes Dec 7, 2020

View reviewed changes

dimberman merged commit 190066c into apache:master Dec 7, 2020

dimberman deleted the fix-entrypoint-for-openshift branch December 7, 2020 21:54

cris-b mentioned this pull request Dec 14, 2020

Allow configuration of args vs command to support container entrypoint in kubernetes executor #8495

Closed

matthieu-foucault mentioned this pull request Feb 4, 2021

feat: PodGenerator.construct_pod uses k8s args instead of command #12762

Closed

dimberman added this to the Airflow 2.0.1 milestone Feb 4, 2021

easontm mentioned this pull request Jun 3, 2021

Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing #13542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes worker pod doesn't use docker container entrypoint #12766

Kubernetes worker pod doesn't use docker container entrypoint #12766

dimberman commented Dec 2, 2020 •

edited

github-actions bot commented Dec 5, 2020

github-actions bot commented Dec 7, 2020

ashb commented Dec 7, 2020

ashb Dec 7, 2020

ashb Dec 7, 2020 •

edited

	if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then
	if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then

		# The Bash and python commands still should verify the basic connections so they are run after the
		# DB check but before the broker check

Kubernetes worker pod doesn't use docker container entrypoint #12766

Kubernetes worker pod doesn't use docker container entrypoint #12766

Conversation

dimberman commented Dec 2, 2020 • edited

github-actions bot commented Dec 5, 2020

github-actions bot commented Dec 7, 2020

ashb commented Dec 7, 2020

ashb Dec 7, 2020

Choose a reason for hiding this comment

ashb Dec 7, 2020 • edited

Choose a reason for hiding this comment

dimberman commented Dec 2, 2020 •

edited

ashb Dec 7, 2020 •

edited