Update cncf's import conf path to use common compat SDK by wjddn279 · Pull Request #64143 · apache/airflow

wjddn279 · 2026-03-24T06:39:16Z

related: #60000

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

potiuk · 2026-03-24T12:09:53Z

@wjddn279 This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

❌ Provider tests: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:, Non-DB tests: providers / Non-DB-prov::3.10:amazon...google. Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.
❌ Kubernetes tests: Failing: Low dep tests: providers / All-prov:LowestDeps:14:3.10:cncf.kubernetes. See the K8s testing documentation. See Kubernetes tests docs.

What to do next:

The comment informs you what you need to do.
Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

wjddn279 · 2026-03-24T13:40:45Z

@jason810496

Hi Jason. As intended by this PR, I removed the fallback from the existing logic. Currently, the compat tests for the previous version are failing — is that expected? I believe it's because the older version doesn't have the functionality to read from provider_config_fallback_defaults.cfg.

Am I misunderstanding something?

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_config.py

jscheffl

OK for me, in other opinion please let me know, else propose to merge the next days.

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/template_rendering.py

wjddn279 · 2026-03-27T07:18:12Z

@jason810496

I think the review has resolved and there is no issue! Can you review once more?

jason810496

Yes, I checked the three key where the fallback was removed (multi_namespace_mode_namespace_list, pod_template_file, namespace) and the defaults are explicitly specified in providers.yaml and the values also exist in provider_config_fallback_defaults.cfg.

I just double checked the options you mentioned as well, yes, they're specified in the provider.yaml.
Thanks for helping out, LGTM.

EDIT: we can't remove the exiting fallback as metioned in the latest review comment.

jason810496

Wait, I realized that we still need those fallback to keep it compatible or we might cause breaking change for users who using older Airflow in the last minute.

Let me explain why: users could still use latest providers in "older" Airflow Core (e.g. Airflow 3.1 or 2.11), for the conf in older Airflow version, they don't respect the provider.yaml.

jscheffl · 2026-03-28T10:07:06Z

Wait, I realized that we still need those fallback to keep it compatible or we might cause breaking change for users who using older Airflow in the last minute.

Let me explain why: users could still use latest providers in "older" Airflow Core (e.g. Airflow 3.1 or 2.11), for the conf in older Airflow version, they don't respect the provider.yaml.

That is the reason why we have the "common.compat" provider - to abstract the Airflow backend version away. So as the change is made and depednency in modelled in provider.yaml this should be fine.

That is alo the reason why we have back-compat tests in CI

jason810496

Wait, I realized that we still need those fallback to keep it compatible or we might cause breaking change for users who using older Airflow in the last minute.
Let me explain why: users could still use latest providers in "older" Airflow Core (e.g. Airflow 3.1 or 2.11), for the conf in older Airflow version, they don't respect the provider.yaml.

That is the reason why we have the "common.compat" provider - to abstract the Airflow backend version away. So as the change is made and depednency in modelled in provider.yaml this should be fine.

That is alo the reason why we have back-compat tests in CI

However, the conf will only respect the provider.yaml in the up coming 3.2.0 ( Make TaskSDK conf respect default config from provider metadata #62696 )

On top of this, if we replace the existing conf.get(..., fallback="default-value") with conf.get(...), which might cause breaking changes for users who using older Airflow backend version upgrade to next provider version.

wjddn279 · 2026-03-30T00:58:16Z

@jason810496

I've done reverting the fallback! Maybe looks good and CI is green!

jason810496 · 2026-03-30T03:46:02Z

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_config.py

        self.dags_folder = self._conf.get(self.core_section, "dags_folder")
        self.parallelism = self._conf.getint(self.core_section, "parallelism")
-        self.pod_template_file = self._conf.get(self.kubernetes_section, "pod_template_file", fallback=None)
+        self.pod_template_file = self._conf.get(self.kubernetes_section, "pod_template_file", fallback="")


Should we fallback to None instead of "" here?

After testing, I found that when the fallback is triggered and the value is None, it causes an error in the tests. It's this part.

airflow/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/cli/kubernetes_command.py

Line 109 in 6d8ccc6

base_worker_pod=PodGenerator.deserialize_model_file(kube_config.pod_template_file),

This PR (#64209) likely eliminates the case where the fallback is triggered in test, but you can think of this change as a fix for the case where None is passed in and causes an error.

Without changing it to a blank string, it currently throws an error as-is.

Thanks for the clarification, then I will prefer to defer this one until #64209 is merged. Let's see if your current change rebase on top of #64209 without any fallback changes will pass or not.

#64209 was merged yesterday, could you please rebase when you have a moment? Thanks.

wjddn279 requested review from hussein-awala, jedcunningham and jscheffl as code owners March 24, 2026 06:39

boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Mar 24, 2026

jason810496 force-pushed the update-k8s-import-conf branch from b27844e to 9346cd5 Compare March 24, 2026 08:57

potiuk marked this pull request as draft March 24, 2026 12:09

wjddn279 force-pushed the update-k8s-import-conf branch from f899ff3 to 832f278 Compare March 24, 2026 12:49

wjddn279 force-pushed the update-k8s-import-conf branch 2 times, most recently from da1e18a to 62f7102 Compare March 25, 2026 05:48

uranusjr reviewed Mar 25, 2026

View reviewed changes

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

uranusjr reviewed Mar 25, 2026

View reviewed changes

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_config.py Outdated Show resolved Hide resolved

wjddn279 marked this pull request as ready for review March 25, 2026 07:33

wjddn279 force-pushed the update-k8s-import-conf branch from 9eeab38 to 7cee6d1 Compare March 25, 2026 09:40

jscheffl approved these changes Mar 25, 2026

View reviewed changes

Nataneljpwd reviewed Mar 26, 2026

View reviewed changes

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/template_rendering.py Show resolved Hide resolved

Nataneljpwd approved these changes Mar 27, 2026

View reviewed changes

jason810496 approved these changes Mar 27, 2026

View reviewed changes

jason810496 requested changes Mar 27, 2026

View reviewed changes

wjddn279 force-pushed the update-k8s-import-conf branch from 7cee6d1 to 045e075 Compare March 28, 2026 12:13

jason810496 reviewed Mar 28, 2026

View reviewed changes

jason810496 reviewed Mar 30, 2026

View reviewed changes

wjddn279 added 3 commits March 31, 2026 10:51

Update cncf's import conf path to use common compat SDK

49fff15

fix fallback

b11e33c

rollback fallback

cfb6028

wjddn279 force-pushed the update-k8s-import-conf branch from 045e075 to cfb6028 Compare March 31, 2026 01:51

Conversation

wjddn279 commented Mar 24, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

potiuk commented Mar 24, 2026

Uh oh!

wjddn279 commented Mar 24, 2026

Uh oh!

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wjddn279 commented Mar 27, 2026

Uh oh!

jason810496 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

wjddn279 commented Mar 30, 2026

Uh oh!

jason810496 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

wjddn279 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

wjddn279 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

jason810496 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

jason810496 Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jason810496 left a comment •

edited

Loading

jscheffl commented Mar 28, 2026 •

edited

Loading