Skip to content

Fix version_added for the concurrent pod creation options - #71157

Open
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:fix-k8s-async-pod-creation-version-added
Open

Fix version_added for the concurrent pod creation options#71157
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:fix-k8s-async-pod-creation-version-added

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The cncf.kubernetes options [kubernetes_executor] async_pod_creation and
pod_creation_max_concurrency are tagged version_added: 10.20.0, but they ship in
10.21.0. I added both in #68480 and got the version
wrong.

Two consequences:

  • The 10.21.0 configuration reference renders "New in version 10.20.0" for both options,
    pointing users at a release where they do not exist.
  • version_added is also what keeps a newer option off an older version's config page:
    conf_constants.py drops any option whose version_added is greater than the package
    version being built. At 10.20.0 the filter does not drop these two, so a 10.20.0 docs
    build lists options the 10.20.0 code has never heard of.

Found while verifying cncf-kubernetes 10.21.0rc1 for
#70953. Three other options in the same section
carry version_added: 10.20.0 legitimately; only these two are wrong.

The value was right when I wrote it and went stale in review. 10.19.0 was the latest
release when the PR was opened on 2026-06-12, so 10.20.0 was the next version. 10.20.0 was
then cut on 2026-07-23 while the PR was still open, and the PR merged six days later, which
put it in 10.21.0. version_added is hand-written and nothing revalidates it: the schema
accepts any string, no pre-commit or breeze check compares it against the version being
released, and the versions: list only gains its entry at release-prep time. Any PR that
adds a config option and stays open across a release boundary hits this.

related: #70953

Testing done

The two options do not exist anywhere in the released 10.20.0 provider, and do exist in
10.21.0rc1. Both wheels installed from PyPI / the RC SVN dist into separate virtualenvs.

Raw logs
$ .venv/bin/python -c "import importlib.metadata as m; print(m.version('apache-airflow-providers-cncf-kubernetes'))"
10.21.0rc1
$ .venv-prev/bin/python -c "import importlib.metadata as m; print(m.version('apache-airflow-providers-cncf-kubernetes'))"
10.20.0

# every option in the k8s provider.yaml claiming version_added: 10.20.0
$ git show providers-cncf-kubernetes/10.21.0rc1:providers/cncf/kubernetes/provider.yaml > /tmp/rc1_provider.yaml
$ python3 -c "
import yaml
d = yaml.safe_load(open('/tmp/rc1_provider.yaml'))
for sec, body in d['config'].items():
    for name, opt in body['options'].items():
        if opt.get('version_added') == '10.20.0':
            print(f'{sec}.{name}')
"
kubernetes_executor.running_pod_log_lines
kubernetes_executor.async_pod_creation
kubernetes_executor.pod_creation_max_concurrency
kubernetes_executor.pod_launch_failure_retries
kubernetes_executor.pod_launch_failure_excluded_container_reasons

# are those five actually present in the released 10.20.0 provider?
$ for opt in async_pod_creation pod_creation_max_concurrency running_pod_log_lines \
             pod_launch_failure_retries pod_launch_failure_excluded_container_reasons; do
    n=$(grep -rl "$opt" .venv-prev/lib/python3*/site-packages/airflow/providers/cncf/kubernetes/ | wc -l)
    echo "10.20.0  $opt -> $n file(s)"
  done
10.20.0  async_pod_creation -> 0 file(s)
10.20.0  pod_creation_max_concurrency -> 0 file(s)
10.20.0  running_pod_log_lines -> 4 file(s)
10.20.0  pod_launch_failure_retries -> 4 file(s)
10.20.0  pod_launch_failure_excluded_container_reasons -> 4 file(s)

# same two options in 10.21.0rc1
$ for opt in async_pod_creation pod_creation_max_concurrency; do
    n=$(grep -rl "$opt" .venv/lib/python3*/site-packages/airflow/providers/cncf/kubernetes/ | wc -l)
    echo "10.21.0rc1  $opt -> $n file(s)"
  done
10.21.0rc1  async_pod_creation -> 7 file(s)
10.21.0rc1  pod_creation_max_concurrency -> 8 file(s)
How the value went stale
# latest release when the PR was opened on 2026-06-12
$ git show 8e63d9bbc77^:providers/cncf/kubernetes/provider.yaml | grep -A3 '^versions:'
versions:
  - 10.19.0
  - 10.18.0
  - 10.17.1

# 10.20.0 was cut while the PR was open, and the PR merged after it
$ git log -1 --format=%cI 8e63d9bbc77          # the commit adding "- 10.20.0" to versions:
2026-07-23T22:48:43+03:00
$ gh pr view 68480 --repo apache/airflow --json createdAt,mergedAt
created: 2026-06-12T19:09:23Z
merged:  2026-07-29T09:58:47Z
$ git merge-base --is-ancestor 8e63d9bbc77 703a0570c84 && echo "merged after 10.20.0 was cut"
merged after 10.20.0 was cut

# nothing validates the field
$ grep -rn version_added dev/breeze/src/ scripts/ | grep -viE 'jinja|template|\.rst'
(no output)
$ python3 -c "import json; print(json.load(open('airflow-core/src/airflow/provider.yaml.schema.json'))['definitions']['option']['properties']['version_added'])"
{'type': ['string', 'null']}

get_provider_info.py carries the same field and is generated from provider.yaml by
breeze at package-prep time (_prepare_get_provider_info_py_file in
dev/breeze/src/airflow_breeze/utils/packages.py), so both are updated together.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: GitHub Copilot CLI following the guidelines

async_pod_creation and pod_creation_max_concurrency are tagged
version_added: 10.20.0, but they ship in cncf-kubernetes 10.21.0. Neither
name appears anywhere in the released 10.20.0 distribution.

The docs build turns this into a ".. versionadded:: 10.20.0" directive on
the configuration reference page, so a reader planning an upgrade is
pointed at a release where neither option exists.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant