Skip to content

Commit

Permalink
Apply pypi suffix to provider dependencies (including apache-airflow)
Browse files Browse the repository at this point in the history
So far we only applied PyPi suffix for cross-provider dependencies -
i.e. when one airflow package depended on another one released at
the same time we added "dev0" suffix to their cross-dependencies,
when building them on CI so that they can be installed together
without conflicts (this is because of decisions made with PEP-440
where version suffixes are in different namespaces than the final
versions, so it is impossible to specify "final" dependency as
minimal and have the ".dev0" satisfy it only for that single package.

With openlineage provider, we need to release it before Airflow 2.7.0
gets released, and openlineage provider depends on Airflow 2.7.0 so
we need to also handle the situation, where OpenLineage has >= 2.7.0
for Airflow, but the dependency that is used to resolve dependencies
in CI uses 2.7.0dev0. This is done by dynamically manipulating the
dependencies in setup.py based on VERSION_SUFFIX_FOR_PYPI variable.

This variable in CI is set to "dev0" thus all packages built have
"dev0" added as version, with this change if any package has
>= <CURRENT_AIRFLOW_VERSION> specified, it will also be extended
with the same "dev0" suffix.
  • Loading branch information
potiuk committed Jul 27, 2023
1 parent 080d6af commit 4b7e538
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 7 deletions.
2 changes: 1 addition & 1 deletion dev/breeze/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.

---------------------------------------------------------------------------------------------------------

Package config hash: c2bade421865f94fb91f652317d76f3686ad017067349265d8a1d465bbfc2f0796d570bf4777acfb31a7703f1ccb6b0a63c655b97cec9eb9fdeeb1676c804c02
Package config hash: 9d095d522c9f6fcf0c5834fcdc050bc98231d17fad07ec054c4e437580129d547b693b66b61442757f81fc1a505483da5267cc973dbf86babba7cd2c11697708

---------------------------------------------------------------------------------------------------------
42 changes: 36 additions & 6 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,35 @@
CURRENT_PYTHON_VERSION = f"{sys.version_info.major}.{sys.version_info.minor}"


#
def apply_pypi_suffix_to_airflow_packages(dependencies: list[str]) -> None:
"""
Looks through the list of dependencies, finds which one are airflow or airflow providers packages
and applies the version suffix to those of them that do not have the suffix applied yet.
:param dependencies: list of dependencies to add suffix to
"""
for i in range(len(dependencies)):
dependency = dependencies[i]
if dependency.startswith("apache-airflow"):
# in case we want to depend on other airflow package, the chance is the package
# has not yet been released to PyPI and we only see it as a local package that is
# being installed with .dev0 suffix in CI. Unfortunately, there is no way in standard
# PEP-440 compliant way to specify version that would be both - releasable, and
# testable to install on CI with .dev0 or .rc suffixes. We could add `--pre` flag to
# enable it, but `--pre` flag is not selective and will work for all packages so
# we would automatically install all "pre-release" packages for all packages that
# we install from PyPI - and this is definitely not what we want. So in order to
# install only airflow packages that are available in sources in .dev0 or .rc version
# we need to dynamically modify the dependencies here.
if ">=" in dependency:
package, version = dependency.split(">=")
version_spec = f">={version}"
version_suffix = os.environ.get("VERSION_SUFFIX_FOR_PYPI")
if version_suffix and version_suffix not in version_spec:
version_spec += version_suffix
dependencies[i] = f"{package}{version_spec}"


# NOTE! IN Airflow 2.4.+ dependencies for providers are maintained in `provider.yaml` files for each
# provider separately. They are loaded here and if you want to modify them, you need to modify
# corresponding provider.yaml file.
Expand All @@ -74,11 +102,13 @@ def fill_provider_dependencies() -> dict[str, dict[str, list[str]]]:
try:
with AIRFLOW_SOURCES_ROOT.joinpath("generated", "provider_dependencies.json").open() as f:
dependencies = json.load(f)
return {
key: value
for key, value in dependencies.items()
if CURRENT_PYTHON_VERSION not in value["excluded-python-versions"] or skip_python_version_check
}
provider_dict = {}
for key, value in dependencies.items():
if value.get(DEPS):
apply_pypi_suffix_to_airflow_packages(value[DEPS])
if CURRENT_PYTHON_VERSION not in value["excluded-python-versions"] or skip_python_version_check:
provider_dict[key] = value
return provider_dict
except Exception as e:
print(f"Exception while loading provider dependencies {e}")
# we can ignore loading dependencies when they are missing - they are only used to generate
Expand Down

0 comments on commit 4b7e538

Please sign in to comment.