Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 35 additions & 40 deletions contributing-docs/07_local_virtualenv.rst
Original file line number Diff line number Diff line change
Expand Up @@ -267,57 +267,52 @@ for the provider is as simple as running:

uv run pytest

Installing "golden" version of dependencies
-------------------------------------------
Locked versions of dependencies
-------------------------------

Whatever virtualenv solution you use, when you want to make sure you are using the same
version of dependencies as in main, you can install recommended version of the dependencies by using pip:
constraint-python<PYTHON_MAJOR_MINOR_VERSION>.txt files as ``constraint`` file. This might be useful
to avoid "works-for-me" syndrome, where you use different version of dependencies than the ones
that are used in main, CI tests and by other contributors.
The ``uv.lock`` file is committed to the Airflow repository and is used by ``uv sync`` to ensure
consistent dependency versions across all developers. When you run ``uv sync``, it uses the lock file
to install exact dependency versions, so you don't need to pass constraint files manually.

There are different constraint files for different python versions. For example this command will install
all basic devel requirements and requirements of google provider as last successfully tested for Python 3.10:

.. code:: bash

uv pip install -e ".[devel,google]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.10.txt"
The ``uv sync`` command prefers the locked versions of dependencies from ``uv.lock``. It will only
attempt to resolve new dependencies when ``pyproject.toml`` files change (e.g. when a new dependency
is added or version bounds are modified). This means that day-to-day ``uv sync`` is fast and
deterministic — it simply installs what the lock file specifies without re-resolving the dependency
tree.

If you want to make sure that ``uv sync`` does not update your lock file at all (for example in CI
or when running tests), you can pass the ``--frozen`` flag:

In the future we will utilise ``uv.lock`` to manage dependencies and constraints, but for the moment we do not
commit ``uv.lock`` file to Airflow repository because we need to figure out automation of updating the ``uv.lock``
very frequently (few times a day sometimes). With Airflow's 700+ dependencies it's all but guaranteed that we
will have 3-4 changes a day and currently automated constraints generation mechanism in ``canary`` build keeps
constraints updated, but for ASF policy reasons we cannot update ``uv.lock`` in the same way - but work is in
progress to fix it.

Make sure to use latest main for such installation, those constraints are "development constraints" and they
are refreshed several times a day to make sure they are up to date with the latest changes in the main branch.
.. code:: bash

Note that this might not always work as expected, because the constraints are not always updated
immediately after the dependencies are updated, sometimes there is a very recent change (few hours, rarely more
than a day) which still runs in ``canary`` build and constraints will not be updated until the canary build
succeeds. Usually what works in this case is running your install command without constraints.
uv sync --frozen

You can upgrade just airflow, without paying attention to provider's dependencies by using
the 'constraints-no-providers' constraint files. This allows you to keep installed provider dependencies
and install to latest supported ones by pure Airflow core.
This will fail if the lock file is out of date with respect to ``pyproject.toml``, rather than
silently updating it. This is useful when you want to guarantee fully reproducible environments.

.. code:: bash
Cooldown via ``exclude-newer``
..............................

uv pip install -e ".[devel]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.10.txt"
The ``[tool.uv]`` section in the top-level ``pyproject.toml`` sets ``exclude-newer = "4 days"``.
This acts as a cooldown period — when ``uv`` resolves new dependencies, it ignores package versions
released in the last 4 days. This protects against broken or yanked releases that might otherwise
immediately break the dependency resolution for all developers. When ``uv`` writes the lock file, it
records the resolved ``exclude-newer`` timestamp so that subsequent ``uv sync`` calls use the same
cutoff, ensuring consistency across machines.

These are examples of the development options available with the local virtualenv in your IDE:
Constraints generated from the lock file
.........................................

* local debugging;
* Airflow source view;
* auto-completion;
* documentation support;
* unit tests.
Airflow also publishes traditional ``pip``-style constraint files (see
`Airflow dependencies and extras <13_airflow_dependencies_and_extras.rst>`_ for details). When
installing Airflow from sources, these constraint files are generated directly from ``uv.lock`` using
``uv export --frozen``, which converts the lock file into a flat list of pinned versions suitable for
``pip install --constraint``. This ensures that both the ``uv sync`` workflow and the ``pip`` constraint
workflow install the same dependency versions.

This document describes minimum requirements and instructions for using a standalone version of the local virtualenv.
The lock file is updated regularly — whenever dependencies are changed via any ``pyproject.toml`` and
when ``breeze ci upgrade`` is run. Make sure to use the latest main branch to get the most
up-to-date ``uv.lock``.

Running Tests
-------------
Expand Down
11 changes: 8 additions & 3 deletions contributing-docs/13_airflow_dependencies_and_extras.rst
Original file line number Diff line number Diff line change
Expand Up @@ -318,17 +318,17 @@ example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)

There are several sets of constraints we keep:

* 'constraints' - these are constraints generated by matching the current Airflow version from sources
* ``constraints`` - these are constraints generated by matching the current Airflow version from sources
and providers that are installed from PyPI. Those are constraints used by the users who want to
install Airflow with pip, they are named ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt``.

* "constraints-source-providers" - these are constraints generated by using providers installed from
* ``constraints-source-providers`` - these are constraints generated by using providers installed from
current sources. While adding new providers their dependencies might change, so this set of providers
is the current set of the constraints for Airflow and providers from the current main sources.
Those providers are used by CI system to keep "stable" set of constraints. They are named
``constraints-source-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``

* "constraints-no-providers" - these are constraints generated from only Apache Airflow, without any
* ``constraints-no-providers`` - these are constraints generated from only Apache Airflow, without any
providers. If you want to manage Airflow separately and then add providers individually, you can
use them. Those constraints are named ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``.

Expand Down Expand Up @@ -375,6 +375,11 @@ using ``constraints-no-providers`` constraint files as well.
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.10.txt"


These constraint files are generated from the ``uv.lock`` file committed in the repository, using
``uv export --frozen`` to convert the lock file into a flat list of pinned versions suitable for
``pip install --constraint``. This means the constraint files always reflect the same dependency
versions that ``uv sync`` installs for developers.

The ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt`` and ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``
will be automatically regenerated by CI job every time after the ``pyproject.toml`` is updated and pushed
if the tests are successful.
Expand Down
13 changes: 11 additions & 2 deletions dev/breeze/src/airflow_breeze/utils/run_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,16 @@ def verify_an_image(
if slim_image:
env["TEST_SLIM_IMAGE"] = "true"
command_result = run_command(
["uv", "run", "--isolated", "pytest", test_path.as_posix(), *pytest_args, *extra_pytest_args],
[
"uv",
"run",
"--frozen",
"--isolated",
"pytest",
test_path.as_posix(),
*pytest_args,
*extra_pytest_args,
],
env=env,
output=output,
check=False,
Expand Down Expand Up @@ -177,7 +186,7 @@ def run_docker_compose_tests(
env["INCLUDE_SUCCESS_OUTPUTS"] = "true"
env["AIRFLOW_UID"] = str(os.getuid())
command_result = run_command(
["uv", "run", "pytest", *pytest_args],
["uv", "run", "--frozen", "pytest", *pytest_args],
env=env,
check=False,
cwd=cwd,
Expand Down
Loading