Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 27 additions & 25 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ on:
branches: [ master ]
paths:
- '**.py'
- 'pyproject.toml'
- '!airflow_dbt_python/__version__.py'
tags:
- "v*"
Expand All @@ -13,6 +14,7 @@ on:
- master
paths:
- '**.py'
- 'pyproject.toml'
- '!airflow_dbt_python/__version__.py'

jobs:
Expand Down Expand Up @@ -75,38 +77,37 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Install Poetry
uses: abatilo/actions-poetry@v2.4.0
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
poetry-version: 2.0.1
version: 0.7.2
python-version: ${{ matrix.python-version }}

- name: Install Airflow & dbt
run: |
poetry env use ${{ matrix.python-version }}
poetry add "apache-airflow~=${{ matrix.airflow-version }}.0" \
"dbt-core~=${{ matrix.dbt-version }}.0" \
--python ${{ matrix.python-version }}
poetry install -E adapters --with dev
poetry run airflow db migrate
poetry run airflow connections create-default-connections
uv add "apache-airflow~=${{ matrix.airflow-version }}.0" \
"dbt-core~=${{ matrix.dbt-version }}.0"
uv sync --all-extras --dev
uv run airflow db migrate
uv run airflow connections create-default-connections

- name: Linting with ruff
run: poetry run ruff check .
run: uv run ruff check .

- name: Static type checking with mypy
# We only run mypy on the latest supported versions of Airflow & dbt,
# so it is currently impossible to write conditions for that depend on package versions.
if: matrix.python-version == '3.12' && matrix.airflow-version == '2.10' && matrix.dbt-version == '1.9'
run: poetry run mypy .
run: uv run mypy .

- name: Code formatting with black
run: poetry run black --check .
- name: Code formatting with ruff
run: uv run ruff format --check .
Comment on lines -102 to +104
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also getting rid of black. Sometimes we use ruff, sometimes black. This is just consolidating under one tool.


- name: Set COVERAGE_FILE in environment
run: echo "COVERAGE_FILE=.coverage.${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ matrix.dbt-version }}" >> $GITHUB_ENV

- name: Run tests with pytest
run: poetry run coverage run -m pytest -v tests/ airflow_dbt_python/utils/
run: uv run coverage run -m pytest -v tests/ airflow_dbt_python/utils/
env:
GITLAB_READ_TOKEN: ${{ secrets.GITLAB_READ_TOKEN }}
GITLAB_USERNAME: ${{ secrets.GITLAB_USERNAME }}
Expand Down Expand Up @@ -141,13 +142,14 @@ jobs:
with:
python-version: '3.12'

- name: Install Poetry
uses: abatilo/actions-poetry@v2.4.0
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
poetry-version: 2.0.1
version: 0.7.2
python-version: ${{ matrix.python-version }}

- name: Install airflow-dbt-python with Poetry
run: poetry install --with dev -E airflow-providers
- name: Install airflow-dbt-python with uv
run: uv sync --dev --extra airflow-providers

- name: Download coverage data.
uses: actions/download-artifact@v4
Expand All @@ -156,16 +158,16 @@ jobs:

- name: Combine coverage & fail if it's <95%.
run: |
poetry run coverage combine
poetry run coverage html --skip-covered --skip-empty
poetry run coverage json
uv run coverage combine
uv run coverage html --skip-covered --skip-empty
uv run coverage json
# Save in env variable for badge
export TOTAL=$(python -c "import json;print(json.load(open('coverage.json'))['totals']['percent_covered_display'])")
echo "total=$TOTAL" >> $GITHUB_ENV
# Report and write to summary.
poetry run coverage report | sed 's/^/ /' >> $GITHUB_STEP_SUMMARY
uv run coverage report | sed 's/^/ /' >> $GITHUB_STEP_SUMMARY
# Report again and fail if under 100%.
poetry run coverage report --fail-under=95
uv run coverage report --fail-under=95

- name: Upload HTML report if check failed.
uses: actions/upload-artifact@v4
Expand Down
13 changes: 7 additions & 6 deletions .github/workflows/docs_pages.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,14 @@ jobs:
with:
python-version: '3.12'

- name: Install Poetry
uses: abatilo/actions-poetry@v2.4.0
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
poetry-version: 2.0.1
version: 0.7.2
python-version: ${{ matrix.python-version }}

- name: Install airflow-dbt-python with Poetry
run: poetry install -E airflow-providers --with docs
- name: Install airflow-dbt-python with uv
run: uv sync --dev --extra airflow-providers

- name: Install Graphviz
run: sudo apt-get install graphviz
Expand All @@ -34,7 +35,7 @@ jobs:
mkdir gh-pages
touch gh-pages/.nojekyll
cd docs/
poetry run sphinx-build -b html . _build
uv run sphinx-build -b html . _build
cp -r _build/* ../gh-pages/

- name: Deploy documentation
Expand Down
25 changes: 12 additions & 13 deletions .github/workflows/pypi_deploy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,22 @@ jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Set up Python 3.12
uses: actions/setup-python@v5.4.0
- uses: actions/checkout@v4.2.2
- uses: actions/setup-python@v5.4.0
with:
python-version: '3.12'

- name: Install Poetry
uses: abatilo/actions-poetry@v2.4.0
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
poetry-version: 2.0.1
version: 0.7.2
python-version: 3.12

- name: Install airflow-dbt-python with Poetry
run: poetry install
- name: Install airflow-dbt-python with uv
run: uv sync --no-dev

- name: Build airflow-dbt-python with Poetry
run: poetry build
- name: Build airflow-dbt-python with uv
run: uv build

- name: Publish airflow-dbt-python to PyPI with Poetry
run: poetry publish --username __token__ --password ${{ secrets.PYPI_API_TOKEN }}
- name: Publish airflow-dbt-python to PyPI with uv
run: uv publish --token ${{ secrets.PYPI_API_TOKEN }}
28 changes: 13 additions & 15 deletions .github/workflows/tagged_release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,33 +26,31 @@ jobs:
echo "CI failed or didn't complete in time"
exit 1

- name: Checkout
uses: actions/checkout@v2.3.4

- name: Set up Python 3.10
uses: actions/setup-python@v5.4.0
- uses: actions/checkout@v4.2.2
- uses: actions/setup-python@v5.4.0
with:
python-version: '3.10'
python-version: '3.12'

- name: Install Poetry
uses: abatilo/actions-poetry@v2.4.0
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
poetry-version: 2.0.1
version: 0.7.2
python-version: 3.12

- name: Install airflow-dbt-python with Poetry
run: poetry install
- name: Install airflow-dbt-python with uv
run: uv sync --no-dev

- name: Build airflow-dbt-python with Poetry
run: poetry build
- name: Build airflow-dbt-python with uv
run: uv build

- name: Set prerelease variable
run: echo "IS_PRERELEASE=$(if $(poetry version -s | grep -qP '^[0-9]+\.[0-9]+\.[0-9]+$'); then echo 'false'; else echo 'true'; fi)" >> $GITHUB_ENV
run: echo "IS_PRERELEASE=$(if $(uv version --short | grep -qP '^[0-9]+\.[0-9]+\.[0-9]+$'); then echo 'false'; else echo 'true'; fi)" >> $GITHUB_ENV

- name: Set release git tag
run: echo "RELEASE_TAG=${{ github.ref_name }}" >> $GITHUB_ENV

- name: Check tag matches package version
run: if [[ "$(poetry version -s)" == ${RELEASE_TAG/v/} ]]; then exit 0; else exit 1; fi
run: if [[ "$(uv version --short)" == ${RELEASE_TAG/v/} ]]; then exit 0; else exit 1; fi

- name: Release new tag
uses: softprops/action-gh-release@v1
Expand Down
6 changes: 2 additions & 4 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ build:
- graphviz
jobs:
pre_install:
- python -m pip install poetry==2.0.1
- poetry config virtualenvs.create false --local
- poetry install -E git -E s3 --with docs
- poetry export -E git -E s3 --with docs -f requirements.txt --without-hashes --output docs/requirements.txt
- python -m pip install uv
- uv export --extra git --extra s3 --group docs --format requirements.txt --no-hashes --output-file docs/requirements.txt

python:
install:
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,10 @@ git clone https://github.com/tomasfarias/airflow-dbt-python.git
cd airflow-dbt-python
```

And installing with *Poetry*:
And installing with *uv*:

``` shell
poetry install
uv pip install .
```

## In AWS MWAA
Expand Down Expand Up @@ -225,10 +225,10 @@ See the [development documentation](https://airflow-dbt-python.readthedocs.io/en

## Testing

Tests are run with *pytest*, can be located in `tests/`. To run them locally, you may use *Poetry*:
Tests are run with *pytest*, can be located in `tests/`. To run them locally, you may use *uv*:

``` shell
poetry run pytest tests/ -vv
uv run pytest tests/ -vv
```

# License
Expand Down
43 changes: 18 additions & 25 deletions docs/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,10 @@ This section dives into the development process of *airflow-dbt-python* by descr

If you would like to dig a bit into the internals of *airflow-dbt-python* and learn about the development philosophy, it's recommended that you check out the :ref:`how_does_it_work` documentation.

Project management: Poetry
Project management: *uv*
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parts of this doc are somewhat out of date. I've done some changes here, but a few more are required, I think.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of the pain that came with resolving dependencies really goes away with uv.

--------------------------

*airflow-dbt-python* uses `Poetry <https://python-poetry.org/>`_ for project management. See `Poetry's installation documentation <https://python-poetry.org/docs/#installation>`_ if you need to install it first.

As of *airflow-dbt-python* version 0.14, we have moved the project to Poetry version >= 1.2.0 to allow us to use dependency groups. Ensure your installation meets this requirement.
*airflow-dbt-python* uses `uv <https://docs.astral.sh/uv/>`_ for project management. See `uv's installation documentation <https://docs.astral.sh/uv/getting-started/installation/>`_ if you need to install it first.

Installing *airflow-dbt-python*
-------------------------------
Expand All @@ -24,33 +22,29 @@ Clone the *airflow-dbt-python* repository:
git clone https://github.com/tomasfarias/airflow-dbt-python.git
cd airflow-dbt-python

And install it with *Poetry*:
Create a local virtual environment (if you don't want *uv* to manage it), and ensure your local development environment is in sync with *uv*:

.. code-block:: shell

poetry install --with dev -E airflow-providers -E postgres
uv sync --dev --extra airflow-providers --extra postgres

The *dev* dependency group includes development tools for code formatting, type checking, and testing.

The additional extras, *airflow-providers*, *postgres*, and *airflow* install dependencies required for testing. If testing a specific Airflow version, the extras may be ommitted, see the following section with more details.

Installing a different version of Airflow
-----------------------------------------
The additional extras, *airflow-providers* and *postgres* install dependencies required for testing. If testing a specific Airflow version, the extras may be omitted, see the following section with more details.

The *airflow* extra dependency group installs the latest supported version of Airflow. However, *airflow-dbt-python* is also tested for older versions of Airflow; as a general rule, besides the latest version of Airflow, we test *airflow-dbt-python* against the latest version available in `AWS MWAA <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_. Although support for older versions of Airflow is not guaranteed, we are open to patches to ensure backwards compatibility as long as they don't increase maintenance load signifincantly.
Support for different versions of *Airflow*
-------------------------------------------

If you wish to install a different version of Airflow for testing you may skip the *airflow* and *airflow-providers* extras of the previous section and use *pip* instead to install any versions of *apache-airflow* and required providers:

.. code-block:: shell
*airflow-dbt-python* supports and is tested with multiple versions of Airflow; as a general rule, besides the latest version of Airflow, we test *airflow-dbt-python* against the latest version available in `AWS MWAA <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_, which usually lags behind a few minor versions. We are open to patches to improve backwards compatibility as long as they don't increase maintenance load significantly.

pip install apache-airflow>=2.2 apache-airflow-providers-amazon>=3.0
If you wish to install a different version of Airflow for testing you may skip the *airflow-providers* extras of the previous section and use *pip* instead to install any versions of *apache-airflow* and required providers.

Modifying dependencies
----------------------

Apache Airflow is a package of significant size that requires a lot of dependencies. Together with ``dbt-core``, it's common to find dependency conflicts all over the place. Ultimately, we allow users to figure these issues out themselves, as most of the dependency conflicts are harmless: We do not interact with the dbt CLI, so any conflicts with CLI-specific packages can be safely ignored, but these requirements are not optional for ``dbt-core``.
Apache Airflow is a package of significant size that requires a lot of dependencies. Together with *dbt-core*, it's common to find dependency conflicts all over the place. Ultimately, we allow users to figure these issues out themselves, as most of the dependency conflicts are harmless: We do not interact with the *dbt* CLI, so any conflicts with CLI-specific packages can be safely ignored, but these requirements are not optional for *dbt-core*.

All being said, this presents a problem when we try to add dependencies or modify existing ones, as ``poetry`` will take a long time to check all possible conflicts. Grabbing a constraints file from `Airflow <https://github.com/apache/airflow>`_ and adding it as an optional group in ``pyproject.toml`` can be a useful strategy to speed up the process, as we are limiting the search domain for ``poetry``. However, this does require relaxing dependencies one by one in between running ``poetry lock --no-update``.
All being said, this presents a problem when we try to add dependencies or modify existing ones. Grabbing a constraints file from `Airflow <https://github.com/apache/airflow>`_ and adding it as an optional group in ``pyproject.toml`` can be a useful strategy.

Pre-commit hooks
----------------
Expand All @@ -62,22 +56,21 @@ These hooks include:
* Trailing whitespace trimming.
* Ensure EOF newline.
* Detect secrets.
* Code formatting (`black <https://github.com/psf/black>`_).
* PEP8 linting (`flake8 <https://github.com/pycqa/flake8/>`_).
* Code formatting (`ruff <https://github.com/astral-sh/ruff>`_).
* Linting (`ruff <https://github.com/astral-sh/ruff>`_).
* Static type checking (`mypy <https://github.com/python/mypy>`_).
* Import sorting (`isort <https://github.com/PyCQA/isort>`_).

All *pre-commit* hooks can be installed with:

.. code-block:: shell

pre-commit install

Alternatively, all of the aforementioned tools are installed with the *dev* dependency group and can ran individually, without *pre-commit* hooks. For example, to format files with *black*:
Alternatively, all of the aforementioned tools are installed with the *dev* dependency group and can ran individually, without *pre-commit* hooks. For example, to format files with *ruff*:

.. code-block:: shell

poetry run black airflow_dbt_python/
uv run ruff format airflow_dbt_python/

Testing *airflow-dbt-python*
----------------------------
Expand All @@ -97,7 +90,7 @@ An Airflow database needs to be initialized in your local environment. This requ
.. code-block:: shell

export AIRFLOW_HOME=$PWD
poetry run airflow db migrate
uv run airflow db migrate

The ``AIRFLOW_HOME`` environment variable has to be set to the same value used when initializing the database for most testing commands, so it's recommended to ``export`` it.

Expand All @@ -114,7 +107,7 @@ All unit tests can be run with:

.. code-block:: shell

poetry run pytest tests/ airflow_dbt_python/ -vv
uv run pytest tests/ airflow_dbt_python/ -vv

The majority of tests are found in the ``tests/`` directory, but we also test `doctest <https://docs.python.org/3.10/library/doctest.html>`_ documentation examples.

Expand All @@ -125,7 +118,7 @@ Generating coverage reports with *coverage.py* can be done with:

.. code-block:: shell

poetry run coverage run -m pytest tests/ airflow_dbt_python/
uv run coverage run -m pytest tests/ airflow_dbt_python/

Unit tests and DAG tests
^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
Loading