Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add notes about Virtualenvs preventing the need for custom images #35306

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/apache-airflow/howto/docker-compose/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,13 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.

.. note::
Creating custom images means that you need to maintain also a level of automation as you need to re-create the images
when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts.
Also keep in mind, that in cases when you run pure Python tasks, you can use the
`Python Virtualenv functions <_howto/operator:PythonVirtualenvOperator>`_ which will
dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.

Special case - adding dependencies via requirements.txt file
============================================================

Expand Down
7 changes: 7 additions & 0 deletions docs/docker-stack/build.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,13 @@ as ``root`` will fail with an appropriate error message.
downgrade or upgrade of airflow. If you upgrade airflow base image, you should also update the version
to match the new version of airflow.

.. note::
Creating custom images means that you need to maintain also a level of automation as you need to re-create the images
when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts.
Also keep in mind, that in cases when you run pure Python tasks, you can use the
`Python Virtualenv functions <https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_
which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.

.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
:language: Dockerfile
:start-after: [START Dockerfile]
Expand Down
7 changes: 7 additions & 0 deletions docs/helm-chart/quick-start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ The Apache Airflow community, releases Docker Images which are ``reference image
However, when you try it out you want to add your own DAGs, custom dependencies,
packages, or even custom providers.

.. note::
Creating custom images means that you need to maintain also a level of automation as you need to re-create the images
when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts.
Also keep in mind, that in cases when you run pure Python tasks, you can use the
`Python Virtualenv functions <https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_
which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.

The best way to achieve it, is to build your own, custom image.

Adding DAGs to your image
Expand Down