Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding extra requirements for build and runtime of the PROD image. #16170

Merged
merged 1 commit into from Jun 1, 2021

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented May 30, 2021

This PR adds capability of adding extra requirements to PROD image:

  1. During the build by placing requirements.txt in the
    docker-context-files folder

  2. During execution of the container - by passing
    _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

docs/apache-airflow/start/docker.rst Outdated Show resolved Hide resolved
docs/docker-stack/entrypoint.rst Outdated Show resolved Hide resolved
docs/docker-stack/entrypoint.rst Outdated Show resolved Hide resolved
docs/docker-stack/entrypoint.rst Outdated Show resolved Hide resolved
docs/docker-stack/entrypoint.rst Outdated Show resolved Hide resolved
docs/apache-airflow/start/docker.rst Outdated Show resolved Hide resolved
@kaxil
Copy link
Member

@kaxil kaxil commented May 30, 2021

hmm .. build docs is failing

docs/docker-stack/entrypoint.rst Outdated Show resolved Hide resolved
docs/docker-stack/build.rst Outdated Show resolved Hide resolved
@potiuk
Copy link
Member Author

@potiuk potiuk commented May 30, 2021

fixed

kaxil
kaxil approved these changes May 30, 2021
@github-actions
Copy link

@github-actions github-actions bot commented May 30, 2021

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@mik-laj
Copy link
Member

@mik-laj mik-laj commented Jun 1, 2021

I wonder if we should add new sections to the Helm Chart documentation to better promote this feature. What do you think to create a new page, e.g. FAQ and add the answer to the question "How to install extra pip packages?" ? I mainly think of users migrating from alternative Helm Charts as airflow-helm/airflow or bitnami/airflow has such a section.

@potiuk potiuk force-pushed the add-support-for-additional-requirements branch from fb98222 to a0cfe41 Jun 1, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 1, 2021

I wonder if we should add new sections to the Helm Chart documentation to better promote this feature. What do you think to create a new page, e.g. FAQ and add the answer to the question "How to install extra pip packages?" ? I mainly think of users migrating from alternative Helm Charts as airflow-helm/airflow or bitnami/airflow has such a section.

Good point @mik-laj.

But rather than creating a new section, I extended two sections in the helm chart docs:

  • production deployment - I mentioned typical scenarios when you need custom image and referred to "Build Images" for details

  • quick-start with kind - I copied a few typical examples (adding apt/PypI packages) next to "Adding DAGs" with step-by-step instructions on how to build the images. This would be a gentle introduction to image building by users who do not know how to do it or even do not know that they could build and use their own image easily (plus see-more reference to "Build Images").

I think that hits the sweet-spot between copy/pasting some parts of documentation where users might need it and having common source of examples.

@potiuk potiuk force-pushed the add-support-for-additional-requirements branch 2 times, most recently from 1cc6d88 to 0744dfe Jun 1, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 1, 2021

Last comments please :)

mik-laj
mik-laj approved these changes Jun 1, 2021
@potiuk potiuk force-pushed the add-support-for-additional-requirements branch 3 times, most recently from 2048b3e to 4f55a07 Jun 1, 2021
This PR adds capability of adding extra requirements to PROD image:

1) During the build by placing requirements.txt in the
   ``docker-context-files`` folder

2) During execution of the container - by passing
   _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.
@potiuk potiuk force-pushed the add-support-for-additional-requirements branch from 4f55a07 to 150a48d Jun 1, 2021
@potiuk potiuk merged commit d245992 into apache:main Jun 1, 2021
4 checks passed
@potiuk potiuk deleted the add-support-for-additional-requirements branch Jun 1, 2021
andormarkus pushed a commit to andormarkus/airflow that referenced this issue Jun 5, 2021
…pache#16170)

This PR adds capability of adding extra requirements to PROD image:

1) During the build by placing requirements.txt in the
   ``docker-context-files`` folder

2) During execution of the container - by passing
   _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.
potiuk added a commit to potiuk/airflow that referenced this issue Jun 22, 2021
…pache#16170)

This PR adds capability of adding extra requirements to PROD image:

1) During the build by placing requirements.txt in the
   ``docker-context-files`` folder

2) During execution of the container - by passing
   _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.

(cherry picked from commit d245992)
@tanujdhiman
Copy link

@tanujdhiman tanujdhiman commented Aug 19, 2021

Hello,
How can I use this variable?

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- pydub dropbox}

I'm using this but didn't get good results. I want to install pydub, dropbox through docker compose file. Is it right syntax or what's wrong with it ?
Thanks

@tanujdhiman
Copy link

@tanujdhiman tanujdhiman commented Aug 19, 2021

Thanks @potiuk for a quick reply. But I saw this page actually but umm, okay.

Can you write down a command which I put in compose file for installing pydub, dropbox, patool ??
Thanks

@potiuk
Copy link
Member Author

@potiuk potiuk commented Aug 19, 2021

Just set env variables. Here is the first googled result about them https://opensource.com/article/19/8/what-are-environment-variables#:~:text=Environment%20variables%20contain%20information%20about,during%20installation%20or%20user%20creation.

To be perfectly honest - don't even try docker-compose if you find it difficult to set environment variable when you run a command. this is really basic thing. You should try to undrstand what you do and not ask someone to copy you some solutions to most basic things.

@tanujdhiman
Copy link

@tanujdhiman tanujdhiman commented Aug 19, 2021

Hmm Actually I tried this command too ::

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-pydub dropbox patool}
But got an error of invalid argument in server..

@tanujdhiman
Copy link

@tanujdhiman tanujdhiman commented Aug 19, 2021

Thanks @potiuk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants