How to run docker command inside docker-compose Airflow2.0?

[alihossein]

I have install Airflow 2.0 from docker-compose.

but I want to some changes and install some python packages.
so I decided to use a helper container inside docker-compose :

 initializer:
    image: apache/airflow:2.0.1
    deploy:
      restart_policy:
        condition: on-failure
    depends_on:
      - airflow-worker
    command: [ "pip3.6 install --no-cache-dir -r requirements.txt"]  
 

but I get this error :

airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'pip3.6 install --no-cache-dir -r requirements.txt' (choose from 'celery', 'cheat-sheet', 'config', 'connections', 'dags', 'db', 'info', 'kerberos', 'kubernetes', 'plugins', 'pools', 'providers', 'roles', 'rotate-fernet-key', 'scheduler', 'sync-perm', 'tasks', 'users', 'variables', 'version', 'webserver'), see help above.
initializer_1        | usage: airflow [-h] GROUP_OR_COMMAND ...

how can I fix this problem? and how can I install my packages inside the image?
thanks.

[potiuk]

The best way is to build and use your own image. Dynamically adding packaged to existing images is a bad idea. See the new docker-stsck documentation for Airflow: https://airflow.apache.org/docs/docker-stack/build.html#a-pypi-package-example

[potiuk]

You could also - potentially - split your your Celery installation into several clusters of workers and have a few Queues with different dependencies added for Airflow (and different images). Then you could mark your Tasks that require certain dependencies to use specific queue. But this is pretty complex to maintain, it's usually better to figure out the "superset" of all dependencies your DAGs need and have one Airflow image to handle them all.

[potiuk]

But yes there is a discussion about potential future implementation where tasks can bring some extra dependencies with them and create a new virtualenv for such task dynamically - but that requires quite sophisticated caching and update mechanism built-in, in order to be maintainable in all configurations (Celery, Local, Kubernetes etc.). This might one day come as a feature to Airflow.

[alihossein]

thanks for the quick answer.
So given these circumstances, if install manually (without Docker) Airflow and Celery, it's not a Good Idea?

if we install manually Airflow On one server, if a new Dag needed new packages, we can install it by pip command on Server.

[potiuk]

Same it image. You rebuild it with new package and push to registry. That's exactly equivalent of installing it on the server, only that if you have distributed workers, they can pull the image (depending on your pull policy). If you have multiple servers and install Airflow separately on each of them, it looks like a lot of completely unnecessary manual work.

[alihossein]

thanks @potiuk