Getting Started Help: Unable to import modules in plugins directory

[mivinson]

Hello,

I am hoping I can get help with configuring the community Docker-Compose Airflow environment. I am following the instructions documented here:
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

Using the docker-compose.yaml below.

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.1/docker-compose.yaml'

I am getting the following DAG import error:

The documentation goes through setting up the directories for dags, logs, and plugins that are mounted within the docker containers. However, when I look in the containers I do not see the /opt/airflow directory listed in the PATH variable. To address this, I've extended the image with the following DockerFile:

FROM apache/airflow:2.3.1
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
ENV PATH=${PATH}:/opt/airflow/

However, I am still getting the same error. I can shell into these containers, open python, and import from my modules from here just fine. I do not understand why Airflow is not able to import from this location. Please help.

[potiuk]

You likely have no __init__ in your plugins folder. Also in python PATH is irrelevant. Usually PYTHONPATH is used but this is not needed as Airflow add relevant folders automatically to the sys.path internally.

See https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html where all this is explained in detail. Also you shoudl try to experiment yourself with python imports to understand how they work.

[mivinson]

Hi @potiuk, thank you for your response. This does not resolve my issue. I have followed the directions provided in this document to create the dag, plugins, and logs folder and I also tried putting an init folder in my plugins directory. Since the documentation expects a user to have plugins, I would expect the docker environment to be configured so you can put files in plugins and import from them. However, as I said before I can go into the container and import these modules by performing the following steps.

$ docker exec -it <container> bin/bash
$ python
>> from plugins.hooks.<module_name> import <class_name>

With or without the init in plugins, this will work.

As this is a day 0 issue with the environment, I would expect that other users have encountered a similar issue.

As an aside, I use airflow professionally on Astronomer's environment, which also implements docker. I have a lot of experience with Python and Airflow, but the issue seems to be with the initial condition of the environment from the above docker-compose file. I have launched the Astronomer environment locally and have been successful in running my DAG.

[potiuk]

I have a lot of experience with Python and Airflow

Cool. So maybe you can help to investigate what the reason might be? Airflow has more than 2000 contributors so if you investigate it, it could be great contribution back to get it fixed ?

[potiuk]

BTW. The docker-compose of ours is REALLY a quick-start. It is not "production" ready nor even "development" ready. So we put very little effort on looking at it and solving similar issues (and we put a lot of warnings that you really need to dive-deeply to get anything than basic behaviours. We literally count on people like you to investigate and possibly fix anything more than basic start airflow "as is". Here are relevant paragraphs:

https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#customizing-the-quick-start-docker-compose

Even if many users think of Docker Compose as “ready to use”, it is really a developer tool, that requires the user to know very well how docker images, containers, docker compose networking, volumes, naming, image building works. It is extremely easy to make mistakes that lead to difficult to diagnose problems and if you are not ready to spend your own time on learning and diagnosing and resolving those problems on your own do not follow this path. You have been warned.

If you customize, or modify images, the compose file and see problem do not expect you will get a lot of help with solving those problems in the Airflow support channels. Most of the problems you will experience are Docker Compose related problems and if you need help in solving them, there are dedicated channels in Docker Compose that you can use.

So it's a very low priority for us, but if you help us with solving it - PRs are most welcome.

[potiuk]

So I am not ignoring you, or mocking you. We literally do not put any efforts there, not even to diagnose and reproduce issues other than "just starting airflow" - which we actually check in our CI automatically :

airflow/.github/workflows/ci.yml

Line 1394 in 85c247a

test-docker-compose-quick-start:

Just wanted to set the right expectations here.

[billzhong]

If someone has found this post through search engine and doesn't want to read the Airflow's Modules Management page, here is a solution to the original poster's question.

Airflow adds the dags, config, and plugins directories to the PYTHONPATH environment variable by default. Therefore, to import a custom module, such as AlgoliaToS3Operator in OP's screenshot, you can use the following statement:

from operators.algolia_operator import AlgoliaToS3Operator

This will allow Airflow to find and import the module without requiring any additional configuration.

[manas-shurutech-04]

This isn't resolving the issue

[potiuk]

This isn't resolving the issue

This comment is absolutely useless (including useless to yourself) - because by not providing information about your setup, erors you have and what you tried to solve it, you don't give others chance to help you.