Replies: 1 comment 2 replies
-
|
Your virtualenv cannot import I believe mounting venv created in Host and using them inside container is generally very bad idea. You can do it, sure, but if you have problems like that you are entirely on your own to solve any problems like that because this is absolutely non-standard, not supported by Python as a language and it will likely have side effects that you won't be able to fix. Also It's quite against the whole idea of using containers that are supposed to be immutable. If you want to really easily modify the virtual envs I recommend you to use the standard built-in functionality of docker compose which allows you to rebuild your image and use it automatically https://github.com/apache/airflow/pull/29598/files is a recently merged PR that will make it in the docs that explains how to do it. You will save a lot of time by doing it - not only yours but people who you are asking for help you to get this very non-standard workflow to work. |
Beta Was this translation helpful? Give feedback.
-
|
1.) @potiuk : " guess the problem is that you do not have it installed in the target env. " 1.) @stromal : I mention above "I can jump in to the container activate manually the python environment import and run python libraries perfectly fine." -> It is installed based on the same requirements. txt that the other Dockerfile base venv1 gets generated that also works and runs this DAG, and once agin I can run thsi Go function inside the dokcer container manually activating the venv2 than 2.) @potiuk : "If you want to really easily modify the virtual envs I recommend you to use the standard built-in functionality of docker compose which allows you to rebuild your image and use it automatically https://github.com/apache/airflow/pull/29598/files is a recently merged PR that will make it in the docs that explains how to do it. " 2.) @stromal : "The DAG works with the original external python environment that is installed via the Dockerfile " -> Yes this is how exactly now I do it but it. And I have explained it above in my original post. It takes about 13 min to build the new images -> than turning down the current container -> starting a new container -> in total this is 20 min it takes too long and too much down time that I cant afford. I also do not want to implement any complex Blue-Green ETLs with Airflow as it requires way too much DevOps work, instead I want a much simpler way just as I have explain it above: just have a simple venv2 that is attached as a volume and that is it. |
Beta Was this translation helpful? Give feedback.
-
|
Well - then you need to dig deeper. It's your problem to solve, as clearly stated above, your seting is not standard and you will keep on having similar (and other) problems. You've been warned. There are mulitple ways how you can speed up docker rrebuilding - for example using cache and splitting requirements into "first" stage and then adding things that I would iterate on: And that's just manual way of doing it. The better way is to used buildkit's cache mounts if you want to cache installed files between runs on the same machine (which you do if you try to mount it): https://pythonspeed.com/articles/docker-cache-pip-downloads/ - then you can easily cache the installation files between runs and building the same docker image between runs will take you seconds not minutes. You can read about it in the best practices of how to speed up your builds : https://docs.docker.com/build/cache/ . Strongly recommend. Seems you have chosen the "hard" way of doing it and you need to deal with the consequnces. And I think you are looking in a completely wrong place (if you want to still continue doing the hard way). The main difference between the way how task is run and the way how you are running it manually is likely because when task is run as part of the Docker container it is run in non-interactive shell vs. the interactive one, and what you see as the diference between your manual checks an tasks run via airflow is caused by different initializaiton of env variables that are happening in those two cases. Also airflow performs daemonisation usually when you start it as part of the container task (depending on your configuration). That might also make a difference. You can read about interactive vs. non-interactive shells (in case you do not realise there is a difference) here: Another reason migh also be because you are using different user to run it (and that user might not have access to all the mounted volume files). Mounting volumes and accessing them from within the container might have multiple problems - depending of they way it is mounted, the operating system you run it on and even container engine that you use. For example rancher or colima on MacOS use completely different appriach than Docker Desktop and various access problems (and resulting missing imports) might occur depending on which users and what initialization (umask especially - on either host and in container) is used. This might lead to various permissions problems when - for example .pyc files generated on the host are read inside the container. Or it might be because the entrypoint is not executed when you exec into the container. When you exec into a running container without executing entrypoint, your environment is different than the one that the process that is run by docker using one. If you are using our docker image, you will find the right way of entering the image interactively in the documentation https://airflow.apache.org/docs/docker-stack/entrypoint.html Those are just a few of multiple problems you might hit when you are trying to use venv created on the host, in the container. And unfortunately - you need to be able to understand and diagnose all those kind of problems on your own. Guessing that "activation" is a problem is just wrong. "Activation" has nothing to do with it. Activation in virtualenv means just setting the PATH so that executed python interpreter is the one you expect to be. I recommend you to try Bash Operator and dump all the environment there and try to spot the differences between interactive and non-interactive runs when you login, maybe that will shed some light on the differences you have between the interactive shell you have and one that is invoked by an unattended, daemonized process and help you with your investigation. I hope the above pointers will help you in your investigations (if you decide to continue using the hard way that is). |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
GOAL
DONE
CHALLANGE
Files and ERRORS
docker-compose.yml
my example DAG that I want to work:
ERROR that I get:
Beta Was this translation helpful? Give feedback.
All reactions