Replies: 5 comments 3 replies
-
Beta Was this translation helpful? Give feedback.
-
|
Were you able to solve it after the weekend? If not, could you try installing the packages directly without poetry? Alternatively, check out the example here showing how you can build the image yourself (rather than upon registration) and just point Prefect to where the flow is stored within the docker image path: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_no_build Lastly, you can consider using Docker image to package dependencies only and pass the image to your DockerRun or KubernetesRn, but for storage, leverage something like S3, GCS, GitHub, Azure storage options. Given your code is already in a public GitHub repo, GitHub storage might be a good option: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_local_run.py |
Beta Was this translation helpful? Give feedback.
-
|
It may work better if you disable Poetry's virtual environments. It looks like the issue here is that the module cannot be found because the healthcheck runs outside the virtual environment. Using a virtual environment in a container is overkill anyway in most cases. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the pointers. I've read through the various discussions and references, wishing I could find additional FYI, I've opened a bounty for this question on StackOverflow: https://stackoverflow.com/questions/73757353/modulenotfounderror-with-one-prefect-flow-but-not-the-other Here are 3 different experiments, with 3 different Dockerfiles, and yet in the scenarios where the build succeeds (A and C), my flow that needs Pandas and Numpy fails in each with The most recent log (which I see in my Prefect web UI) from experiment C is here: |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Your Dockerfile seems way too complicated. You should be able to just do something like this: # This image (built via Dockerfile.prefect) is used for the Prefect backend.
FROM prefecthq/prefect:1.3.1-python3.10 as python-base
# Install poetry
RUN pip install poetry==1.2.1
# Copy the project and aid module to the image
COPY pyproject.toml poetry.lock aid /opt/aid
WORKDIR /opt/aid
# Install packages in the system Python
RUN poetry config virtualenvs.create false
# Install required packages
RUN poetry install --only mainThis build is very slow on my machine so I haven't had a chance to work through your full reproduction. There are a few important things here:
You'll want to modify your Docker storage to include I'm not entirely sure what your end goals are here but hopefully this helps. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @madkinsz , thank you so much for the specific Dockerfile. It's a huge help, and I appreciate the guidance about keeping it simple. I'm optimistic this can be solved. Unfortunately, when I run the flow based on the Dockerfile you provided, it still results in Additional details (the Flow run log and build log) are here: b-long/aid#4 . I also noticed, there a bit of a silent failure with Prefect. If you have a Dockerfile containing: and you build using the Docker CLI, with However, when the image is built via the Prefect/Python flow, it doesn't fail. The source code ( |
Beta Was this translation helpful? Give feedback.
-
|
That's peculiar. I'm not sure why it's not erroring there. It seems like adding a trailing |
Beta Was this translation helpful? Give feedback.
-
|
If you still couldn't resolve it, I'd encourage you to change the approach a little and install things directly in the container for easier troubleshooting. This means your Dockerfile could be as simple as: then build the image and exec into it: then try to install your modules in the container |
Beta Was this translation helpful? Give feedback.
-
I have a minimal project using Prefect v1.3.1, which you can find here: https://github.com/b-long/aid . I'm having a little difficulty getting the deployment right, after migrating from native code to dockerized code. I am using the latest version of Prefect v1 and need to solve this before I'm able to migrate to Prefect 2.x
I'm using Poetry, but I'm pretty sure there's a pattern here that would apply to Hatch, Pipenv, Flit or other tooling. The problem is observed in registration which logs this toward the end:
If you're a Prefect Guru, I'm guessing you'll be able to solve this issue by reading the log file (full detail below) and looking at my Dockerfile: https://github.com/b-long/aid/blob/main/Dockerfile.prefect .
Otherwise, reproducing the error is easy, as long as you have
docker,poetry, and add the following to the end of your/etc/hostsfile:You'll need to launch 3 terminals or terminal tabs. Start with step 1, and once you see the large print
"WELCOME TO PREFECT SERVER"text in the terminal, run the script for step 2 and then run the script for step 3.After the 3 scripts have been run, you can open your browser to http://localhost:8080/ and execute the flow in the UI.
The whole system is partially working, in that I'm able to do all of the steps:
If you register the flow without making changes to the code, everything works. See "Registration working for current code" (below) for a large log file. You can run the flow using "Quick Run" in the UI, and you'll see the flow has log output with success:
However, if you modify the shell script in step 3, and change which line is commented at the end, so that you are registering the flow
basic_prefect_with_pandas.py, registration will fail. The modified shell script will end with this:# poetry run python aid/prefect/basic_prefect.py poetry run python aid/prefect/basic_prefect_with_pandas.pyI've tried various configuration from #4042 , but I'm just not able to solve it. Any guidance would be appreciated!
Screenshot of everything working (
basic_prefect.py):Registration working for current code
Registration failing for
basic_prefect_with_pandas.pyBeta Was this translation helpful? Give feedback.
All reactions