-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker buildimage intermittently builds image #1243
Comments
Hi @richard-avelar ! It's possible this is related to docker caching the images, especially if you haven't changed your Dockerfile at all in between runs. To test this, could you try using from prefect import task, Flow
from prefect.tasks.docker import images
build_image = images.BuildImage(path='.', nocache=True)
with Flow("ETL") as flow:
e = build_image()
flow.run() Also, just an FYI: your current implementation creates a Task which calls another task's |
Hey @cicdw, thanks for the reply!
and the output displayed
|
Oooo I'm sorry, yea - so your image isn't even being built! So, if we strip out the Prefect logic, the import docker
client = docker.APIClient(base_url="unix:///var/run/docker.sock", version="auto")
client.build(path=".", tag=None, nocache=False, rm=True, forcerm=False) So I think we want to understand why this doesn't do what we think it should. Maybe try: response = [line for line in client.build(path=".", tag=None, nocache=False, rm=True, forcerm=False)]
print(response) and we can see if there's anything informative in that output. |
Interesting, that seems to work 😅
The output being a very long list of docker build output steps, progress, etc. Hmm, The docker specific code seems to work then but I just cant seem to get it to work in prefect for some reason. The docker specific code took about a full 2 minutes to finish running (as is typical for this image to take to complete building), but the prefect code takes a couple of seconds to run and complete execution which also seemed off to me like there was never a serious attempt to build or it silently failed the build attempt and exited |
Great! So I have a guess here -> it's possible that In the meantime, I recommend you implement a custom Task like so: import docker
@task
def build_image(path):
client = docker.APIClient(base_url="unix:///var/run/docker.sock", version="auto")
client.build(path=path, tag=None, nocache=False, rm=True, forcerm=False)
response = [line for line in client.build(path=".", tag=None, nocache=False, rm=True, forcerm=False)]
return And I'd bet this works. Looks like I need to audit our Docker Tasks for this behavior! |
Sounds good to me! |
Touchups for agent work queue by name
Hello, I am playing around with prefect and trying to use the buildimage for docker
I noticed that this intermittently builds an image once about ever 4-5 times I run it.
Anyone have an idea what is going on?
The text was updated successfully, but these errors were encountered: