Parallel build does not wait for base image

[gaetansnl]

I'm building an image B and then 5 images based on B for performance reason.
Since parallel builds are enabled, it seems that B is build 3 times instead of one.

An option to specify dependencies for docker_build and custom_build would be great. Or maybe is there is an existing solution ?

Thank you

[dbentley]

We don't support deps from image to image.

We do allow deps from resource to resource, using resource_deps. As a hack, you could make a fake resource (like a Job that just executes true) to allow this until we support image deps.

[maiamcc]

hey @gaetansnl, following up: yes, this is great feature request!

I want to clarify that resource dependencies will only really be useful on the first build. (i.e. resource dependencies only enforce that resource A must be ready before resource B builds for the first time; but do NOT enforce that, when resource A rebuilds, resource B must then rebuild as well.)

As a hack, you might consider:

local_resource('build base image',
    cmd='docker build . -t base-image)
docker_build('child-1', '.', dockerfile='dockerfile.1',
    resource_deps='build base image')
docker_build('child-2', '.', dockerfile='dockerfile.2',
    resource_deps='build base image')
...

with this tiltfile, the local resource defined at the top would build your base image at the beginning of your tilt run. nothing here would stop the image from rebuilding 3x on a change to ., but at least you get the benefit of SOME caching (and hopefully you can leverage live update to reduce the number of docker builds).

I don't know when we'll be able to implement good support for base images + parallel updates, so in the meantime you can use a hack like the above or turn off parallel updates (update_settings(max_parallel_updates=1). Let me know if you have any questions!

Note for Tilt Team: the lightest-weight technical solution is probably to add a filter to NextTargetToBuild like RemoveTargetsSharingImageTargets or something, which removes from consideration any targets that have image targets matching a currently building image.

[nicks]

OK, with the latest Tilt release, the scheduler won't try to parallelize two deployments that have the same base image. But it will still repeat the builds. I'm going to leave this open for the general case of skipping the redundant builds entirely.

[kollender]

@nicks any updates regarding redundant builds?

[nicks]

@kollender Nope, we're still scoping out possible solutions. This is currently difficult to solve without a major internal rearchitecture or some clever idea.

[tclasen]

@nicks , the major internal rearchitecture you will need is to change the list of build objects from graph flattened into a list via a topological sort into a tree-like structure where every node in the graph has a reference to the items blocking it.

Your code: https://github.com/windmilleng/tilt/blob/master/pkg/model/target_graph.go#L5

Example implementation from another project: https://github.com/thought-machine/please/blob/dc22858f1e067bdf3f9ce79cea5ac1a2edae63c7/src/core/graph.go#L17

Both projects also use starlark to mark these dependencies, so there should be a lot of overlap in codebases.

[nicks]

as of #3362, Tilt will no longer build the same image multiple times in this case

But it will still build them in serial. I filed a separate issue for the parallelization: #3468

There's already a separate issue filed to make custom_build support image dependencies: #2856

[nicks]

unfortunately, we had to roll the fix for this back 😭

[nicks]

rolled forward a fix in #3553 that i feel a lot better about