Cannot give a local image to FROM when using docker-container #2343
Comments
I think this might be expected; if you're using a containerised builder, that builder uses its own cache for images, which is not shared with the If you use the I wonder if it's possible though to make the container builder itself keep the image (for cases where the second build is running on the same builder: @crazy-max are the Otherwise, it's probably best to push your images to a registry (which would also address the multi-arch use-case, if you happen to be building a multi-arch image). But, yes, I think the UX / experience could be improved somewhat here to make this easier to use (or more clear "how" to do this) |
|
I think the solution we're going with is to do this outside of buildkit like docker/buildx#447 |
|
Is there any workaround for this? Attempting to build an image on an instance outside of my network (therefore it cannot access my custom registry) and I need to use a base image from that registry. I have copied the tar file for that image, but I cannot find a way to load that image into buildkit. |
|
FWIW, this is exactly what we raised in #2210 . builder-container has a regression in functionality compared to docker builder, at least as far as cache is concerned. I had looked into the containerized builder works, and was surprised that it actually was missing only a few things to make it work. If I recall correctly:
You could do other things, like I still know of a few people willing to help with it a bit, but it has been 9 months since we opened the issue. |
|
@tonistiigi how does this help it? |
|
As far as I can tell - please do correct me - the buildx bake context stuff allows you to, essentially, "alias" a It looks pretty hesitant to use that target reference:
I don't understand how this manages to store the output of one to the other, especially with containerized builder. Is it just because it is building both at once, so the builder knows about all of the outputs and can "hold onto" them? More important UX question: Does this mean that, in order to do the simple "use a a local cached image" that we have been used to for docker since day 1, where I do
all to replicate a "docker-simple" (TM) functionality from the existing flow? |
|
@deitch As you have been explained multiple times, nothing changes with the existing build behavior. If you want your builds to access Docker images you need to use the Docker driver. The same would be to set the requirement to build in k8s pods, but not do it using k8s driver.
Yes, when using Docker images as intermediate storage you needed to run multiple |
Understood. But isn't the point of buildkit and the various drivers so that we can open new capabilities? buildkit does this phenomenally well, but some of those capabilities are not yet retrofitted into the docker driver. So you have to choose between:
I think we are trying to bridge this gap: if docker driver could do everything buildkit can (which it eventually will, I understand), and buildkit containerized could do everything docker could (again, within reason), then the choice between features Docker and features buildkit (and if you need both, you are out of luck) wouldn't be a problem.
Sure, and I built tons of images that way in the early days. And when I could move to multistage after it existed, I did. But is it a fair assumption that every (or even a majority) of builds are simple linear "start at point A, go through some intermediates, get to point Z" and thus multistage candidates? It is very common to have a base image (A), build some "golden base" (B), then some intermediates (C, D, E, F), then some finals (Q, R, S, T, U, V, W, X, Y, Z). This "tree" isn't a single build that can go in a multistage Dockerfile, and the intermediates aren't stored in a remote registry: they might be private, they might be part of a testing process and aren't valid to be stored until the finals (Q, R, S, T, U, V, W, X, Y, Z) are generated and fully tested (if at all). The local cache holds all of those interim images, and the later steps absolutely are not a single multistage build, simply cannot be. They cannot even be run at the same time (like
These are separate build processes, might be running at separate times. This isn't a single multi-stage Dockerfile (imagine what a nightmare that single file would be), and isn't run as a single step or stage in a CI process. Let's be practical: someone has been doing this for years with And their entire build process breaks. Because step 1 stored output B, and step 2 cannot find B for C,D,E,F (repeat for step 3).
That's just it. The above very common use case describes how "one command" isn't practical here. It has to be multiple commands. Yet The part I am having a really hard time figuring out is, why the strong objection to proper caching inside buildkit container? You already store blobs, you already have the OCI layout, the number of additional steps to get to parity with default docker behaviour is so small (and people have offered to help build it). Why the resistance? |
|
I have the same sort of non-linear use case as @deitch describes and have found myself blocked by this, as an additional anecdote. |
BuildKit is not supposed to be a reimplementation of all the Docker features. Obviously, Docker has useful features, like an image store, and if you like it, you should continue to use it. BuildKit does not supersede the whole Docker engine but is scoped for a specific job: building from well-defined (immutable) sources to the expected build result. If your problem statement is that you want to build some images, then take a break and later use the images you built in another build, then you need a place to keep them between your builds. There are plenty of options: docker image store, containerd image store, or any registry implementation. With named context we have made the registry case even simpler as you don't need to do any changes in Dockerfile (or use build args) if you want to switch between release images and local dev/staging registry image. Defining an image store is fundamentally conflicting with builder design and BuildKit features. For example, BuildKit has an automatic garbage collector or smart pruning filters for managing storage. Users don't need to think about where their build are running, and everything can be moved to different infrastructure. There is support for multi-node build that splits build between multiple machines. All this would be impossible if there was some kind of name-based storage. The user does not need to think that there is a special machine somewhere that contains their images, how they are named, how much storage they take, what happens if another version with the same name is built, what happens if one of them is deleted, what do you do after you run |
|
This is a pretty good explanation @tonistiigi ; thanks for that. I will try to respond in detail.
Definitely. "Do one job and do it well". The challenge is that docker may have some features (local reusable image store), buildkit may others (cross-arch builds, remote contexts, etc.), and if you need to use both, you are stuck.
Agreed 100% there. You need somewhere to store them.
Well, not exactly. docker image store doesn't support anything other than local architecture (and expands the images), and both docker store and containerd store are not accessible to container-driver buildkit. I think you have narrowed the problem down well; I am going to modify it somewhat to make it more explicit: if you want to build images using buildkit because of its features that docker does not have, and you want to "take a break" between builds, then you need somewhere to store those images that buildkit supports sourcing from. That list is pretty narrow. Actually, it is pretty much: network registry. And if your image does not or cannot go to the registry (e.g. during development lifecycle, when you are not yet ready to push, or an interim image that you never will push), what solution do you habve?
Yes, I see that. Takes a while to get it, but basically named contexts are "image aliases". Come to think of it, this would have been nice in docker from day 1.
This is a pretty solid argument. it doesn't eliminate the need to solve the problem, but it makes a solid case for why a local cache may not be the approach. You basically are saying:
I can get behind it, but what is that solution?
So what is the right answer? Reading what you wrote, you almost want to be able to have a pluggable implementation of image aliasing, so I can use the following Dockerfile: FROM myimage:1.2.3and then run Can named context actually do that? I think maybe some of it? |
You don't need a new flag. This is already supported. |
Yeah, I wasn't suggesting a new flag. Just using language that was instantly understandable in our conceptual discussion.
Yeah, I got that from your descriptions. Makes sense. Is there anything like the current docker behaviour of, "look in this cache, and if not there, go pull it"?
Meaning you would be open to it? I might be willing to submit a PR for that. Point me at the right place in buildkit where to hook it in. The hard part, I think, still is the "generic idea". When I do a classic I am up for getting us to have
Do you see where I am driving? |
|
Sorry, but why not being able to specify a FROM ../my-base-image
[...] |
Yes. Take a look at how the
That does not work well. First of all names are needed for the build-context mechanism exposed to frontends. Secondly, OCI has not even decided on a standard for naming multiple images within the OCI layout.
I'm not sure what the example you provided is supposed to do but Dockerfile is a secure and portable build mechanism. It does not have permission to randomly start reading files from user's/host's disk. |
It's supposed to do the same as: FROM my-base-image
[...]docker build --build-context my-base-image=../my-base-image .And about the |
|
@felipecrs Bake has a different security model indeed. Atm. for local bake files it is allowed for them to point to any local path. For remote files, we have disabled parent access. In future releases, the plan is to move to docker/buildx#179 to control what is allowed and what is not. |
Yeah, I get what Tonis is saying here. the |
Quite. I have generally used That doesn't change the question of specific name vs generic. It removes it entirely from buildkit's scope, actually. It wouldn't know or care; it would just pass it on to the driver.
That isn't a fundamental product reason as to why it wouldn't be desired; it is a technical engineering reason why it wouldn't work with the current design. I will try and put it in other terms.
While I am aware that we could (almost) get to that by running a customized local registry as read-through cache, with perhaps some special logic that handles whatever aliasing is desired. Of course, all of the above is much more complex than just, "I build something using buildkit, then build something else, and buildkit caches the results of the first". |
|
While explaining this issue to some people, I realized the most basic use case why it matters, and why multi-staged docker files don't solve it, why bake and build contexts only partially do it. Let's say that I am developing an image that normally is public, call it A normal, sane, build process has me do the following:
The above is a great, simple chaining process, very Docker-ish, and 100% depends on Even with build-contexts, how would I do my normal process? What can I give to In theory, I could do a Am I doing a decent enough job explaining how this breaks normal processes? |
|
Oh and by the way, I totally get your argument that, "this is an optimized builder, not image manager/cacher; docker does both, this is just half of it." What I am trying to do is understand, "ok, what is the other half that complements it so we can get buildkit-build awesomeness with image caching"? |
|
Thinking a bit more about (partial) solutions: buildkit caches many blobs - but not image refs, for the reasons you described @tonistiigi - but only layers, not json blobs (config, manifest, index). What if it also cached those, which is trivial? The same way that buildkit currently, when it hits "I need blob X", it looks in local cache and, if it cannot find it, goes to the registry for the image currently being looked up, it could do the same for manifests and such. Then the only thing left to do would be resolve image-> hash That would allow something like: or even shorthand Buildkit could:
It isn't perfect, but it gets rid of a lot of the headache without breaking the current buildkit model/mindset:
It still means the command-line needs to know a lot, but it is a big step forward while staying in the same context. Thoughts @tonistiigi ? |
I'm not sure if I agree with this part. Because bake would be as simple as: To build the two images dependent on |
Are you suggesting that the dev process becomes one that uses bake? So:
Still not quite the simplicity of "just have a local read-through cache", but something. I think the part that throws me is that, I have to use a completely different command-line for the second image, depending on if my first image is locally built or not. |
I would just rather use
Well, that will be defined in your docker buildx build bake my-second-image |
|
And to support both cases in the same # docker-bake.hcl
target "base" {
dockerfile = "baseapp.Dockerfile"
}
target "app" {
contexts = {
baseapp = "target:base"
}
}Building app, also building basedocker buildx bake appBuilding app, using previously pushed basedocker buildx bake app --set=app.contexts.baseapp=docker-image://my-previously-pushed-base |
|
@thaJeztah thank you! that worked perfectly! |
I'm sorry, I'm still very confused for the solution. I've been working on this for several days and have reviewed all issues I've found. I'm attempting to do a containerized build using docker with the docker:dind service running Dockerfile: is the solution to run the following?: Edit: |
|
I think this is the issue with Docker itself because it uses registry to pull images and locally stored images are not in some registry, So possible solution is push your images to your repository and then login to your registry via local. |
|
where middle layer for debug？where FROM base image？ |
|
Add me to the pile of casualties affected by this issue. I support multiarch devcontainers and after upgrading docker-compose (in VSCode devcontainers) can no longer reference locally built images to build FROM and instead tries to get the local images failing on This feels like a regression and not a feature that was introduced. Pretty frustrating as I'm blocked and am looking through these comments for a workaround. |
|
If you are on a recent version of Docker Desktop or the Docker Engine, you can enable the containerd image store integration. That provides a multi-arch store without requiring a docker-container builder. It's still in preview, but work is progressing and will be moved out of "experimental" in the not too distant future; see https://docs.docker.com/storage/containerd/ Make sure to switch the builder instance back to the default (not the container-builder, otherwise it continues to use the |
|
@thaJeztah I think I found the root cause of my specific issue. VSCode Dev Containers extension hard codes I have commented on their code review; I seem to have been confused (it's a complex stack) |
|
@thaJeztah actually, I think I ruled out dev containers after all. I'll describe more of my setup. My compose file looks like: version: '2.4'
services:
some-service:
build:
context: .
dockerfile: ../../shared/Dockerfile.compose
args:
BASE_DOCKER_IMAGE: ${MY_DOCKER_IMAGE}
PLATFORM: ${PLATFORM}
#... etcI have a And the ARG BASE_DOCKER_IMAGE
ARG PLATFORM
FROM --platform=linux/${PLATFORM} ${BASE_DOCKER_IMAGE}When I run: I get the error Locally, if I run I can do things like build docker compose with non-local images and images hosted in a corporate docker image store. However, I don't appear to |
|
BuildKit image building used to work with locally built docker images and apparently no longer does so due to cache. How do I get support for multi-arch images and referencing local images when building from docker-compose? This definitely used to work and currently does not. |
|
I found my own workaround: I ran the following CLI command: And now it works again. Thanks to https://stackoverflow.com/questions/20481225/how-can-i-use-a-local-image-as-the-base-image-with-a-dockerfile |
|
The root cause of my issue is likely self-induced. I was researching multi-arch images at some point and I'm pretty sure that I switched my Nothing to see here... |
that workaround does work for me too |
|
Running into this as well, and while using |
|
@superstator |
For most workflows, we want to use the "docker" driver, while the "docker-container" driver should be used when images need to be pushed to the registry (so we can use registry-based caching). When using the "docker-container" driver, images are built in a build container, and there is no access to the docker image store (see moby/buildkit#2343). This means that when building a new image, we cannot use "local" images as base images. This is an issue because the base image may not exist yet in the directory, and because we may want to use a modified base image which is not the same as the one in the registry. Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
For most workflows, we want to use the "docker" driver, while the "docker-container" driver should be used when images need to be pushed to the registry (so we can use registry-based caching). When using the "docker-container" driver, images are built in a build container, and there is no access to the docker image store (see moby/buildkit#2343). This means that when building a new image, we cannot use "local" images as base images. This is an issue because the base image may not exist yet in the directory, and because we may want to use a modified base image which is not the same as the one in the registry. Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
I have a Dockerfile which references another image, which is stored locally (overrideable with a build arg):
The image is there:
When I try to build this Dockerfile with
--builder xyz, which is backed by a docker-container driver, I get this:If I try to build it with exactly the same parameters, but omitting
--builder xyz, it builds just fine.The text was updated successfully, but these errors were encountered: