Build: Support for printing outline/targets of the current build #1100

tonistiigi · 2022-05-07T01:00:58Z

This implements docker buildx build --print=[value] for running a secondary request that frontend supports and print its value. This is buildx side of moby/buildkit#2841 support.

Currently accepted values are:

--print=outline - print build arguments, secrets, ssh for current stage
--print=targets - list all possible `--target` values
--print=subrequests.describe - list all subrequest types frontend supports

To test, check out the PR and run make install. Buildkit update is not needed. If you don't use the updated frontend with outline support there is a fallback to load one in this PR.

Looking for feedback on the flag names, output formatting and if more data would be needed in moby/buildkit#2841 . We can't really use flags like --list or --describe as we want this subrequest logic to be extendable in the future with more use cases. Any frontend can define that it implements a certain function and then that function can be called with --print atm.

Related discussion on bake level #1072

tonistiigi · 2022-05-07T01:09:36Z

Demo of this feature in action: https://drive.google.com/file/d/1H92gzwctwRkllVKtgF6lDpDmk-4NACLs/view

cpuguy83 · 2022-05-07T02:01:16Z

Nice!
For more data (I just watched your demo, haven't peeked at the code), a parsable list of sources would be fantastic.

ciaranmcnulty · 2022-05-07T07:39:16Z

That's cool. We used to have Dockerfiles like this, building in bake:

ARG PHP_VERSION
FROM ${PHP_VERSION} as final

but Dependabot can't then update them. It was annoying because we want to include the PHP version in the tag.

I guess now we can inline the version into the Dockerfile again, run this outline to extract the version(s), then use those versions in the tags we push?

It would be very interesting in future to be able to access outline values from bake itself

tonistiigi · 2022-05-09T18:27:05Z

It would be very interesting in future to be able to access outline values from bake itself

This is somewhat tricky as it can easily go to an infinite loop like this. Your bake config modifies the Dockerfile args and then Dockerfile args modify bake config again.

crazy-max · 2022-06-16T14:12:23Z

...feedback on the flag names

--print flag lgtm.

--print=outline - print build arguments, secrets, ssh for current stage
--print=targets - list all possible `--target` values

As outline is tied to a target, I wonder if we could use it as an object to show what we want:

$ docker buildx build --print=outline.args https://github.com/moby/buildkit.git
BUILD ARG         VALUE       DESCRIPTION
RUNC_VERSION      v1.0.2
BUILDKIT_TARGET   buildkitd   defines buildkitd worker mode (buildkitd, buildkitd.oci_only, buildkitd.containerd_only)
ALPINE_VERSION    3.15        sets version for the base layers
BUILDKITD_TAGS                defines additional Go build tags for compiling buildkitd

For a follow-up, maybe also be able to specify an output format that defaults to tabwriter if not specified:

$ docker buildx build --print=format=json,outline.args https://github.com/moby/buildkit.git
{
  "RUNC_VERSION": "v1.0.2",
  "BUILDKIT_TARGET": "buildkitd",
  "ALPINE_VERSION": "3.15",
  "BUILDKITD_TAGS": ""
}

...output formatting

About the output I think we should hide the progress output and just display solve errors:

$ docker buildx build --print targets https://github.com/moby/buildkit.git
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 12.07kB 0.1s done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 56B 0.0s done
#2 DONE 0.1s

#3 resolve image config for docker.io/docker/dockerfile-upstream:1.4
#3 DONE 0.5s

#4 docker-image://docker.io/docker/dockerfile-upstream:1.4@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2
#4 resolve docker.io/docker/dockerfile-upstream:1.4@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2 0.0s done
#4 CACHED

#5 resolve image config for docker.io/docker/dockerfile-upstream:1.4-outline
#5 DONE 0.2s

#6 docker-image://docker.io/docker/dockerfile-upstream:1.4-outline@sha256:ccd574ab34a8875c64bb6a8fb3cfae2e6d62d31b28b9f688075cc14c9b669a59
#6 resolve docker.io/docker/dockerfile-upstream:1.4-outline@sha256:ccd574ab34a8875c64bb6a8fb3cfae2e6d62d31b28b9f688075cc14c9b669a59 0.0s done
#6 DONE 0.0s

#6 docker-image://docker.io/docker/dockerfile-upstream:1.4-outline@sha256:ccd574ab34a8875c64bb6a8fb3cfae2e6d62d31b28b9f688075cc14c9b669a59
#6 CACHED
TARGET                             DESCRIPTION
git                                stage is used for checking out remote repository sources
xx                                 is a helper for cross-compilation
integration-tests                  prepares an image suitable for running all tests
buildkit (default)                 builds the buildkit container image
golatest
gobuild-base
runc-src
runc
...

$ docker buildx build --print targets https://github.com/moby/buildkit.git
TARGET                             DESCRIPTION
git                                stage is used for checking out remote repository sources
xx                                 is a helper for cross-compilation
integration-tests                  prepares an image suitable for running all tests
buildkit (default)                 builds the buildkit container image
golatest
gobuild-base
runc-src
runc
...

jedevc

Left some initial thoughts 🎉

Not sure about having a fallback like specified, silently upgrading to a new frontend seems like we could cause subtle compatability issues - if we do go with a fallback, I think it's worth warning the user that we've done so. Also would need a way to pick a fallback for different frontends, we shouldn't just pick dockerfile if we've specified a different syntax.

For a follow-up, maybe also be able to specify an output format that defaults to tabwriter if not specified:

👍 I think this is nice, would make this info very readable 🎉

About the output I think we should hide the progress output and just display solve errors:

I don't think we should do anything special here, as I understand the code, a frontend could possibly choose to continue it's normal operation even with a subrequest (and do some supplement on top). I think we should support use case though, maybe with a new --progress=none option or something in a buildkit follow-up (could be useful for other scenarios as well).

jedevc · 2022-08-05T13:37:30Z

build/build.go

+							for {
+								if opt.PrintFunc != nil {
+									if _, ok := req.FrontendOpt["frontend.caps"]; !ok {
+										req.FrontendOpt["frontend.caps"] = "moby.buildkit.frontend.subrequests+forward"


Think it would be nice to have consts for the subrequest caps on the buildkit side for these caps similar to solver/pb/caps.go

This should be defined in some package that defines keys for Dockerfile frontend compatibility interface. https://github.com/moby/buildkit/blob/master/frontend/dockerfile/builder/build.go#L43-L47 Unfortunately no such package exists atm

jedevc · 2022-08-05T13:43:41Z

build/build.go

-								resultHandleFunc(dp.driverIndex, &ResultContext{cc, res})
+							var isFallback bool
+							var origErr error
+							for {


Not sure if we should use a loop here to do the retries? As I understand it, it's so that we can have a second try with a fallback, but that might be easier to read in-serial, instead of with the loop/continue which is hard to reason about control flow.

Not sure I understand. Without the loop, a bunch of this code would need to be duplicated.

I was thinking we could try and move the logic into a separate function/method, as a generic makeRequest or something function, then we could use a bool return to check if it was successful, and if not, we could make a fallback call to the same function, which would keep the logic needed in BuildWithResultHandler to a minimum.

jedevc · 2022-08-05T13:56:05Z

build/build.go

+							rr.ExporterResponse = map[string]string{}
+						}
+						for k, v := range printRes {
+							rr.ExporterResponse[k] = string(v)


Because we store the values in here, are they going to be output to the metadata file as well? I think that makes sense.

I wonder if we need to do something here to make sure we don't write to any known keys here, if the frontend's returned Metadata contains a duplicate key for whatever reason, we might potentially corrupt some data here. Maybe it's worth keeping this info separate with a composite struct?

Actually, I wonder if we could do something similar to annotations, where we have the prefix annotation., here we could have the prefix subrequest. or something.

Because we store the values in here, are they going to be output to the metadata file as well?

Yes.

frontend's returned Metadata contains a duplicate key for whatever reason,

How can it be duplicated if it is a map?

here we could have the prefix subrequest. or something.

I don't think that would be right. The point of subrequests is to take a different code path and return results for it, so it is not "do build and also return extra keys that would need to be namespace". Oth, subrequest build does not mean that it needs to return json/text. A subrequest could for example do a build with a different configuration and return a different image. In that case, it does want to return the correct image keys so they get exported correctly, not namespaced special keys.

Sorry, I meant duplicated values between res.Metadata and rr.ExporterResponse since we do a merge here.

Agreed though, thanks for the clarification on subrequests.

commands/build.go

jedevc · 2022-08-05T14:01:36Z

commands/build.go

+	for _, field := range fields {
+		parts := strings.SplitN(field, "=", 2)
+		if len(parts) == 2 {
+			if parts[0] == "format" {


This feels a bit confusing to me since we already have the --format flag - I guess this one is to indicate the source format, as opposed to the output format? I think it might be good to have a different name for this to help clarify, or maybe is there a way we could avoid needing to have to specify the format here?

This was made to match #1100 (comment)

jedevc · 2022-08-05T14:03:41Z

commands/print.go

+
+func printValue(printer printFunc, version string, format string, res map[string]string) error {
+	if format == "json" {
+		fmt.Fprintln(os.Stdout, res["result.json"])


I think we should have result.{txt,json} as consts in buildkit maybe, or even a func (format string) (filename string) somewhere, to avoid having this info repeated between client-server.

This is not hardcoded anywhere but defined in the subrequest definition that frontend shows. If handling a specific one, client should look what outputs it has from the description. This is a generic "best-effort" because we let the user specify any function name/

sources does not seem decoded: https://gist.github.com/crazy-max/dfb1e5fc84f258be128e2fba90c467e1

Could you use the decodeExporterResponse?:

buildx/commands/build.go

Lines 607 to 623 in 98f9f80

func decodeExporterResponse(exporterResponse map[string]string) map[string]interface{} {

out := make(map[string]interface{})

for k, v := range exporterResponse {

dt, err := base64.StdEncoding.DecodeString(v)

if err != nil {

out[k] = v

continue

}

var raw map[string]interface{}

if err = json.Unmarshal(dt, &raw); err != nil || len(raw) == 0 {

out[k] = v

continue

}

out[k] = json.RawMessage(dt)

}

return out

}

A source is the original source file that we don't control. It can be any kind of bytes. I think we shouldn't make assumptions about this. Even if it is a Dockerfile, showing some encoded string in here for the whole big file would not make it clearer and the user would need to do some decoding from a string to try to get the original form.

This is not encoded JSON so decodeExporterResponse would not do anything here.

Ah right that makes sense.

commands/build.go

Print flag can be used to make additional information requests about the build and print their results. Currently Dockerfile supports: outline, targets, subrequests.describe Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

cpuguy83 · 2022-08-09T22:08:06Z

I noticed the exporter still runs with this.
e.g. if I run the command with --output=foo, buildx will create the foo directory even though it's just an outline request.

cpuguy83

One minor thing mentioned already, not sure if anyone cares.
Otherwise LGTM

cpuguy83 · 2022-08-09T22:15:35Z

Might be nice to work out a new subcommand for these requests like buildx info <target> or buildx targets, just a thought. I expect I'd be using this just with the additional --print flag on existing build commands though for "real" work.

tonistiigi · 2022-08-10T01:34:11Z

@cpuguy83

if I run the command with --output=foo, buildx will create the foo directory even though it's just an outline request.

This is intentional as we will not make assumptions about what functions frontends will implement. These functions can run the regular(or custom) build and return results. These results would be exported with -o then.

tonistiigi force-pushed the print-outline branch from 4ecfdc9 to 60c543a Compare May 7, 2022 01:09

tonistiigi mentioned this pull request Aug 5, 2022

dockerfile: support for outline requests moby/buildkit#2841

Merged

tonistiigi force-pushed the print-outline branch from 60c543a to aca95ba Compare August 5, 2022 07:53

tonistiigi changed the title ~~[WIP] Build: Support for printing outline/targets of the current build~~ Build: Support for printing outline/targets of the current build Aug 5, 2022

tonistiigi marked this pull request as ready for review August 5, 2022 07:54

tonistiigi added the status/do-not-merge label Aug 5, 2022

jedevc reviewed Aug 5, 2022

View reviewed changes

tonistiigi added this to the v0.9.0 milestone Aug 8, 2022

crazy-max reviewed Aug 8, 2022

View reviewed changes

commands/build.go Show resolved Hide resolved

tonistiigi force-pushed the print-outline branch from aca95ba to bea1214 Compare August 9, 2022 01:08

tonistiigi added 3 commits August 9, 2022 08:24

build: add experimental support for print flag

eefa818

Print flag can be used to make additional information requests about the build and print their results. Currently Dockerfile supports: outline, targets, subrequests.describe Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

build: add fallback to outline requests if not supported by frontend

cab437a

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

add formatting support to print function

c834ba1

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the print-outline branch from bea1214 to c834ba1 Compare August 9, 2022 15:25

tonistiigi removed the status/do-not-merge label Aug 9, 2022

crazy-max approved these changes Aug 9, 2022

View reviewed changes

cpuguy83 approved these changes Aug 9, 2022

View reviewed changes

tonistiigi merged commit da1f4b8 into docker:master Aug 10, 2022

tonistiigi mentioned this pull request Aug 12, 2022

commands: use buildx env for experimental opt-in #1267

Merged

This was referenced Apr 15, 2024

bake: subrequest support for printing outline/targets/lint #2410

Open

build: mark information requests as internal #2416

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: Support for printing outline/targets of the current build #1100

Build: Support for printing outline/targets of the current build #1100

tonistiigi commented May 7, 2022 •

edited

tonistiigi commented May 7, 2022

cpuguy83 commented May 7, 2022

ciaranmcnulty commented May 7, 2022 •

edited

tonistiigi commented May 9, 2022

crazy-max commented Jun 16, 2022 •

edited

jedevc left a comment •

edited

jedevc Aug 5, 2022

tonistiigi Aug 8, 2022

jedevc Aug 5, 2022

tonistiigi Aug 8, 2022

jedevc Aug 8, 2022

jedevc Aug 5, 2022

jedevc Aug 5, 2022

tonistiigi Aug 8, 2022

jedevc Aug 8, 2022

jedevc Aug 5, 2022

tonistiigi Aug 8, 2022

jedevc Aug 5, 2022

tonistiigi Aug 8, 2022

crazy-max Aug 9, 2022

tonistiigi Aug 9, 2022

tonistiigi Aug 9, 2022

crazy-max Aug 9, 2022

cpuguy83 commented Aug 9, 2022

cpuguy83 left a comment

cpuguy83 commented Aug 9, 2022

tonistiigi commented Aug 10, 2022

	func decodeExporterResponse(exporterResponse map[string]string) map[string]interface{} {
	out := make(map[string]interface{})
	for k, v := range exporterResponse {
	dt, err := base64.StdEncoding.DecodeString(v)
	if err != nil {
	out[k] = v
	continue
	}
	var raw map[string]interface{}
	if err = json.Unmarshal(dt, &raw); err != nil \|\| len(raw) == 0 {
	out[k] = v
	continue
	}
	out[k] = json.RawMessage(dt)
	}
	return out
	}

Build: Support for printing outline/targets of the current build #1100

Build: Support for printing outline/targets of the current build #1100

Conversation

tonistiigi commented May 7, 2022 • edited

tonistiigi commented May 7, 2022

cpuguy83 commented May 7, 2022

ciaranmcnulty commented May 7, 2022 • edited

tonistiigi commented May 9, 2022

crazy-max commented Jun 16, 2022 • edited

jedevc left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpuguy83 commented Aug 9, 2022

cpuguy83 left a comment

Choose a reason for hiding this comment

cpuguy83 commented Aug 9, 2022

tonistiigi commented Aug 10, 2022

tonistiigi commented May 7, 2022 •

edited

ciaranmcnulty commented May 7, 2022 •

edited

crazy-max commented Jun 16, 2022 •

edited

jedevc left a comment •

edited