dockerfile: support for outline requests #2841

tonistiigi · 2022-05-02T06:37:00Z

Adds a new subrequest frontend.outline that returns
all the build arguments the current build definition
takes without actually running the build.

This allows getting information about the possible build parameters without needing to study the whole Dockerfile. Every build arg also includes the current value, description and location of the build argument. Description is parsed from the associated comment.

I also plan to add a request that lists all possible build targets.

The second commit adds ability to buildctl to see the result of the subrequests (at least when they return result.* metadata). Previously this was only possible with the Go API.

buildctl build --frontend dockerfile.v0 --local dockerfile=. --local context=. --opt requestid=frontend.subrequests.describe

buildctl build --frontend dockerfile.v0 --local dockerfile=. --local context=. --opt requestid=frontend.outline

{
  "args": [
    {
      "Name": "RUNC_VERSION",
      "Description": "",
      "Value": "v1.0.2",
      "Location": {
        "ranges": [
          {
            "start": {
              "Line": 3
            },
            "end": {
              "Line": 3
            }
          }
        ]
      }
    },
    {
      "Name": "BUILDKIT_TARGET",
      "Description": "",
      "Value": "buildkitd",
      "Location": {
        "ranges": [
          {
            "start": {
              "Line": 10
            },
            "end": {
              "Line": 10
            }
          }
        ]
      }
    },
    {
      "Name": "ALPINE_VERSION",

AkihiroSuda · 2022-05-02T10:34:01Z

What would be the expected use cases? 👀

tonistiigi · 2022-05-02T18:03:14Z

What would be the expected use cases? 👀

When you clone a project and want to build it or thinking what flags docker build takes you currently need to open the Dockerfile and study it to understand what parameters are possible. Even doing that is complicated if there are many stages and you can't easily visualize what stages depend on other stages. This allows getting a list of all accepted parameters and their descriptions so users can see what is being built and what configuration options they have.

crazy-max · 2022-05-06T07:37:40Z

frontend/dockerfile/dockerfile2llb/outline.go

+				args = append(args, outline.Arg{
+					Name:        a.definition.Key,
+					Value:       a.value,
+					Description: a.definition.Comment,


I wonder if the comments used as description for args and stages should have a special markup to avoid unwanted comments like https://github.com/crazy-max/moby/blob/82ede62397d3de933e79483f84d3a8092587485f/Dockerfile#L237-L239

I have this use case for example where I want to explicitly tell what stages the user should care about: https://github.com/crazy-max/moby/blob/82ede62397d3de933e79483f84d3a8092587485f/Dockerfile#L618-L643

It already does this

AkihiroSuda · 2022-07-23T03:59:13Z

Needs rebase

ktock · 2022-07-25T06:04:24Z

SGTM about the feature but how should multi-staged Dockerfile be handled by this feature?
Currently it seems possible that the outline loses info for some args on multi-stage Dockerfile.
For example, the following Dockerfile contains FOO at line 4 and 8 but outline shows arg only of line 4.

ARG FOO

FROM busybox AS foo-1
ARG FOO=foo1
RUN echo $FOO > /foo1

FROM busybox AS foo-2
ARG FOO=foo2
RUN echo $FOO > /foo2

FROM scratch
COPY --from=foo-1 /foo1 /foo1
COPY --from=foo-2 /foo2 /foo2

buildctl build --frontend dockerfile.v0 --local dockerfile=/tmp/ctx --local context=/tmp/ctx --opt requestid=frontend.outline
{
  "args": [
    {
      "Name": "FOO",
      "Description": "",
      "Value": "foo1",
      "Location": {
        "ranges": [
          {
            "start": {
              "Line": 4
            },
            "end": {
              "Line": 4
            }
          }
        ]
      }
    }
  ],
  "sources": [
    "QVJHIEZPTwoKRlJPTSBidXN5Ym94IEFTIGZvby0xCkFSRyBGT089Zm9vMQpSVU4gZWNobyAkRk9PID4gL2ZvbzEKCkZST00gYnVzeWJveCBBUyBmb28tMgpBUkcgRk9PPWZvbzIKUlVOIGVjaG8gJEZPTyA+IC9mb28yCgpGUk9NIHNjcmF0Y2gKQ09QWSAtLWZyb209Zm9vLTEgL2ZvbzEgL2ZvbzEKQ09QWSAtLWZyb209Zm9vLTIgL2ZvbzIgL2ZvbzIK"
  ]
}

tonistiigi · 2022-07-26T04:40:36Z

@ktock A build arg can only have one value because there is only one way to pass that in. What you are seeing there is the default because no value was set. I don't think this is a practical case where arg with different default values are used together as there is no way to control them differently. So in practical cases there would either be two args or they would have the same config if they are controlled by the same build arg. We could consider separating out the default and passed value, although for the user flow they would be combined.

I'm going to do a second pass of this based on feedback. Will add JSON and Text output separately and some versioning so that we can detect when client/daemon is newer than expected and returns a modified structure.

ktock · 2022-07-27T08:01:51Z

@ktock A build arg can only have one value because there is only one way to pass that in. What you are seeing there is the default because no value was set. I don't think this is a practical case where arg with different default values are used together as there is no way to control them differently. So in practical cases there would either be two args or they would have the same config if they are controlled by the same build arg. We could consider separating out the default and passed value, although for the user flow they would be combined.

Thank you for the explanation. SGTM.

ktock

Overall LGTM

This allows getting a list of all accepted parameters and their descriptions so users can see what is being built and what configuration options they have.

Sounds like godoc but for Dockerfile (dockerfiledoc)?

Even doing that is complicated if there are many stages and you can't easily visualize what stages depend on other stages

Maybe we'll need a field to show the dependency in the targets.Target struct.

ktock · 2022-08-02T02:37:58Z

frontend/dockerfile/dockerfile2llb/outline.go

+				args = append(args, outline.Arg{
+					Name:        a.definition.Key,
+					Value:       a.value,
+					Description: a.definition.Comment,


How can we add a description to the FROM of the final stage without name?

Why not define a name if you want to add a description? It would still work as a default target but would be descriptive.

tonistiigi · 2022-08-02T05:31:23Z

@bpaquet S3 CI seems to be failing quite reliably after the rebase here but I don't quite see what could be the connection or where it is failing. Could you take a look.

cmd/buildctl/build.go

Adds a new subrequest frontend.outline that returns all the build arguments the current build definition takes without actually running the build. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Make it easier for all buildkit clients to have formatted subrequest results. Version can be used for client to detect if frontend returns structure that is doesn’t know about. Then client can make a choice to just print the already formatted contents. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi · 2022-08-05T05:39:48Z

Maybe we'll need a field to show the dependency in the targets.Target struct.

@ktock I've added the base name and platform to the target struct. I don't know a sensible way to print them though so atm they are just in the JSON output.

I've also added versioning so these structures can be updated in the future(and client can detect if the version is different than what they know) and moved the text printer helpers from docker/buildx#1100 to here so more clients can print that output directly.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

jedevc · 2022-08-05T14:21:09Z

frontend/subrequests/outline/outline.go

+	"github.com/moby/buildkit/solver/pb"
+)
+
+const RequestSubrequestsOutline = "frontend.outline"


This feels weird to me, since the describe request is of the form "frontend.subrequests.describe". I get that that describe is meta about the other subrequests, but it means that we have the weird feature that a client querying on buildx has: outline, targets and subrequests.describe. I think we should be able to hide from the user that we have subrequests behind --print, at least on the buildx side - maybe this is something to solve on the client instead, not sure.

Yes, this is bit weird. But I didn't want users to need to type in --print subrequests.outline. On the same time I don't want to break subrequests.describe because we already released that some time ago and maybe somebody is using it. I also think that --print describe printing the list of possible commands is a bit misleading(user would more likely expect it to describe their build).

jedevc · 2022-08-05T14:44:27Z

cmd/buildctl/build.go

+		return err
+	}
+
+	if txt, ok := subMetadata["result.txt"]; ok {


Would we want a case for result.json as well? Feels like we could format it here, in case the frontend hasn't done it.

We would need a specific printer for this and I don't want buildctl to have too much hardcoded info about specific requests.

Or are you saying that if there is a result.json we should only print that directly and not the full result like it is in current commit.

I guess we have an exception for result.txt, I'm just wondering why we shouldn't also have one for result.json. I think at the least, we should print directly and not the whole result if we find a result.json similar to result.txt.

Although maybe for buildctl, since it's a dev tool, we should assume no knowledge of the response, and always print out everything?

and always print out everything?

That what it does atm. There is a special case for result.txt, so if it exists then the expectation is that frontend has already formatted the result and we can just print that output. Otherwise we just print whatever frontend provided(including result.json). If it makes more sense I can make it so that it only prints result.json if it exists and skips the rest.

cpuguy83

I've been using ListTargets and Dockerfile2Outline to great affect here: https://github.com/deislabs/gnarly/blob/a4b11b88d008a64ae12c75712718d0400a309c7f/generate.go#L29-L49

Definitely looking forward to having this in the API.

frontend/dockerfile/dockerfile2llb/convert.go

Even though this is a pretty invalid use-case there are cases where these could carry some information. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the outline branch 3 times, most recently from f3e7028 to 058ebce Compare May 3, 2022 06:52

This was referenced May 6, 2022

Dockerfile: update comments #2852

Merged

Dockerfile: update comments tonistiigi/binfmt#86

Merged

crazy-max reviewed May 6, 2022

View reviewed changes

tonistiigi force-pushed the outline branch from c8919b8 to 7252911 Compare May 6, 2022 20:42

tonistiigi mentioned this pull request May 7, 2022

Build: Support for printing outline/targets of the current build docker/buildx#1100

Merged

AkihiroSuda requested a review from ktock July 23, 2022 03:58

tonistiigi force-pushed the outline branch 2 times, most recently from fc9c0aa to 656f6fb Compare August 1, 2022 05:20

ktock reviewed Aug 2, 2022

View reviewed changes

bpaquet reviewed Aug 2, 2022

View reviewed changes

cmd/buildctl/build.go Outdated Show resolved Hide resolved

tonistiigi added 8 commits August 4, 2022 21:16

dockerfile: support for outline requests

ec9a3ac

Adds a new subrequest frontend.outline that returns all the build arguments the current build definition takes without actually running the build. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

buildctl: print subrequest metadata result to stdout

b1a2aa7

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

dockerfile: add secret and ssh information to outline

3b1fd05

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

dockerfile: add request for listing targets

f950ba7

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

dockerfile: add target stage information to outline

e7c8b22

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

subrequests: add base and platform to targets request

b88e435

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

subrequest: always return version with subrequest result

9924649

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the outline branch from 656f6fb to f7bbe2f Compare August 5, 2022 04:46

tonistiigi requested a review from ktock August 5, 2022 05:36

ktock approved these changes Aug 5, 2022

View reviewed changes

subrequests: normalize json fields to lowercase

d1b0d8a

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the outline branch from b03799a to d1b0d8a Compare August 5, 2022 07:40

jedevc reviewed Aug 5, 2022

View reviewed changes

cpuguy83 reviewed Aug 5, 2022

View reviewed changes

frontend/dockerfile/dockerfile2llb/convert.go Outdated Show resolved Hide resolved

dockerfile: remove filtering out empty targets

28a156b

Even though this is a pretty invalid use-case there are cases where these could carry some information. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the outline branch from 3e7724b to 28a156b Compare August 8, 2022 19:11

jedevc approved these changes Aug 9, 2022

View reviewed changes

crazy-max approved these changes Aug 9, 2022

View reviewed changes

tonistiigi merged commit 8488654 into moby:master Aug 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dockerfile: support for outline requests #2841

dockerfile: support for outline requests #2841

tonistiigi commented May 2, 2022

AkihiroSuda commented May 2, 2022

tonistiigi commented May 2, 2022

crazy-max May 6, 2022 •

edited

tonistiigi May 6, 2022

AkihiroSuda commented Jul 23, 2022

ktock commented Jul 25, 2022 •

edited

tonistiigi commented Jul 26, 2022

ktock commented Jul 27, 2022

ktock left a comment

ktock Aug 2, 2022

tonistiigi Aug 2, 2022

tonistiigi commented Aug 2, 2022

tonistiigi commented Aug 5, 2022

jedevc Aug 5, 2022

tonistiigi Aug 5, 2022

jedevc Aug 5, 2022

tonistiigi Aug 5, 2022

jedevc Aug 5, 2022

tonistiigi Aug 5, 2022

cpuguy83 left a comment

dockerfile: support for outline requests #2841

dockerfile: support for outline requests #2841

Conversation

tonistiigi commented May 2, 2022

AkihiroSuda commented May 2, 2022

tonistiigi commented May 2, 2022

crazy-max May 6, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Jul 23, 2022

ktock commented Jul 25, 2022 • edited

tonistiigi commented Jul 26, 2022

ktock commented Jul 27, 2022

ktock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonistiigi commented Aug 2, 2022

tonistiigi commented Aug 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpuguy83 left a comment

Choose a reason for hiding this comment

crazy-max May 6, 2022 •

edited

ktock commented Jul 25, 2022 •

edited