control: add info service #2725

crazy-max · 2022-03-14T08:25:50Z

adds version service that returns BuildKit package, version and revision:

$ buildctl debug version
github.com/moby/buildkit v0.10.0-6-g7c3e9fdd.m 7c3e9fdd48c867f48a07a80cde64cc2d578cb332.m

could be handy for buildx:

$ docker buildx ls
NAME/NODE     DRIVER/ENDPOINT                        STATUS                 BUILDKIT VERSION      PLATFORMS
builder       docker-container
  builder0    unix:///var/run/docker.sock            stopped
  mac-mini-m1 tcp://mac-mini-m1.home.xxxxxx.com:2376 stopped                                      linux/arm64*
  sifive      tcp://sifive.home.xxxxxx.com:2375      running                N/A                   linux/riscv64*
builder2 *    docker-container
  builder20   unix:///var/run/docker.sock            running                N/A                   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64    
builder3      docker-container
  builder30   unix:///var/run/docker.sock            inactive
builder4      docker-container
  builder40   unix:///var/run/docker.sock            running                v0.10.0-6-g7c3e9fdd.m linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64    
mybuilder     docker-container
  mybuilder0  unix:///var/run/docker.sock            inactive
test          docker-container
  test0       unix:///var/run/docker.sock            inactive
desktop-linux                                        protocol not available
default       docker
  default     default                                running                N/A                   linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

crazy-max · 2022-03-14T08:47:33Z

Test that looks flaky but not related to this PR: https://github.com/moby/buildkit/runs/5534127937?check_suite_focus=true#step:6:1092 cc @sipsma

=== CONT  TestIntegration/TestDiffUpperScratch/worker=containerd-1.5
    mergediff_test.go:1298: 
        	Error Trace:	client_test.go:5195
        	            				mergediff_test.go:1298
        	            				run.go:202
        	Error:      	Received unexpected error:
        	            	failed to apply diffs: 1 error occurred:
        	            		* failed to create differ: 1 error occurred:
        	            		* failed to mount /tmp/bktest_buildkitd1020045742/tmp/buildkit-mount104196041: [{Type:bind Source:/tmp/bktest_containerd1162610903/root/io.containerd.snapshotter.v1.overlayfs/snapshots/19/fs Options:[ro rbind]}]: no such file or directory
        	            	
        	            	
        	            	
        	            	
        	            	github.com/moby/buildkit/util/stack.Enable
        	            		/src/util/stack/stack.go:77
        	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
        	            		/src/util/grpcerrors/grpcerrors.go:188
        	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        	            		/src/util/grpcerrors/intercept.go:41
        	            	google.golang.org/grpc.(*ClientConn).Invoke
        	            		/src/vendor/google.golang.org/grpc/call.go:35
        	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        	            		/src/api/services/control/control.pb.go:1552
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:208
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1581
        	            	failed to solve
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:221
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1581
        	Test:       	TestIntegration/TestDiffUpperScratch/worker=containerd-1.5

errordeveloper · 2022-03-14T12:37:25Z

api/services/control/control.proto

+	string package = 1;
+	string version = 2;
+	string revision = 3;


is this basically semver or something else?

api/services/control/control.proto

tonistiigi · 2022-03-15T01:44:21Z

api/services/control/control.proto

@@ -157,3 +158,11 @@ message ListWorkersRequest {
 message ListWorkersResponse {
 	repeated moby.buildkit.v1.types.WorkerRecord record = 1;


I think it would be good to add it to WorkerRecord as well in case in the future we will allow worker versions mismatch.

crazy-max · 2022-03-15T16:40:28Z

Last commit adds a new field to expose built-in dockerfile frontend version. This one is a bit tedious so might be consider in a follow-up if you want.

I'm using a comparison table in a JSON file to guess the dockerfile frotnend version linked to the buildkit one: https://github.com/moby/buildkit/pull/2725/files#diff-edcbcc2959221b86d31f2cda9e7d365d8e317e111d4f5913c149a7ee40005dd1R1-R32. This JSON file could be reused elsewhere like displaying the frontend version linked to the BuildKit one on Moby. If buildkit version is a dirty one, we try to find the closest one. Will look like this:

$ buildctl debug info
BuildKit:   
  Package:  github.com/moby/buildkit
  Version:  v0.10.0-7-g50735431.m
  Revision: 5073543122a9b8d043db23a5c7c2dbad9ce945db.m

Built-in Dockerfile Frontend:
  Version:                    ~1.4.0

api/services/control/control.proto

tonistiigi · 2022-03-16T16:43:26Z

cmd/buildctl/debug/info.go

+	"github.com/urfave/cli"
+)
+
+var InfoCommand = cli.Command{


Why is the command needed? We already have version command. Is there a valid use-case?

We don't have a version command, only the --version flag. I think info is only for debug purposes like the workers one.

If we add this also to WorkerRecord then it can just be shown on exiting commands. Trying to avoid adding new things if there isn't a clear need for them.

tonistiigi · 2022-03-16T16:46:48Z

frontend/dockerfile/release/releases.json

@@ -0,0 +1,32 @@
+{
+  "0.10.0": "1.4.0",


I'd rather avoid this. It is messy to manage. If someone needs this information it can be implied from the buildkit revision. Users should not try to detect the default version but put a fixed version on top of the Dockerfile.

Sure will remove this commit. I still think it's useful to know this information. Searching through GitHub Releases is not easy as buildkit releases are not tied to the frontend one. Agree that it's a poor implementation to handle that. Will think about a better way to do it in a follow-up.

api/services/control/control.proto

tonistiigi · 2022-03-16T16:48:50Z

api/services/control/control.proto

@@ -157,3 +158,11 @@ message ListWorkersRequest {
 message ListWorkersResponse {
 	repeated moby.buildkit.v1.types.WorkerRecord record = 1;


Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max · 2022-04-08T11:12:58Z

added to worker record:

$ buildctl debug workers -v
ID:             nqb09zw7vu11yer337j323rek
Platforms:      linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6
BuildKit:       github.com/moby/buildkit v0.10.0-43-gd2a757f8 d2a757f862957bf8a4cfb6f1bb990fa6dc33d153
Labels:
        org.mobyproject.buildkit.worker.executor:               oci
        org.mobyproject.buildkit.worker.hostname:               pc-sff
        org.mobyproject.buildkit.worker.network:                host
        org.mobyproject.buildkit.worker.oci.process-mode:       sandbox
        org.mobyproject.buildkit.worker.snapshotter:            overlayfs
GC Policy rule#0:
        All:            false
        Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
        Keep Duration:  48h0m0s
        Keep Bytes:     512MB
GC Policy rule#1:
        All:            false
        Keep Duration:  1440h0m0s
        Keep Bytes:     26GB
GC Policy rule#2:
        All:            false
        Keep Bytes:     26GB
GC Policy rule#3:
        All:            true
        Keep Bytes:     26GB

$ buildctl debug workers --format "{{json .}}" | jq
[
  {
    "id": "nqb09zw7vu11yer337j323rek",
    "labels": {
      "org.mobyproject.buildkit.worker.executor": "oci",
      "org.mobyproject.buildkit.worker.hostname": "pc-sff",
      "org.mobyproject.buildkit.worker.network": "host",
      "org.mobyproject.buildkit.worker.oci.process-mode": "sandbox",
      "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
    },
    "platforms": [
      {
        "architecture": "amd64",
        "os": "linux"
      },
      {
        "architecture": "amd64",
        "os": "linux",
        "variant": "v2"
      },
      {
        "architecture": "amd64",
        "os": "linux",
        "variant": "v3"
      },
      {
        "architecture": "arm64",
        "os": "linux"
      },
      {
        "architecture": "riscv64",
        "os": "linux"
      },
      {
        "architecture": "ppc64le",
        "os": "linux"
      },
      {
        "architecture": "s390x",
        "os": "linux"
      },
      {
        "architecture": "386",
        "os": "linux"
      },
      {
        "architecture": "mips64le",
        "os": "linux"
      },
      {
        "architecture": "mips64",
        "os": "linux"
      },
      {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      }
    ],
    "gcPolicy": [
      {
        "filter": [
          "type==source.local,type==exec.cachemount,type==source.git.checkout"
        ],
        "all": false,
        "keepDuration": 172800000000000,
        "keepBytes": 512000000
      },
      {
        "filter": null,
        "all": false,
        "keepDuration": 5184000000000000,
        "keepBytes": 26000000000
      },
      {
        "filter": null,
        "all": false,
        "keepDuration": 0,
        "keepBytes": 26000000000
      },
      {
        "filter": null,
        "all": true,
        "keepDuration": 0,
        "keepBytes": 26000000000
      }
    ],
    "buildkitVersion": {
      "Package": "github.com/moby/buildkit",
      "Version": "v0.10.0-43-gd2a757f8",
      "Revision": "d2a757f862957bf8a4cfb6f1bb990fa6dc33d153"
    }
  }
]

cmd/buildctl/debug/info.go

crazy-max marked this pull request as ready for review March 14, 2022 08:38

crazy-max mentioned this pull request Mar 14, 2022

ls: display buildkit version of the nodes docker/buildx#998

Merged

errordeveloper reviewed Mar 14, 2022

View reviewed changes

tonistiigi reviewed Mar 15, 2022

View reviewed changes

crazy-max changed the title ~~control: add version service~~ control: add info service Mar 15, 2022

crazy-max force-pushed the control-buildkit-version branch 2 times, most recently from 89c8051 to 896bae4 Compare March 15, 2022 16:33

crazy-max force-pushed the control-buildkit-version branch from 896bae4 to 43ad8ab Compare March 15, 2022 17:01

sipsma mentioned this pull request Mar 15, 2022

snapshot: use temporary lease during diffapply #2729

Merged

tonistiigi reviewed Mar 16, 2022

View reviewed changes

crazy-max marked this pull request as draft March 16, 2022 18:13

crazy-max force-pushed the control-buildkit-version branch from 43ad8ab to 56f0715 Compare March 21, 2022 17:49

crazy-max mentioned this pull request Mar 22, 2022

TestDiffUpperScratch fails with dockerd worker #2746

Open

control: add info service

859ad49

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the control-buildkit-version branch from 56f0715 to 859ad49 Compare March 22, 2022 12:50

crazy-max force-pushed the control-buildkit-version branch from e269946 to 3469aa2 Compare April 8, 2022 11:05

control: add buildkit version to worker record

d2a757f

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the control-buildkit-version branch from 3469aa2 to d2a757f Compare April 8, 2022 11:06

crazy-max marked this pull request as ready for review April 8, 2022 11:13

crazy-max requested a review from tonistiigi April 8, 2022 11:13

AkihiroSuda reviewed Apr 8, 2022

View reviewed changes

cmd/buildctl/debug/info.go Show resolved Hide resolved

AkihiroSuda approved these changes Apr 8, 2022

View reviewed changes

tonistiigi approved these changes Apr 8, 2022

View reviewed changes

tonistiigi merged commit 86c33b6 into moby:master Apr 8, 2022

crazy-max deleted the control-buildkit-version branch April 8, 2022 23:06

ktock mentioned this pull request Jul 9, 2022

Display buildkit and runc version in nerdctl containerd/nerdctl#1209

Merged

crazy-max mentioned this pull request Aug 5, 2022

ci(dockerd): trigger through chat-ops #3003

Draft

thaJeztah mentioned this pull request Oct 4, 2022

[v0.10 backport] control: add info service #3145

Closed

crazy-max mentioned this pull request Apr 13, 2023

Set BuildKit version using buildinfo moby/moby#45304

Merged

crazy-max mentioned this pull request May 16, 2024

Expose default dockerfile syntax version being used docker/buildx#2459

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

control: add info service #2725

control: add info service #2725

crazy-max commented Mar 14, 2022

crazy-max commented Mar 14, 2022

errordeveloper Mar 14, 2022

tonistiigi Mar 15, 2022

tonistiigi Mar 16, 2022

crazy-max commented Mar 15, 2022 •

edited

tonistiigi Mar 16, 2022

crazy-max Mar 16, 2022

tonistiigi Mar 16, 2022

tonistiigi Mar 16, 2022

crazy-max Mar 16, 2022 •

edited

tonistiigi Mar 16, 2022

crazy-max commented Apr 8, 2022

		@@ -157,3 +158,11 @@ message ListWorkersRequest {
		message ListWorkersResponse {
		repeated moby.buildkit.v1.types.WorkerRecord record = 1;

control: add info service #2725

control: add info service #2725

Conversation

crazy-max commented Mar 14, 2022

crazy-max commented Mar 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max commented Mar 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max Mar 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max commented Apr 8, 2022

crazy-max commented Mar 15, 2022 •

edited

crazy-max Mar 16, 2022 •

edited