HttpHeaders empty for build commands

[orodbhen]

Description

For compose plugin 2.12.2 (maybe earlier), build commands fail to pass the HttpHeaders field to the engine from the config.json file. HttpHeaders is still present, but empty.

This is also an issue for docker build, at least as recent as 23.0.4, but still working in 20.10.12.

This breaks plugins that require information to be passed in header fields.

Reverting client to 20.10.12, while leaving engine at 23.0.4, fixes docker build but not docker compose build.

Other docker commands appear to work without issues.

Reproduce

1. Run docker compose build or docker build with a plugin installed that requires HttpHeaders.

Expected behavior

All client commands pass the HttpHeaders field through to the engine for use by plugins.

docker version

Client: Docker Engine - Community
 Version:           23.0.4
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        f480fb1
 Built:             Fri Apr 14 10:36:38 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.4
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       cbce331
  Built:            Fri Apr 14 10:34:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 23.0.4
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Authorization: authz
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: /etc/docker/seccomp-profile.json
 Kernel Version: 3.10.0-1160.88.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.625GiB
 Name: myhost.example.com
 ID: JMUE:X72S:CTW3:EG6I:GLEN:HGQ5:OMIK:4WWI:RMPI:ZW3W:XULS:DNHI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Operating System: CentOS 7.9

[thaJeztah]

Does it work with docker build on 23.0 but when setting DOCKER_BUILDKIT=0 ?

Docker 23.0 uses buildx as default for docker build, which is also used by docker compose; I suspect here that buildx currently doesn't use this config, or it's not set if it's connecting directly using the BuildKit grpc API.

[orodbhen]

Does it work with docker build on 23.0 but when setting DOCKER_BUILDKIT=0 ?

Docker 23.0 uses buildx as default for docker build, which is also used by docker compose; I suspect here that buildx currently doesn't use this config, or it's not set if it's connecting directly using the BuildKit grpc API.

Yes, it works with DOCKER_BUILDKIT=0. Besides this, enabling BuildKit also seems to cause insecure-registries to be ignored. Of course, that's a different issue.

[thaJeztah]

Thanks for testing that assumption!

This breaks plugins that require information to be passed in header fields.

Would you have an example of a plugin (and how / for what purpose the header would be used?). Is this to (e.g.) block build actions through an authz plugin? (I'm wondering if buildx is hitting the BuildKit gRPC API if any of that would still be possible).

Besides this, enabling BuildKit also seems to cause insecure-registries to be ignored. Of course, that's a different issue.

That looks to be what's reported in #43268 (but there may be other tickets). I seem to recall I've seen some work on this being done somewhere, but don't know what the status is.

@tonistiigi @jedevc could one of you help fill in the blanks here?

[tonistiigi]

Buildx only uses the grpc endpoint. Only HTTP endpoints that are hit are /grpc and optionally /session.

That looks to be what's reported in #43268 (but there may be other tickets).

There are no known issues with a reproducer for this that I'm aware of in the latest releases.

[orodbhen]

Thanks for testing that assumption!

This breaks plugins that require information to be passed in header fields.

Would you have an example of a plugin (and how / for what purpose the header would be used?). Is this to (e.g.) block build actions through an authz plugin? (I'm wondering if buildx is hitting the BuildKit gRPC API if any of that would still be possible).

Besides this, enabling BuildKit also seems to cause insecure-registries to be ignored. Of course, that's a different issue.

That looks to be what's reported in #43268 (but there may be other tickets). I seem to recall I've seen some work on this being done somewhere, but don't know what the status is.

@tonistiigi @jedevc could one of you help fill in the blanks here?

Its an in-house AuthZ plugin that we use, which verifies user identity and access based on a set of HTTP headers stored in the user's ~.docker/config.json file. I haven't seen any issues with commands other than build, but there are probably some we haven't tested.

[orodbhen]

If this isn't intended to work with buildkit, may what's needed is a doc update to https://docs.docker.com/engine/reference/commandline/cli/.

The property HttpHeaders specifies a set of headers to include in all messages sent from the Docker client to the daemon. Docker does not try to interpret or understand these headers; it simply puts them into the messages. Docker does not allow these headers to change any headers it sets for itself.

[thaJeztah]

Is it actually hitting the build endpoint (but without the headers) or are builds not hitting the endpoint? (as mentioned above, building with buildx would hit the /grpc or /session endpoints, not /build)

[orodbhen]

Actually, I'm seeing this for docker compose run as well.

I'll check on the endpoints.

[orodbhen]

Actually, I'm not able to reproduce the compose run error, so I'm not sure of the conditions that caused it.

I'm seeing the issue consistently for endpoints /grpc and /containers/buildx_buildkit_default/json.

[orodbhen]

Also, this setting appears to have no effect in /etc/docker/daemon.json:

"features": {
  "buildkit": false
}

The daemon was restarted after updating the file.