Unable to download a file with ADD because of the user agent

[edlundin]

Description
The ADD instruction uses the user agent go-http-client/1.1 when the source is an URL. If for some reason this user agent is blacklisted, downloading a file using ADD becomes impossible.

Context
I was trying to bust a cached git repository, cloned from my company's own repositories, using ADD. Unfortunately, my company has a list of banned user agents, including go-http-client/1.1, that prevents me from downloading a file with this instruction.
I am aware that several workarounds exist, hence this issue is not a priority, but for this use case, nothing is as simple as using ADD.

Describe the results you received:
The build fails with a message similar to failed to load cache key: Get $URL: EOF.
Where $URL is the one fed to the src argument of the ADD instruction.

Describe the results you expected:
The file to be downloaded by the ADD instruction.

Possible solution:
I believe that if there was an optional flag --user-agent, to set the user agent used by ADD, it would fix the issue. Since the flag would be optional, go-http-client/1.1 would still be the default user agent.

Output of docker version:

Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:14:53 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.6.0)

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 4
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 7.771GiB
 Name: docker-desktop
 ID: F35V:M5AT:P6EN:5TI6:GUTN:V7Y5:RPOW:OFYY:KWIF:LYVY:TMUE:ZITH
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 42
  Goroutines: 46
  System Time: 2021-04-24T09:00:52.2080478Z
  EventsListeners: 4
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details (AWS, VirtualBox, physical, etc.):
Docker images are mainly built inside WSL2.

[thaJeztah]

Looks to be the same for both buildkit and the classic builder;

On Docker Desktop for mac; In one terminal

docker run -it --rm -p 8080:80 nginx:alpine

In another shell;

DOCKER_BUILDKIT=0 docker build -<<EOF
FROM busybox
ADD http://host.docker.internal:8080 /foo.html
EOF

DOCKER_BUILDKIT=1 docker build -<<EOF
FROM busybox
ADD http://host.docker.internal:8080 /foo.html
EOF

Which prints these in the logs of the nginx container:

172.17.0.1 - - [26/Apr/2021:15:56:11 +0000] "GET / HTTP/1.1" 200 612 "-" "Go-http-client/1.1" "-"
172.17.0.1 - - [26/Apr/2021:15:56:30 +0000] "GET / HTTP/1.1" 200 612 "-" "Go-http-client/1.1" "-"

[edlundin]

I'm trying to write a PR for this feature.

It is my understanding that the ADD instruction is first parsed from a Dockerfile here:

https://github.com/moby/buildkit/blob/70deac12b5857a1aa4da65e90b262368e2f71500/frontend/dockerfile/instructions/parse.go#L239-L254

The parsed data are stored in this struct:

https://github.com/moby/buildkit/blob/70deac12b5857a1aa4da65e90b262368e2f71500/frontend/dockerfile/instructions/commands.go#L183-L193

Then, it's handled by moby via:

moby/builder/dockerfile/dispatchers.go

Lines 89 to 110 in 797b974

	// ADD foo /path
	//
	// Add the file 'foo' to '/path'. Tarball and Remote URL (http, https) handling
	// exist here. If you do not wish to have this automatic handling, use COPY.
	//
	func dispatchAdd(d dispatchRequest, c *instructions.AddCommand) error {
	if c.Chmod != "" {
	return errors.New("the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled")
	}
	downloader := newRemoteSourceDownloader(d.builder.Output, d.builder.Stdout)
	copier := copierFromDispatchRequest(d, downloader, nil)
	defer copier.Cleanup()
	copyInstruction, err := copier.createCopyInstruction(c.SourcesAndDest, "ADD")
	if err != nil {
	return err
	}
	copyInstruction.chownStr = c.Chown
	copyInstruction.allowLocalDecompression = true
	return d.builder.performCopy(d, copyInstruction)
	}

Coincidentally, I forked both moby/buildkit and moby/moby, since one parses the instruction and the other acts upon it.
I made changes on each fork, but while I can test each project individually, it seems that I can't use my forked buildkit inside my forked moby.

Does anybody know how I could test my implementation? I thought of modifying the imports during the tests, but it's not a realistic solution.