Cannot specify --storage-opt to "docker build"

[heaths]

Description

We are working on containerizing a very large application where the default volume size may be insufficient but cannot pass --storage-opt to docker build. Changing the daemon configuration may be a blocker for customers wishing to build upon our base image we're trying to build. We can change the amount of memory provided to the container(s) which is necessary during build, so it would be nice to be able to specify the volume size as well.

But rather than merely support storage-opt for the build command, would it make more sense to be allowed to pass through arguments from build to use when running containers for each RUN statement?

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:57:19 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:47 2017
 OS/Arch:      windows/amd64
 Experimental: true

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 28
Server Version: 17.06.2-ce
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 15063 (15063.0.amd64fre.rs2_release.170317-1834)
Operating System: Windows 10 Enterprise
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 31.92GiB
Name: heathswrk
ID: PPYQ:6372:4QVR:ERCP:U676:ZHX5:T6FG:RKYE:O4XN:FDP4:7DMT:CQ7P
Docker Root Dir: E:\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 32
 System Time: 2017-09-22T13:28:45.4667296-07:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Using Hyper-V isolation on Windows 10 (currently all that's supported).

[heaths]

I should mention I'm aware of docker/for-win#999 which also creates a problem since we have no way to increase volume size for docker build until that change is released to the stable build (since customers may not want to run preview bits).

[friism]

Docker CE 17.09 (stable) shouldn't be that far of, and will have the fix in the engine.

Further, for the users that are consuming your base image, will they also be generating incredibly large layers on top of your base image?

[heaths]

Possibly. There's basically two goals here. We're containerizing Build Tools for Visual Studio 2017 but its not yet as feature-rich (i.e. doesn't support building all common project types) as Visual Studio 2017 itself, so even if we made a base image of Build Tools available on which customers may build, customers may also need to install Visual Studio 2017 into an image for now. So besides chasing down any blocking bugs to just building a complete image ourselves, I also need to provide guidance for customers building their own. So being able to pass more options (or just specific ones, like --storage-opt) to control intermediate containers would be very helpful.

[darstahl]

I agree that this is something we want. We've had multiple asks for this (ex. #33469), and it's much more reasonable to have a base image size specified at build time than being a global daemon option imo.

cc @jhowardmsft @johnstep

[heaths]

After discussing different options for implementation with @jhowardmsft and @johnstep, I'm working on a change to add --storage-opt to the build command akin to the run command.

[IvanBoyko]

Apologies for bringing up an old topic, but it seems it was never implemented.

We've been using VS Build Tools 2017 since August 2017, and never had this problem.
Our installation works, although I do see in %temp%\dd_client_*.log Errors like this:

2018-11-29T07:35:46 : Error : Visual Studio needs at least 25.63 GB of disk space. You might want to free up space on C:\ or change your target drive.

[heaths]

@johnstep created PR #35780 but it was rejected after the default size was raised to 127GB (same as the default for Hyper-V). Apart from being rather short-sighted, IMO (using build tools containers for large solutions might require even more space, which means it should be configurable), I don't know if this made it into Docker 4 Windows in either the edge or stable channels. It's been nearly a year so I'd hope so.

[crossan007]

Following this, as I'm also looking to create large base images.

[lowenna]

@crossan007 How large? The default scratch is now 127GB. Bigger than that?

[JoshuaKing]

@jhowardmsft New here. By default scratch do you mean the max size is 127GB for a container being built? I ask because I'm trying to build a simple container following Microsoft's instructions (install Visual Studio Build Tools).

When building, after pulling and extracting the images I still have 50GB on the windows host machine remaining. Then it conducts the Visual Studio Build Tools install and starts rapidly consuming disk space (about 20GB consumed) but I still have 30GB to work with. Part way through the install it fails with the out-of-disk-space error:

Error 0x80070070: Failed to write to part stream.
Error 0x80070070: Failed to write to archive file stream.
!ERROR: 0x80070070 [There is not enough space on the disk.]

I am guessing that docker build on Windows at least, does not allow the container to consume 127GB and has a low limit of 20GB? Is there any way around this?

[heaths]

@jhowardmsft, certain installs I manage are also starting to grow beyond 127GB for a full install of all first- and third-party software. We will need larger sizes - something like my original change would have partly provided. Seems only fitting that if docker run can specify a larger size, docker build should be able to as well.

[crossan007]

I'm working on a CI process for an application where my base layer is will be ~156GB. The 127GB cap is causing the build phase to fail.

On top of this base layer, I intend to add components built by the current run of my CI process.

I've added the following to my C:\ProgramData\Docker\Config\Daemon.json file, but seems to have no effect:

"storage-opts": [
    "size=300GB"
  ]

[crossan007]

An option that would also fit my scenario (but I haven't really found) would be the ability to mount a volume to a container having "writefilter" mechanism in use so that the volume itself is immutable, and writes to the volume only affect the layer representing the mount to the current container.

[TBBle]

I've added the following to my C:\ProgramData\Docker\Config\Daemon.json file, but seems to have no effect:

"storage-opts": [
    "size=300GB"
  ]

This should be fixed by #41636 in the 21.xx release.

There still isn't an implementation of --storage-opt for docker build though, so you'd have to set the config large enough for all your possible needs in the meantime. ue4-docker needs 600GB for this value in some configurations, and that has been tested successfully.

[slonopotamus]

20GB limit for COPY was never merged to 20.10.x branch.

[omrumcetin]

Do you have any kind of plan to solve this, we can not use windows containers with big images.

[TBBle]

Once Docker v22.06 is released, it will be possible to work with large images in docker build by setting a large storage-opt size in the daemon config file. --storage-opt however has not been made available for docker build for per-build configuration behaviour.

The costs of making the config file number quite large are minimal (the VHD file created expands on-demand so builds won't use much more space when run with a larger size in the config) and since plumbing storage-opts through to the builder hasn't been done, I assume it's not seen as particularly high value for any contributors given the config-level setting bug-fix above.

I'll note also that #35780 in 2018 attempted to implement this and hit some unresolved issues. I haven't checked to see if those issues still apply though.

[heaths]

@TBBle it was done but rejected. It's been long enough I imagine it would have to be done again, but I'd be happy to contribute. The problem with the config file is that users have to change it every time they need to increase the build size. The convenience of plumbing through the command line option is being able to change it as needed only - same as with passing storage opts to docker run or even memory opts to docker build.

[garyo]

I certainly wish this would be implemented -- ideally in the Dockerfile, but on the docker build command line at least. Having to ship a README telling users to edit their docker daemon config to build an image is annoying.