[Bug]: Docker Build Fails only with Testcontainers

[DanielHabenicht]

Testcontainers version

3.2.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x64

.NET version

6.0.404

Docker version

Client:
 Cloud integration: v1.0.31
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:28:08 2023
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.18.0 (104112)
 Engine:
  Version:          20.10.24
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.7
  Git commit:       5d6db84
  Built:            Tue Apr  4 18:18:42 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.4)
  compose: Docker Compose (Docker Inc., v2.17.2)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.19)
  init: Creates Docker-related starter files for your project (Docker Inc., v0.1.0-beta.2)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.9.0)

Server:
 Containers: 100
  Running: 0
  Paused: 0
  Stopped: 100
 Images: 183
 Server Version: 20.10.24
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 49.97GiB
 Name: docker-desktop
 ID: XNNW:ZAMV:SXVL:PJJ5:FSVK:MU7H:B3QR:FKLA:BU5B:L4DF:B5B4:FUI2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

What happened?

Following Dockerfile buils with docker but not with Testcontainers:

FROM smallstep/step-ca AS certs
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
RUN mkdir -p exampleCA
RUN step certificate create "Smallstep Root CA" "/home/step/exampleCA/cacert.pem" "/home/step/exampleCA/cakey.pem" \
    --no-password --insecure \
    --profile root-ca \
    --not-before "2021-01-01T00:00:00+00:00" \
    --not-after "2031-01-01T00:00:00+00:00" \
    --san "example.com" \
    --san "mail.example.com" \
    --kty RSA --size 2048

via

        TestcontainersSettings.Logger = logger;
        var image = new ImageFromDockerfileBuilder()
            .WithDockerfileDirectory(CommonDirectoryPath.GetSolutionDirectory(), Path.Combine("services", "mail"))
            .WithDockerfile("Dockerfile")
            .WithBuildArgument("RESOURCE_REAPER_SESSION_ID", ResourceReaper.DefaultSessionId.ToString("D"))
            // .WithCleanUp(true)
            .Build();
        await image.CreateAsync().ConfigureAwait(false);

Relevant log output

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:06.56]     AclTool.MailClient.Tests.MailClientTest.SendRetrieve [FAIL]
  Failed AclTool.MailClient.Tests.MailClientTest.SendRetrieve [5 s]
  Error Message:
   System.InvalidOperationException : Docker image localhost/testcontainers/42c1e266-ace0-4ed9-b169-84363ddc3c07:latest has not been created.
  Stack Trace:
     at DotNet.Testcontainers.Clients.DockerImageOperations.BuildAsync(IImageFromDockerfileConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.BuildAsync(IImageFromDockerfileConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Images.FutureDockerImage.UnsafeCreateAsync(CancellationToken ct)
   at DotNet.Testcontainers.Images.FutureDockerImage.CreateAsync(CancellationToken ct)
   at AclTool.MailClient.Tests.MailClientTest.SendRetrieve() in C:\Develop\TFS\SVN.AclTool\src\AclTool.MailClient.Tests\MailClientTest.cs:line 31
--- End of stack trace from previous location ---
  Standard Output Messages:
 info [] Connected to Docker:
   Host: npipe://./pipe/docker_engine
   Server Version: 20.10.24
   Kernel Version: 5.10.102.1-microsoft-standard-WSL2
   API Version: 1.41
   Operating System: Docker Desktop
   Total Memory: 49.97 GB
 info [] Pattern ^(.+)\/\.idea added to the regex cache
 info [] Pattern ^([\\\/]?(\.idea\b|$)) added to the regex cache
 info [] Pattern ^([\\\/]?((.+)\/\.vs\b|$)) added to the regex cache
 info [] Pattern ^([\\\/]?(\.vs\b|$)) added to the regex cache
 info [] Pattern ^([\\\/]?(\.dockerignore\b|$)) added to the regex cache
 info [] Pattern ^([\\\/]?(Dockerfile\b|$)) added to the regex cache
 trce [] Step 1/13 : FROM smallstep/step-ca AS certs
 trce []  ---> 624886655681

 trce [] Step 2/13 : ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
 trce []  ---> Using cache

 trce []  ---> 2d37b22a6e0e

 trce [] Step 3/13 : LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
 trce []  ---> Running in ce5cc26dce8b

 trce [] Removing intermediate container ce5cc26dce8b

 trce []  ---> a9e364896075

 trce [] Step 4/13 : RUN pwd
 trce []  ---> Running in 277b0851559b

 trce [] /home/step

 trce [] Removing intermediate container 277b0851559b

 trce []  ---> 8aad84034619

 trce [] Step 5/13 : RUN mkdir -p exampleCA
 trce []  ---> Running in 548e0b6db0ab

 trce [] Removing intermediate container 548e0b6db0ab

 trce []  ---> 0248862a6a65

 trce [] Step 6/13 : RUN ls /home/step
 trce []  ---> Running in 77e61f318fe4

 trce [] Removing intermediate container 77e61f318fe4

 trce []  ---> 6652a344d349

 trce [] Step 7/13 : RUN touch test
 trce []  ---> Running in 457e448e0b04

 trce [] Removing intermediate container 457e448e0b04

 trce []  ---> 8955d3a50664

 trce [] Step 8/13 : RUN step certificate create "Smallstep Root CA" "/home/step/exampleCA/cacert.pem" "/home/step/exampleCA/cakey.pem"     --no-password --insecure     --profile root-ca     --not-before "2021-01-01T00:00:00+00:00"     --not-after "2031-01-01T00:00:00+00:00"     --san "example.com"     --san "mail.example.com"     --kty RSA --size 2048
 trce []  ---> Running in 3486f349862a

 trce [] \x1b[91mopen /home/step/exampleCA/cakey.pem: no such file or directory
\x1b[0m
 fail [] The command '/bin/sh -c step certificate create "Smallstep Root CA" "/home/step/exampleCA/cacert.pem" "/home/step/exampleCA/cakey.pem"     --no-password --insecure     --profile root-ca     --not-before "2021-01-01T00:00:00+00:00"     --not-after "2031-01-01T00:00:00+00:00"     --san "example.com"     --san "mail.example.com"     --kty RSA --size 2048' returned a non-zero code: 1



Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: < 1 ms - AclTool.MailClient.Tests.dll (net6.0)

Additional information

Works if executed with docker build .

Seems like testcontainers is still using the old docker build pipeline and not buildkit?

[DanielHabenicht]

Seems like its not a Testcontainers issue by itself, it only occurs with BuilKit disabled (e.g. $env:DOCKER_BUILDKIT = 0). Is there a way to build with buildkit enabled?

[DanielHabenicht]

The legacy docker build engine seems to recreate the folder that is created in the step before, thus erroring in the subsequent step as it is empty.

[HofmeisterAn]

Removing intermediate container 548e0b6db0ab

Interessting observation, I think due to Docker removing the intermediate layer, the previous steps are not present in the final step. For example, if you look into RUN mkdir -p exampleCA, you will notice that the exampleCA directory is missing. That's why you receive the following error message:

/home/step/exampleCA/cakey.pem: no such file or directory

If you chain the commands and run them in a single command, it works fine.

Seems like its not a Testcontainers issue by itself, it only occurs with BuilKit disabled (e.g. $env:DOCKER_BUILDKIT = 0). Is there a way to build with buildkit enabled?

We encountered a similar request in Node, and it appears that BuildKit is not available through the Docker Engine API: testcontainers/testcontainers-node#571 (comment).

[DanielHabenicht]

Yeah, I resolved it for now by writing the files into the /tmp directory which does not seem to be reset between layers.
The idea for another folder came from this comment: https://forums.docker.com/t/resolved-files-missing-after-dockerfile-run-downloads-them/4827

[DanielHabenicht]

We encountered a similar request in Node, and it appears that BuildKit is not available through the Docker Engine API: testcontainers/testcontainers-node#571 (comment).

Hopefully that will change as they deprecate it and add an EoL

[DanielHabenicht]

I dont't know where to file it, but probably a central ticket for buildkit compatibiliy would be better than one in each seperate project.
Are the container build by ressource reaper or by each testcontainers implementation individually?

[HofmeisterAn]

If you like, we can create it here. I can pin the issue and forward it to the other Testcontainers maintainers.