Merge pull request #93889 from claudiubelu/test-images/switch-to-buildx

Test images: switch to buildx

test/images/Makefile

@@ -33,10 +33,11 @@ endif
 all: all-container
 
 all-container:
-	./image-util.sh build $(WHAT)
+	./image-util.sh build $(WHAT) "docker"
 
-all-push: all-container
-	./image-util.sh push $(WHAT)
+all-push:
+	bash -x ./image-util.sh build $(WHAT) "registry"
+	bash -x ./image-util.sh push $(WHAT)
 
 all-build-and-push:
 	./image-util.sh build_and_push ${WHAT}

test/images/README.md

@@ -11,80 +11,18 @@ new images, test the changes made, promote the newly built staging images.
 
 ## Prerequisites
 
-In order to build the docker test images, a Linux node is required. The node will require `make`
-and `docker (version 18.06.0 or newer)`. Manifest lists were introduced in 18.03.0, but 18.06.0
-is recommended in order to avoid certain issues.
+In order to build the docker test images, a Linux node is required. The node will require `make`,
+`docker (version 19.03.0 or newer)`, and ``docker buildx``, which will be used to build multiarch
+images, as well as Windows images. In order to properly build multiarch and Windows images, some
+initialization is required:
 
-The node must be able to push the images to the desired container registry, make sure you are
-authenticated with the registry you're pushing to.
-
-Windows Container images are not built by default, since they cannot be built on Linux. For
-that, a Windows node with Docker installed and configured for remote management is required.
-
-
-### Windows node(s) setup
-
-In order to build the Windows container images, a node with Windows 10 or Windows Server 2019
-with the latest updates installed is required. The node will have to have Docker installed,
-preferably version 18.06.0 or newer.
-
-Keep in mind that the Windows node might not be able to build container images for newer OS versions
-than itself (even with `--isolation=hyperv`), so keeping the node up to date and / or upgrading it
-to the latest Windows Server edition is ideal.
-
-Windows test images must be built for Windows Server 2019 (1809) and Windows Server 1903, thus,
-if the node does not have Hyper-V enabled, or it is not supported, multiple Windows nodes are required,
-one per OS version.
-
-Additionally, remote management must be configured for the node's Docker daemon. Exposing the
-Docker daemon without requiring any authentication is not recommended, and thus, it must be
-configured with TLS to ensure that only authorised people can interact with it. For this, the
-following `powershell` script can be executed:
-
-```powershell
-mkdir .docker
-docker run --isolation=hyperv --user=ContainerAdministrator --rm `
-  -e SERVER_NAME=$(hostname) `
-  -e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP `
-  -v "c:\programdata\docker:c:\programdata\docker" `
-  -v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
-# restart the Docker daemon.
-Restart-Service docker
-```
-
-For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/).
-
-A firewall rule to allow connections to the Docker daemon is necessary:
-
-```powershell
-New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376
-```
-
-If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node.
-For example, in Azure, this is done by running the following command:
-
-```console
-az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376
-```
-
-The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker`
-will have to copied to the `~/.docker-${os_version)/` on the Linux build node, where `${os_version}`
-is `1809` or `1903`.
-
-```powershell
-scp.exe -r $env:USERPROFILE\.docker ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker-$os_version
-```
-
-After all this, the Linux build node should be able to connect to the Windows build node:
-
-```bash
-docker --tlsverify --tlscacert ~/.docker-${os_version}/ca.pem --tlscert ~/.docker-${os_version}/cert.pem --tlskey ~/.docker-${os_version}/key.pem -H "$REMOTE_DOCKER_URL" version
+```shell
+docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+docker buildx create --name img-builder --use
+docker buildx inspect --bootstrap
 ```
 
-For more information and troubleshooting about enabling Docker remote management, see
-[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost)
-
-Finally, the node must be able to push the images to the desired container registry, make sure you are
+The node must be able to push the images to the desired container registry, make sure you are
 authenticated with the registry you're pushing to.
 
 
@@ -136,6 +74,38 @@ The images are built through `make`. Since some images (e.g.: `busybox`) are use
 other images, it is recommended to build them first, if needed.
 
 
+### Windows test images considerations
+
+Ideally, the same `Dockerfile` can be used to build both Windows and Linux images. However, that isn't
+always possible. If a different `Dockerfile` is needed for an image, it should be named `Dockerfile_windows`.
+When building, `image-util.sh` will first check for this file name when building Windows images.
+
+The building process uses `docker buildx` to build both Windows and Linux images, but there are a few
+limitations when it comes to the Windows images:
+
+- The Dockerfile can have multiple stages, including Windows and Linux stages for the same image, but
+  the Windows stage cannot have any `RUN` commands (see the agnhost's `Dockerfile_windows` as an example).
+- The Windows stage cannot have any `WORKDIR` commands due to a bug (https://github.com/docker/buildx/issues/378)
+- When copying Windows symlink files to a Windows image, `docker buildx` changes the symlink target,
+  prepending `Files\` to them (https://github.com/docker/buildx/issues/373) (for example, the symlink
+  target `C:\bin\busybox.exe` becomes `Files\C:\bin\busybox.exe`). This can be avoided by having symlink
+  targets with relative paths and having the target duplicated (for example, the symlink target
+  `busybox.exe` becomes `Files\busybox.exe` when copied, so the binary `C:\bin\Files\busybox.exe`
+  should exist in order for the symlink to be used correctly). See the busybox's `Dockerfile_windows` as
+  an example.
+- `docker buildx` overwrites the image's PATH environment variable to a Linux PATH environment variable,
+  which won't work properly on Windows. See https://github.com/moby/buildkit/issues/1560
+- The base image for all the Windows images is nanoserver, which is ~10 times smaller than Windows Servercore.
+  Most binaries added to the image will work out of the box, but some will not due to missing dependencies
+  (**atention**: the image will still build successfully, even if the added binaries will not work).
+  For example, `coredns.exe` requires `netapi32.dll`, which cannot be found on a nanoserver image, but
+  we can copy it from a servercore image (see the agnhost image's `Dockerfile_windows` file as an example).
+  A good rule of thumb is to use 64-bit applications instead of 32-bit as they have fewer dependencies.
+  You can determine what dependencies are missing by running `procmon.exe` on the container's host
+  (make sure that process isolation is used, not Hyper-V isolation).
+  [This](https://stefanscherer.github.io/find-dependencies-in-windows-containers/) is a useful guide on how to use `procmon.exe`.
+
+
 ## Building images
 
 The images are built through `make`. Since some images (`agnhost`) are used as a base for other images,
@@ -160,14 +130,6 @@ registry. That can changed by running this command instead:
 REGISTRY=foo_registry make all-push WHAT=agnhost
 ```
 
-In order to also include Windows Container images into the final manifest lists, the `REMOTE_DOCKER_URL` argument
-in the form `tcp://[host]:[port][path]` (for more details, see [here]([https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option]/))
-will also have to be specified:
-
-```bash
-REMOTE_DOCKER_URL_1909=remote_docker_url_1909 REMOTE_DOCKER_URL_1903=remote_docker_url_1903 REMOTE_DOCKER_URL_1809=remote_docker_url_1809 REGISTRY=foo_registry make all-push WHAT=test-webserver
-```
-
 *NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`)
 require the `agnhost` image to be published in an authenticated repo as well:
 
@@ -215,15 +177,3 @@ After all the above has been done, run the desired tests.
 ```bash
 sudo chmod o+x /etc/docker
 ```
-
-`nc` is being used by some E2E tests, which is why we are including a Linux-like `nc.exe` into the Windows `busybox` image. The image could fail to build during that step with an error that looks like this:
-
-```console
-re-exec error: exit status 1: output: time="..." level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) path=\\\\?\\C:\\ProgramData\\...
-```
-
-The issue is caused by the Windows Defender which is removing the `nc.exe` binary from the filesystem. For more details on this issue, see [here](https://github.com/diegocr/netcat/issues/6). To fix this, you can simply run the following powershell command to temporarily disable Windows Defender:
-
-```powershell
-Set-MpPreference -DisableRealtimeMonitoring $true
-```

test/images/agnhost/Dockerfile_windows

@@ -14,7 +14,23 @@
 
 ARG BASEIMAGE
 ARG REGISTRY
-FROM $REGISTRY/windows-image-builder-helper:1.0 as helper
+ARG OS_VERSION
+
+# We're using a Linux image to unpack the archives, then we're copying them over to Windows.
+FROM --platform=linux/amd64 alpine:3.6 as prep
+
+ADD https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_windows_amd64.tgz /coredns.tgz
+ADD https://iperf.fr/download/windows/iperf-2.0.9-win64.zip /iperf.zip
+
+# we're also creating an empty /uploads folder, which we're copying over to Windows  because
+# we can't RUN commands on a Windows image.
+RUN tar -xzvf /coredns.tgz &&\
+    unzip iperf.zip &&\
+    mv iperf-2.0.9-win64 iperf &&\
+    mkdir /uploads
+
+FROM e2eteam/busybox-helper:1.29.0 as busybox-helper
+FROM --platform=linux/amd64 $REGISTRY/windows-servercore-cache:1.0-linux-amd64-$OS_VERSION as servercore-helper
 FROM $BASEIMAGE
 
 # from dnsutils image
@@ -26,20 +42,13 @@ FROM $BASEIMAGE
 # - curl, nc: used by a lot of e2e tests (inherited from BASEIMAGE)
 # from iperf image
 # install necessary packages: iperf
-COPY --from=helper /dig /dig
-COPY --from=helper /Windows/System32/netapi32.dll /Windows/System32/netapi32.dll
-
-RUN setx /M PATH "C:\dig\;%PATH%
-RUN powershell -Command "\
-    curl.exe -L 'https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_windows_amd64.tgz' -o C:\coredns.tgz;\
-    tar.exe -xzvf C:\coredns.tgz;\
-    Remove-Item C:\coredns.tgz"
+COPY --from=busybox-helper /dig /dig
+COPY --from=servercore-helper /Windows/System32/netapi32.dll /Windows/System32/netapi32.dll
+COPY --from=prep /coredns.exe /coredns.exe
+COPY --from=prep /iperf /iperf
 
-RUN powershell -Command "\
-    curl.exe -L 'https://iperf.fr/download/windows/iperf-2.0.9-win64.zip' -o C:\iperf.zip;\
-    Expand-Archive -Path C:\iperf.zip -DestinationPath C:\ -Force;\
-    Rename-Item C:\iperf-2.0.9-win64 C:\iperf;\
-    Remove-Item C:\iperf.zip"
+# NOTE(claudiub): docker buildx sets the PATH env variable to a Linux-like PATH, which is not desirable.
+ENV PATH="C:\dig\;C:\bin\;C:\curl\;C:\Windows\system32;C:\Windows;C:\Program Files\PowerShell;"
 
 # PORT 80 needed by: test-webserver
 # PORT 8080 needed by: netexec, nettest
@@ -48,7 +57,7 @@ RUN powershell -Command "\
 EXPOSE 80 8080 8081 9376
 
 # from netexec
-RUN mkdir C:\uploads
+COPY --from=prep /uploads /uploads
 
 # from porter
 ADD porter/localhost.crt localhost.crt
@@ -57,12 +66,12 @@ ADD porter/localhost.key localhost.key
 # from mounttest
 ADD mounttest/filePermissions.ps1 filePermissions.ps1
 
-ADD agnhost agnhost
+ADD agnhost /agnhost
 
 # needed for the entrypoint-tester related tests. Some of the entrypoint-tester related tests
 # overrides this image's entrypoint with agnhost-2 binary, and will verify that the correct
 # entrypoint is used by the containers.
-RUN mklink agnhost-2 agnhost
+ADD agnhost /agnhost-2
 
 ENTRYPOINT ["/agnhost"]
 CMD ["pause"]

test/images/busybox/Dockerfile_windows

@@ -13,59 +13,45 @@
 # limitations under the License.
 
 ARG BASEIMAGE
-from $BASEIMAGE as prep
+ARG REGISTRY
 
-ENV CURL_VERSION=7.57.0 \
-    PS_VERSION=6.2.0
-WORKDIR /curl
-ADD https://skanthak.homepage.t-online.de/download/curl-$CURL_VERSION.cab curl.cab
-ADD https://github.com/PowerShell/PowerShell/releases/download/v$PS_VERSION/PowerShell-$PS_VERSION-win-x64.zip /PowerShell/powershell.zip
-ADD https://eternallybored.org/misc/netcat/netcat-win32-1.12.zip /netcat//netcat.zip
+# We're using a Linux image to unpack the archive, then we're copying it over to Windows.
+FROM --platform=linux/amd64 alpine:3.6 as prep
 
-USER ContainerAdministrator
-RUN expand /R curl.cab /F:* . &\
-    cd C:\PowerShell &\
-    tar.exe -xf powershell.zip &\
-    del powershell.zip &\
-    mklink powershell.exe pwsh.exe &\
-    cd C:\netcat &\
-    tar.exe -xf netcat.zip &\
-    del netcat.zip
+RUN mkdir /tmp-dir
 
+FROM e2eteam/busybox-helper:1.29.0 as busybox-helper
+FROM e2eteam/powershell-helper:6.2.7 as ps-helper
 FROM $BASEIMAGE
 
-COPY --from=prep /curl/AMD64 /curl/CURL.LIC /curl/
-COPY --from=prep ["/PowerShell", "/Program Files/PowerShell"]
-COPY --from=prep /netcat/nc64.exe /bin/nc.exe
-ADD https://github.com/kubernetes-sigs/windows-testing/raw/master/images/busybox/busybox.exe /bin/busybox.exe
+COPY --from=prep /tmp-dir /tmp
+COPY --from=busybox-helper /bin /bin
+
+# NOTE(claudiub): Unfortunately, docker buildx has some issues copying over Windows symlinks.
+# "Files/" is always prepended to the symlink target. The symlinks themselves are relative paths,
+# so, in order to make use of them, we can simply add a busybox binary to C:\bin\Files\busybox.exe.
+COPY --from=busybox-helper /bin/busybox.exe /bin/Files/
+COPY --from=busybox-helper /curl /curl
+COPY --from=busybox-helper /netcat/nc64.exe /bin/nc.exe
+
+# include powershell and its Module analysis cache.
+COPY --from=ps-helper ["/PowerShell", "/Program Files/PowerShell"]
+
+# NOTE(claudiub): For the same reason mentioned above, we have to copy pwsh.exe to Files/pwsh.exe
+COPY --from=ps-helper ["/PowerShell/pwsh.exe", "/Program Files/PowerShell/Files/"]
+COPY --from=ps-helper /Users/ContainerAdministrator/AppData/Local/Microsoft/Windows/PowerShell/docker/ModuleAnalysisCache /Users/ContainerAdministrator/AppData/Local/Microsoft/Windows/PowerShell/docker/ModuleAnalysisCache
+
 ADD hostname /bin/hostname.exe
 
 USER ContainerAdministrator
-RUN FOR /f "tokens=*" %i IN ('C:\bin\busybox --list') DO mklink C:\bin\%i.exe C:\bin\busybox.exe
-# Set the path
-RUN setx /M PATH "C:\bin;C:\curl\;%PATH%;C:\Program Files\PowerShell;" &\
-    mkdir C:\tmp
 
-# Copy PowerShell Core from the installer container
-ENV ProgramFiles="C:\Program Files" \
+# NOTE(claudiub): docker buildx sets the PATH env variable to a Linux-like PATH, which is not desirable.
+ENV PATH="C:\bin;C:\curl;C:\Windows\System32;C:\Windows;C:\Program Files\PowerShell;" \
+    ProgramFiles="C:\Program Files" \
     # set a fixed location for the Module analysis cache
     LOCALAPPDATA="C:\Users\ContainerAdministrator\AppData\Local" \
-    PSModuleAnalysisCachePath="$LOCALAPPDATA\Microsoft\Windows\PowerShell\docker\ModuleAnalysisCache" \
+    PSModuleAnalysisCachePath="C:\Users\ContainerAdministrator\AppData\Local\Microsoft\Windows\PowerShell\docker\ModuleAnalysisCache" \
     # Persist %PSCORE% ENV variable for user convenience
-    PSCORE="$ProgramFiles\PowerShell\pwsh.exe"
-
-# intialize powershell module cache
-RUN powershell \
-        -NoLogo \
-        -NoProfile \
-        -Command " \
-          $stopTime = (get-date).AddMinutes(15); \
-          $ErrorActionPreference = 'Stop' ; \
-          $ProgressPreference = 'SilentlyContinue' ; \
-          while(!(Test-Path -Path $env:PSModuleAnalysisCachePath)) {  \
-            Write-Host "'Waiting for $env:PSModuleAnalysisCachePath'" ; \
-            if((get-date) -gt $stopTime) { throw 'timout expired'} \
-            Start-Sleep -Seconds 6 ; \
-          }"
+    PSCORE="C:\Program Files\PowerShell\pwsh.exe"
 
 ENTRYPOINT ["cmd.exe", "/s", "/c"]

test/images/cloudbuild.yaml

@@ -9,41 +9,22 @@ options:
   substitution_option: ALLOW_LOOSE
   machineType: 'N1_HIGHCPU_8'
 steps:
-  - name: gcr.io/cloud-builders/gcloud
+  - name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200422-b25d964'
     entrypoint: 'bash'
-    # NOTE(claudiub): We need to get the ca.pem, cert.pem, key.pem files and put create the
-    # /certs/.docker-1809/, /certs/.docker-1903/, /certs/.docker-1909/ folders, which will contain the files.
-    args:
-    - -c
-    - |
-      mkdir -p .docker-windows
-      gcloud secrets versions access latest --project=k8s-infra-prow-build-trusted --secret=windows-remote-docker_ca-pem > .docker-windows/ca.pem
-      gcloud secrets versions access latest --project=k8s-infra-prow-build-trusted --secret=windows-remote-docker_cert-pem > .docker-windows/cert.pem
-      gcloud secrets versions access latest --project=k8s-infra-prow-build-trusted --secret=windows-remote-docker_key-pem > .docker-windows/key.pem
-      cp -r .docker-windows /certs/.docker-1809
-      cp -r .docker-windows /certs/.docker-1903
-      cp -r .docker-windows /certs/.docker-1909
-    volumes:
-    - name: 'certs'
-      path: '/certs'
-  - name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4'
-    entrypoint: make
     dir: ./test/images/
     env:
     - DOCKER_CLI_EXPERIMENTAL=enabled
     - TAG=$_GIT_TAG
     - BASE_REF=$_PULL_BASE_REF
     - WHAT=$_WHAT
     - REGISTRY=gcr.io/k8s-staging-e2e-test-images
-    - DOCKER_CERT_BASE_PATH=/certs
-    - REMOTE_DOCKER_URL_1809=tcp://img-promoter-1809.eastus.cloudapp.azure.com:2376
-    - REMOTE_DOCKER_URL_1903=tcp://img-promoter-1903.eastus.cloudapp.azure.com:2376
-    - REMOTE_DOCKER_URL_1909=tcp://img-promoter-1909.eastus.cloudapp.azure.com:2376
     args:
-    - all-build-and-push
-    volumes:
-    - name: 'certs'
-      path: '/certs'
+    - '-c'
+    - |
+      docker run --rm --privileged multiarch/qemu-user-static --reset -p yes \
+      && docker buildx create --name img-builder --use \
+      && docker buildx inspect --bootstrap \
+      && make all-build-and-push
 substitutions:
   # _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and
   # can be used as a substitution