Build volume test images on Windows

  * Adds Windows dockerfile for volume-ownership image
  * Build volume-copy-up on Windows
  * Adds a helper tool that fetches the owner username and SID of
a file or folder
  * Adds README
  * Remove 2004 from Windows versions
  * Add ltsc2022 to Windows versions

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>

integration/images/README.md

@@ -0,0 +1,127 @@
+# Test image overview
+
+Test images for Linux can be built as usual using buildx.
+
+While it is possible to build Windows docker images on Linux (if we avoid the ```RUN``` or ```WORKDIR``` options), the ```volume-ownership``` and ```volume-copy-up``` images need to be built on Windows for the tests to be relevant. The reason for this is that when building images on Linux, Windows specific security info (DACL and ownership) does not get attached to the test files and folders inside the image. The ```TestVolumeCopyUp``` and ```TestVolumeOwnership``` tests will not be relevant, as owners of the files will always be ```ContainerAdministrator```.
+
+Building images on Windows nodes also allows us to potentially add new users inside the images or enable new testing scenarios that require different services or applications to run inside the container.
+
+This document describes the needed bits to build the Windows container images on a remote Windows node.
+
+## Setting up the Windows build node
+
+We can build images for all relevant Windows versions on a single Windows node as long as that Windows node is a version greater or equal to the image versions we're trying to build. For example, on a Windows Server 2022 node, we can build images for 1809, 2004, 20H2 and ltsc2022, while if we were running on Windows server 2019 machine, we would only be able to generate images for 1809. To build images for different versions of Windows, we need to enable the ```Hyper-V``` role, and use ```--isolation=hyperv``` as an argument to docker build.
+
+Note, this will also work if nested hyperv is enabled. This means that the images can be built on Azure (nested Hyper-V is enabled by default), or on any modern linux machine using KVM and libvirt.
+Note, at the time of this writing, the recommended version to build on is Windows Server 2022 (ltsc2022).
+
+
+### Enabling nested VMX on Libvirt
+
+To enable nested Hyper-V on libvirt, simply install Windows Server 2022 as usual, then shutdown the guest and edit it's config:
+
+```bash
+# replace win2k22 with the name of your Windows VM
+virsh edit win2k22
+```
+
+and add/edit the CPU section to look like this:
+
+```xml
+<cpu mode='custom' match='exact' check='partial'>
+    <model fallback='allow'>Broadwell</model>
+    <feature policy='require' name='vmx'/>
+</cpu>
+```
+
+Hyper-V should now work inside your KVM machine. It's not terribly fast, but it should suffice for building images.
+
+### Enable necessary roles
+
+Install the needed roles and tools:
+
+```powershell
+# Enable Hyper-V and management tools
+Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V,Microsoft-Hyper-V-Management-Clients,Microsoft-Hyper-V-Management-PowerShell -All -NoRestart
+
+# Enable SSH (this can be skipped if you don't need it)
+Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
+
+# Install Docker
+Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force -Confirm:$false
+Install-Module -Name DockerMsftProvider -Repository PSGallery -Force -Confirm:$false
+Install-Package -Name docker -ProviderName DockerMsftProvider -Force -Confirm:$false
+```
+
+At this point we can reboot for the changes to take effect:
+
+```powershell
+Restart-Computer -Force
+```
+
+### Configure needed services
+
+Start sshd and enable it to run on startup:
+
+```powershell
+Start-Service sshd
+Set-Service -Name sshd -StartupType 'Automatic'
+```
+
+Open Firewall port for ssh:
+
+```powershell
+New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
+```
+
+These following steps are taken from the [k8s windows image builder helper page](https://github.com/kubernetes/kubernetes/blob/master/test/images/windows/README.md).
+
+
+Enable TLS authentication for docker and enable remote access:
+
+```powershell
+# Replace YOUR_SERVER_IP_GOES_HERE with the IP addresses you'll use to access
+# this node. This will be the private IP and VIP/Floating IP of the server.
+docker run --isolation=hyperv --user=ContainerAdministrator --rm `
+   -e SERVER_NAME=$(hostname) `
+   -e IP_ADDRESSES=127.0.0.1,YOUR_SERVER_IP_GOES_HERE `
+   -v "c:\programdata\docker:c:\programdata\docker" `
+   -v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
+```
+
+Restart Docker:
+
+```powershell
+Stop-Service docker
+Start-Service docker
+```
+
+After this, the files (```ca.pem```, ```cert.pem``` and ```key.pem```) needed to authenticate to docker will be present in ```$env:USERPROFILE\.docker``` on the Windows machine. You will need to copy those files to your linux machine in ```$HOME/.docker```. They are needed in order to authenticate against the Windows docker daemon during our image build process.
+
+Open Firewall port for docker:
+
+```powershell
+New-NetFirewallRule -Name 'Docker-TLS-In-TCP' -DisplayName 'Docker (TLS)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 2376
+```
+
+Note, if you're running in a cloud, make sure you also open the port in your NSG/Security group.
+
+## Building the images
+
+With the above mentioned files copied to ```$HOME/.docker``` we can now start building the images:
+
+```bash
+git clone https://github.com/containerd/containerd
+cd containerd/integration/images/volume-copy-up
+
+make setup-buildx
+make configure-docker
+# 192.168.122.107 corresponds to the IP address of your windows build node.
+# This builds the images and pushes them to the registry specified by PROJ
+# The Windows images will be built on the Windows node and pushed from there.
+# You will need to make sure that docker is configured and able to push to the
+# project you want to push to.
+make build-registry PROJ=docker.example.com REMOTE_DOCKER_URL=192.168.122.107:2376
+# Create a manifest and update it with all supported operating systems and architectures.
+make push-manifest PROJ=docker.samfira.com REMOTE_DOCKER_URL=192.168.122.107:2376
+```

integration/images/volume-copy-up/Dockerfile_windows

@@ -1,42 +1,34 @@
-#   Copyright The containerd Authors.
-
-#   Licensed under the Apache License, Version 2.0 (the "License");
-#   you may not use this file except in compliance with the License.
-#   You may obtain a copy of the License at
-
-#       http://www.apache.org/licenses/LICENSE-2.0
-
-#   Unless required by applicable law or agreed to in writing, software
-#   distributed under the License is distributed on an "AS IS" BASIS,
-#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#   See the License for the specific language governing permissions and
-#   limitations under the License.
+# Copyright The containerd Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 ARG BASE
+FROM $BASE
 
-FROM --platform=linux/amd64 busybox as prep
+ADD https://github.com/kubernetes-sigs/windows-testing/raw/3fea3d48ea8337b2aaca755c1d719e34b45f46b9/images/busybox/busybox.exe /bin/busybox.exe
 
-# Similar to: https://github.com/kubernetes/kubernetes/blob/7ad7c0757ac7fa37dfae9c7cdc628cf04e35e5cb/test/images/busybox/Dockerfile_windows
+ENV BUSYBOX_EXES="[ [[ ar arch ash awk base64 basename bash bunzip2 bzcat bzip2 cal cat chmod cksum clear cmp comm cp cpio cut date dc dd df diff dirname dos2unix dpkg-deb du echo ed egrep env expand e xpr factor false fgrep find fold fsync ftpget ftpput getopt grep groups gunzip gzip hd head hexdump id ipcalc kill killall less link ln logname ls lzcat lzma lzop lzopcat man md5sum mkdir mktemp mv nl od paste patch pgrep pidof pipe_progress pkill printenv printf ps pwd rev rm rmdir rpm rpm2cpio sed seq sh sha1sum sha256sum sha3sum sha512sum shred shuf sleep sort split ssl_client stat strings sum ta c tail tar tee test timeout touch tr true truncate ttysize uname uncompress unexpand uniq unix2dos unlink unlzma unlzop unxz unzip usleep uudecode uuencode vi watch wc wget which whoami whois xargs xxd xz xzcat yes zcat"
 
-# Available busybox functions retrieved by running busybox.exe --list
-ENV BUSYBOX_EXES="[ [[ ar arch ash awk base64 basename bash bunzip2 bzcat bzip2 cal cat chmod cksum clear cmp comm cp cpio cut date dc dd df diff dirname dos2unix dpkg-deb du echo ed egrep env expand expr factor false fgrep find fold fsync ftpget ftpput getopt grep groups gunzip gzip hd head hexdump id ipcalc kill killall less link ln logname ls lzcat lzma lzop lzopcat man md5sum mkdir mktemp mv nl od paste patch pgrep pidof pipe_progress pkill printenv printf ps pwd rev rm rmdir rpm rpm2cpio sed seq sh sha1sum sha256sum sha3sum sha512sum shred shuf sleep sort split ssl_client stat strings sum tac tail tar tee test timeout touch tr true truncate ttysize uname uncompress unexpand uniq unix2dos unlink unlzma unlzop unxz unzip usleep uudecode uuencode vi watch wc wget which whoami whois xargs xxd xz xzcat yes zcat"
+USER ContainerAdministrator
+WORKDIR C:/bin
 
-ADD https://github.com/kubernetes-sigs/windows-testing/raw/master/images/busybox/busybox.exe /busybox-dir/busybox.exe
+RUN cmd.exe /c "@echo off && FOR %i in (%BUSYBOX_EXES%) do (mklink %i.exe busybox.exe)"
 
-# NOTE(claudiub): We're creating symlinks for each of the busybox binaries and after that we're copying
-# # them over to Windows. Unfortunately, docker buildx has some issues copying over Windows symlinks.
-# # "Files/" is always prepended to the symlink target. The symlinks themselves are relative paths,
-# # so, in order to make use of them, we can simply add a busybox binary to Files\busybox.exe.
-RUN cd /busybox-dir/ && \
-    for busybox_binary in $BUSYBOX_EXES; do ln -s busybox.exe $busybox_binary.exe; done && \
-    mkdir Files && \
-    cp busybox.exe Files/busybox.exe
+USER ContainerUser
 
-RUN sh -c "mkdir /test_dir; echo test_content > /test_dir/test_file"
+RUN cmd.exe /c mkdir C:\test_dir
 
-FROM $BASE
+RUN /bin/sh.exe -c "echo test_content > /test_dir/test_file"
 
-COPY --from=prep /busybox-dir /bin
-COPY --from=prep /test_dir /test_dir
 ENV PATH="C:\bin;C:\Windows\System32;C:\Windows;"
-VOLUME "/test_dir"
+VOLUME "C:/test_dir"

integration/images/volume-copy-up/Makefile

@@ -17,33 +17,50 @@ all: build
 PROJ=gcr.io/k8s-cri-containerd
 VERSION=2.1
 IMAGE=$(PROJ)/volume-copy-up:$(VERSION)
+DOCKER_CERT_PATH ?= "$(HOME)/.docker"
+REMOTE_DOCKER_URL ?=
+DOCKER_REMOTE_ARGS ?=
+
+
+ifneq ($(REMOTE_DOCKER_URL),)
+	DOCKER_REMOTE_ARGS = --tlsverify --tlscacert "$(DOCKER_CERT_PATH)/ca.pem" \
+		--tlscert "$(DOCKER_CERT_PATH)/cert.pem" \
+		--tlskey "$(DOCKER_CERT_PATH)/key.pem" \
+		-H "$(REMOTE_DOCKER_URL)"
+endif
 
 # Operating systems supported: linux, windows
 OS ?= linux
 # Architectures supported: amd64, arm64
 ARCH ?= amd64
-# OS Version for the Windows images: 1809, 2004, 20H2
+# OS Version for the Windows images: 1809, 20H2, ltsc2022
 OSVERSION ?= 1809
 
 # The output type could either be docker (local), or registry.
 # If it is registry, it will also allow us to push the Windows images.
 OUTPUT_TYPE ?= docker
 
-ALL_OS = linux windows
+ALL_OS = linux
 ALL_ARCH.linux = amd64 arm64
 ALL_OS_ARCH.linux = $(foreach arch, ${ALL_ARCH.linux}, linux-$(arch))
-ALL_OSVERSIONS.windows := 1809 2004 20H2
+
+ifneq ($(REMOTE_DOCKER_URL),)
+ALL_OS += windows
+ALL_OSVERSIONS.windows := 1809 20H2 ltsc2022
 ALL_OS_ARCH.windows = $(foreach osversion, ${ALL_OSVERSIONS.windows}, windows-amd64-${osversion})
-ALL_OS_ARCH = $(foreach os, $(ALL_OS), ${ALL_OS_ARCH.${os}})
+BASE.windows := mcr.microsoft.com/windows/nanoserver
+endif
 
 BASE.linux.amd64 := busybox
 BASE.linux.arm64 := arm64v8/busybox
 BASE.linux := ${BASE.linux.${ARCH}}
-BASE.windows := mcr.microsoft.com/windows/nanoserver
 BASE := ${BASE.${OS}}
 
+ALL_OS_ARCH = $(foreach os, $(ALL_OS), ${ALL_OS_ARCH.${os}})
+
 configure-docker:
-	gcloud auth configure-docker
+	gcloud auth configure-docker --quiet
+	gcloud auth configure-docker --quiet $(shell echo $(PROJ) | cut -f 1 -d "/") || true
 
 setup-buildx:
 	docker buildx use img-builder || docker buildx create --name img-builder --use
@@ -67,9 +84,10 @@ container: .container-${OS}-$(ARCH)
 		-t $(IMAGE)-${OS}-${ARCH} --build-arg BASE=${BASE} .
 
 .container-windows-$(ARCH):
-	docker buildx build --pull --output=type=${OUTPUT_TYPE} --platform ${OS}/${ARCH} \
+	docker $(DOCKER_REMOTE_ARGS) build --isolation=hyperv --no-cache --pull \
 		-t $(IMAGE)-${OS}-${ARCH}-${OSVERSION} --build-arg BASE=${BASE}:${OSVERSION} \
 		-f Dockerfile_windows .
+	docker $(DOCKER_REMOTE_ARGS) push $(IMAGE)-${OS}-${ARCH}-${OSVERSION}
 
 # For Windows images, we also need to include the "os.version" in the manifest list images,
 # so the Windows node can pull the proper image it needs.
@@ -80,7 +98,8 @@ push-manifest:
 	set -x; \
 	for osversion in ${ALL_OSVERSIONS.windows}; do \
 		full_version=`docker manifest inspect ${BASE.windows}:$${osversion} | grep "os.version" | head -n 1 | awk -F\" '{print $$4}'` || true; \
-		docker manifest annotate --os windows --arch amd64 --os-version $${full_version} ${IMAGE} ${IMAGE}-windows-amd64-$${osversion}; \
+		docker manifest annotate --os windows --arch amd64 --os-version $${full_version} \
+		${IMAGE} ${IMAGE}-windows-amd64-$${osversion}; \
 	done
 	docker manifest push --purge ${IMAGE}
 

integration/images/volume-ownership/Dockerfile_windows

@@ -0,0 +1,36 @@
+# Copyright The containerd Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE
+FROM $BASE
+
+ADD https://github.com/kubernetes-sigs/windows-testing/raw/3fea3d48ea8337b2aaca755c1d719e34b45f46b9/images/busybox/busybox.exe /bin/busybox.exe
+
+ENV BUSYBOX_EXES="[ [[ ar arch ash awk base64 basename bash bunzip2 bzcat bzip2 cal cat chmod cksum clear cmp comm cp cpio cut date dc dd df diff dirname dos2unix dpkg-deb du echo ed egrep env expand e xpr factor false fgrep find fold fsync ftpget ftpput getopt grep groups gunzip gzip hd head hexdump id ipcalc kill killall less link ln logname ls lzcat lzma lzop lzopcat man md5sum mkdir mktemp mv nl od paste patch pgrep pidof pipe_progress pkill printenv printf ps pwd rev rm rmdir rpm rpm2cpio sed seq sh sha1sum sha256sum sha3sum sha512sum shred shuf sleep sort split ssl_client stat strings sum ta c tail tar tee test timeout touch tr true truncate ttysize uname uncompress unexpand uniq unix2dos unlink unlzma unlzop unxz unzip usleep uudecode uuencode vi watch wc wget which whoami whois xargs xxd xz xzcat yes zcat"
+
+USER ContainerAdministrator
+WORKDIR C:/bin
+
+ADD tools/get_owner_windows.exe C:/bin/get_owner.exe
+RUN cmd.exe /c "@echo off && FOR %i in (%BUSYBOX_EXES%) do (mklink %i.exe busybox.exe)"
+
+RUN cmd.exe /c mkdir C:\volumes
+
+USER ContainerUser
+
+RUN mkdir C:\volumes\test_dir
+RUN /bin/sh.exe -c "echo test_content > /volumes/test_dir/test_file"
+
+ENV PATH="C:\bin;C:\Windows\System32;C:\Windows;"
+VOLUME "C:/volumes/test_dir"