New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test images: Adds Windows Container images support (part 1) #76838
Changes from all commits
22a5fbc
efcdb92
25b7a13
a4ceb14
296464d
3cdb7a8
fe15a46
b71fbdb
91dc590
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,6 +18,75 @@ is recommended in order to avoid certain issues. | |
The node must be able to push the images to the desired container registry, make sure you are | ||
authenticated with the registry you're pushing to. | ||
|
||
Windows Container images are not built by default, since they cannot be built on Linux. For | ||
that, a Windows node with Docker installed and configured for remote management is required. | ||
|
||
|
||
### Windows node(s) setup | ||
|
||
In order to build the Windows container images, a node with Windows 10 or Windows Server 2019 | ||
with the latest updates installed is required. The node will have to have Docker installed, | ||
preferably version 18.06.0 or newer. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you be more explicit about why this version is preferred? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hm, there's no particular reason why it has to be 18.06.0 or newer on the Windows nodes. 18.06.0 or newer was chosen for consistency with the requirements on the Linux node (see L11). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. most build things are 19.03 by now |
||
|
||
Keep in mind that the Windows node might not be able to build container images for newer OS versions | ||
than itself (even with `--isolation=hyperv`), so keeping the node up to date and / or upgrading it | ||
to the latest Windows Server edition is ideal. | ||
|
||
Windows test images must be built for Windows Server 2019 (1809) and Windows Server 1903, thus, | ||
if the node does not have Hyper-V enabled, or it is not supported, multiple Windows nodes are required, | ||
one per OS version. | ||
|
||
Additionally, remote management must be configured for the node's Docker daemon. Exposing the | ||
Docker daemon without requiring any authentication is not recommended, and thus, it must be | ||
configured with TLS to ensure that only authorised people can interact with it. For this, the | ||
following `powershell` script can be executed: | ||
|
||
```powershell | ||
mkdir .docker | ||
docker run --isolation=hyperv --user=ContainerAdministrator --rm ` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is --isolation=hyperv required for it to work? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That image is for WS 1809, AFAIK. So, if the Windows node has a different version, yes. |
||
-e SERVER_NAME=$(hostname) ` | ||
-e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP ` | ||
-v "c:\programdata\docker:c:\programdata\docker" ` | ||
-v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5 | ||
# restart the Docker daemon. | ||
Restart-Service docker | ||
``` | ||
|
||
For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/). | ||
|
||
A firewall rule to allow connections to the Docker daemon is necessary: | ||
|
||
```powershell | ||
New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376 | ||
``` | ||
|
||
If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node. | ||
For example, in Azure, this is done by running the following command: | ||
|
||
```console | ||
az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376 | ||
``` | ||
|
||
The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker` | ||
will have to copied to the `~/.docker-${os_version)/` on the Linux build node, where `${os_version}` | ||
is `1809` or `1903`. | ||
|
||
```powershell | ||
scp.exe -r $env:USERPROFILE\.docker ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker-$os_version | ||
``` | ||
|
||
After all this, the Linux build node should be able to connect to the Windows build node: | ||
|
||
```bash | ||
docker --tlsverify --tlscacert ~/.docker-${os_version}/ca.pem --tlscert ~/.docker-${os_version}/cert.pem --tlskey ~/.docker-${os_version}/key.pem -H "$REMOTE_DOCKER_URL" version | ||
``` | ||
|
||
For more information and troubleshooting about enabling Docker remote management, see | ||
[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost) | ||
|
||
Finally, the node must be able to push the images to the desired container registry, make sure you are | ||
authenticated with the registry you're pushing to. | ||
|
||
|
||
## Making changes to images | ||
|
||
|
@@ -63,6 +132,9 @@ For this, you will need the image manifest list's digest, which can be obtained | |
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest' | ||
``` | ||
|
||
The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for | ||
other images, it is recommended to build them first, if needed. | ||
|
||
|
||
## Building images | ||
|
||
|
@@ -88,6 +160,14 @@ registry. That can changed by running this command instead: | |
REGISTRY=foo_registry make all-push WHAT=agnhost | ||
``` | ||
|
||
In order to also include Windows Container images into the final manifest lists, the `REMOTE_DOCKER_URL` argument | ||
in the form `tcp://[host]:[port][path]` (for more details, see [here]([https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option]/)) | ||
will also have to be specified: | ||
|
||
```bash | ||
REMOTE_DOCKER_URL_1909=remote_docker_url_1909 REMOTE_DOCKER_URL_1903=remote_docker_url_1903 REMOTE_DOCKER_URL_1809=remote_docker_url_1809 REGISTRY=foo_registry make all-push WHAT=test-webserver | ||
``` | ||
claudiubelu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
*NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`) | ||
require the `agnhost` image to be published in an authenticated repo as well: | ||
|
||
|
@@ -135,3 +215,15 @@ After all the above has been done, run the desired tests. | |
```bash | ||
sudo chmod o+x /etc/docker | ||
``` | ||
|
||
`nc` is being used by some E2E tests, which is why we are including a Linux-like `nc.exe` into the Windows `busybox` image. The image could fail to build during that step with an error that looks like this: | ||
|
||
```console | ||
re-exec error: exit status 1: output: time="..." level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) path=\\\\?\\C:\\ProgramData\\... | ||
``` | ||
|
||
The issue is caused by the Windows Defender which is removing the `nc.exe` binary from the filesystem. For more details on this issue, see [here](https://github.com/diegocr/netcat/issues/6). To fix this, you can simply run the following powershell command to temporarily disable Windows Defender: | ||
|
||
```powershell | ||
Set-MpPreference -DisableRealtimeMonitoring $true | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,8 @@ | ||
amd64=alpine:3.6 | ||
arm=arm32v6/alpine:3.6 | ||
arm64=arm64v8/alpine:3.6 | ||
ppc64le=ppc64le/alpine:3.6 | ||
s390x=s390x/alpine:3.6 | ||
linux/amd64=alpine:3.6 | ||
linux/arm=arm32v6/alpine:3.6 | ||
linux/arm64=arm64v8/alpine:3.6 | ||
linux/ppc64le=ppc64le/alpine:3.6 | ||
linux/s390x=s390x/alpine:3.6 | ||
windows/amd64/1809=REGISTRY/busybox:1.29-windows-amd64-1809 | ||
windows/amd64/1903=REGISTRY/busybox:1.29-windows-amd64-1903 | ||
windows/amd64/1909=REGISTRY/busybox:1.29-windows-amd64-1909 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
# Copyright 2020 The Kubernetes Authors. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
ARG BASEIMAGE | ||
FROM $BASEIMAGE | ||
|
||
# from dnsutils image | ||
# install necessary packages: | ||
# - bind-tools: contains dig, which can used in DNS tests. | ||
# - CoreDNS: used in some DNS tests. | ||
# from hostexec image | ||
# installed necessary packages: | ||
# - curl, nc: used by a lot of e2e tests (inherited from BASEIMAGE) | ||
# from iperf image | ||
# install necessary packages: iperf | ||
ENV chocolateyUseWindowsCompression false | ||
RUN powershell -Command "\ | ||
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1')); \ | ||
claudiubelu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
choco feature disable --name showDownloadProgress; \ | ||
choco install bind-toolsonly --version 9.10.3 -y | ||
RUN powershell -Command "\ | ||
wget -uri 'https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_windows_amd64.tgz' -OutFile C:\coredns.tgz;\ | ||
tar -xzvf C:\coredns.tgz;\ | ||
Remove-Item C:\coredns.tgz" | ||
|
||
RUN powershell -Command "\ | ||
wget -uri 'https://iperf.fr/download/windows/iperf-2.0.9-win64.zip' -OutFile C:\iperf.zip;\ | ||
Expand-Archive -Path C:\iperf.zip -DestinationPath C:\ -Force;\ | ||
Rename-Item C:\iperf-2.0.9-win64 C:\iperf;\ | ||
Remove-Item C:\iperf.zip" | ||
|
||
# PORT 80 needed by: test-webserver | ||
# PORT 8080 needed by: netexec, nettest | ||
# PORT 8081 needed by: netexec | ||
# PORT 9376 needed by: serve-hostname | ||
EXPOSE 80 8080 8081 9376 | ||
|
||
# from netexec | ||
RUN mkdir C:\uploads | ||
|
||
# from porter | ||
ADD porter/localhost.crt localhost.crt | ||
ADD porter/localhost.key localhost.key | ||
|
||
ADD agnhost agnhost | ||
|
||
# needed for the entrypoint-tester related tests. Some of the entrypoint-tester related tests | ||
# overrides this image's entrypoint with agnhost-2 binary, and will verify that the correct | ||
# entrypoint is used by the containers. | ||
RUN mklink agnhost-2 agnhost | ||
|
||
ENTRYPOINT ["/agnhost"] | ||
CMD ["pause"] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
2.10 | ||
2.11 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
amd64=alpine:3.8 | ||
arm=arm32v6/alpine:3.8 | ||
arm64=arm64v8/alpine:3.8 | ||
ppc64le=ppc64le/alpine:3.8 | ||
linux/amd64=alpine:3.8 | ||
linux/arm=arm32v6/alpine:3.8 | ||
claudiubelu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
linux/arm64=arm64v8/alpine:3.8 | ||
linux/ppc64le=ppc64le/alpine:3.8 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
1.0 | ||
1.1 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
windows/amd64/1809=mcr.microsoft.com/windows/servercore:ltsc2019 | ||
windows/amd64/1903=mcr.microsoft.com/windows/servercore:1903 | ||
windows/amd64/1909=mcr.microsoft.com/windows/servercore:1909 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
package(default_visibility = ["//visibility:public"]) | ||
|
||
load( | ||
"@io_bazel_rules_go//go:def.bzl", | ||
"go_binary", | ||
"go_library", | ||
) | ||
|
||
go_binary( | ||
name = "hostname", | ||
embed = [":go_default_library"], | ||
) | ||
|
||
go_library( | ||
name = "go_default_library", | ||
srcs = ["hostname.go"], | ||
importpath = "k8s.io/kubernetes/test/images/busybox", | ||
) | ||
|
||
filegroup( | ||
name = "package-srcs", | ||
srcs = glob(["**"]), | ||
tags = ["automanaged"], | ||
visibility = ["//visibility:private"], | ||
) | ||
|
||
filegroup( | ||
name = "all-srcs", | ||
srcs = [":package-srcs"], | ||
tags = ["automanaged"], | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Line 14: s/In order to build the docker test images, a Linux node is required./In order to build the docker test Linux container images, a Linux node is required./
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unrelated to the PR, and it's still correct: you do need a Linux node to build all images. With this PR, you'll be able to also build Windows images remotely, but you do it from the Linux node.