Starting section on CRI multiarch/version support #1254

PatrickLang · 2019-09-23T19:11:08Z

One of the details we need to work through is how to disambiguate the OS, architecture and version used when pulling an image and creating a sandbox. Today this is done with a label sandbox-platform during pull, and with annotations when creating a sandbox+container. Instead of having this in weakly typed labels/annotations, we should make it a strongly typed field in CRI.

@feiskyer and @Random-Liu - can you help by reviewing this?

@jterry75 should be able to help clarify some of the Windows specifics as well from the containerd side

/hold
/sig windows
/sig node

k8s-ci-robot · 2019-09-23T19:11:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: PatrickLang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-windows/OWNERS~~ [PatrickLang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

feiskyer · 2019-09-24T14:43:58Z

keps/sig-windows/20190424-windows-cri-containerd.md

+}
+```
+
+Today, PodSandboxConfig has an annotation that can be used as a workaround, but it's stated that "Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead.".


could you add a part describing how could those new configs be used from Pod spec?

jterry75 · 2019-09-24T17:01:00Z

keps/sig-windows/20190424-windows-cri-containerd.md

+The proposed additions are:
+
+```
+string os = 9;


@PatrickLang - If we do this we likely don't need to have separate RuntimeHandlers for runhcs-wcow-hypervisor and runhcs-wcow-hypervisor-1903 above. This makes it explicit at activation time rather an implicit from the RuntimeHandler (which I like better anyways)

PatrickLang · 2019-09-30T20:42:15Z

cc @yliaog

I'm working on more updates to this PR with examples and tradeoffs of different ways to disambiguate os/arch/version

Adding 3 options for proceeding with RuntimeClass

PatrickLang · 2019-10-01T17:20:16Z

/assign @tallclair

Can you help review from RuntimeClass standpoint?

tallclair

I don't have any objects to the RuntimeClass approaches proposed here. IIUC, neither require any changes to RuntimeClass to support those 2 options?

tallclair · 2019-10-01T18:36:17Z

keps/sig-windows/20190424-windows-cri-containerd.md

+
+
+


nit: empty lines

tallclair · 2019-10-01T18:47:00Z

keps/sig-windows/20190424-windows-cri-containerd.md

+metadata:
+  name: mypod
+spec:
+  runtimeClassName: WindowsServer1809


An advantage of this approach is that by making the runtimeClassName explicit, the user must update the deployment which:

Forces them to ensure they have an image built for the new versions

Makes it clear what version they're expecting, and which version a given pod is running

Automatically triggers a rolling update to move to the new version, with the supported workflows

This disadvantage is it places more work on the dev teams to keep their deployments up-to-date, and also has a risk of a combinatorial explosion of runtime classes.

Interesting point. It also enables us to have different ideas around things like RuntimeOverhead and such per version.

@tallclair - Given your feedback would you recommend that we do not have a handler that is "versionless"? IE: no runhcs-wcow-process but only runhcs-wcow-process-1809 etc

PatrickLang · 2019-10-03T21:12:14Z

keps/sig-windows/20190424-windows-cri-containerd.md

+kind: RuntimeClass
+metadata:
+  name: windows-1809-amd64
+handler: 'windows-hyperv-1809'  # only handler accepted when using dockershim


Review feedback - this doesn't get passed to pull, so it doesn't solve the problem of selecting the right image at pull time if its multiarch

that comment is wrong - copy/paste error. handler: windows-hyperv-1809 is an example using containerd

PatrickLang · 2019-10-03T21:13:51Z

keps/sig-windows/20190424-windows-cri-containerd.md

+
+###### Details of Multi-arch images
+
+Here's one example of a container image that has a multi-arch manifest with entries for Linux & Windows, multiple CPU architectures, and multiple Windows os versions.


review feedback - put a bold thing that says multi-arch is actually multi-version too

PatrickLang · 2019-10-03T21:15:34Z

keps/sig-windows/20190424-windows-cri-containerd.md

+spec:
+  os: windows
+  osversion: 1809
+  architecture: amd64


open issue - would also need variant. per OCI distribution, there can be multiple architectures with different variants. see arm/v7, arm/v8

PatrickLang · 2019-10-03T21:15:50Z

keps/sig-windows/20190424-windows-cri-containerd.md

+digest                                                                  platform
+------                                                                  --------
+sha256:f04192a946a4473d0001ed9c9422a9e3c9c659de8498778d29bfe6c98672ad9f @{architecture=amd64; os=linux}
+sha256:b7da46cdbc9a0c0ed154cabbb0591dea596e67d62d84c2a3ef34aaefe98f186d @{architecture=arm; os=linux; variant=v7}


find a arm/v8 sample?

PatrickLang · 2019-10-03T21:17:32Z

keps/sig-windows/20190424-windows-cri-containerd.md

+handler: 'docker'  # only handler accepted when using dockershim
+scheduling:
+  nodeSelector:
+    kubernetes.io/os: 'windows'


todo: open mini kep for os.version. that's needed regardless of this. propose setting it from kubelet

If kubelet needs to query the Windows OS version through something like syscall.GetVersion, we will also need to make sure there is a manifest generated: golang/go#17835 (comment)

PatrickLang · 2019-10-03T21:21:51Z

keps/sig-windows/20190424-windows-cri-containerd.md

+        [plugins.cri.containerd.runtimes.windows-hyperv-1809]
+          runtime_type = "io.containerd.runhcs.v1"
+          [plugins.cri.containerd.runtimes.runhcs-wcow-hypervisor.options]
+            SandboxImage = "{{WINDOWSSANDBOXIMAGE}}"


take out the template - change to a multiarch image for this example

PatrickLang · 2019-10-03T21:29:47Z

keps/sig-windows/20190424-windows-cri-containerd.md

+  nodeSelector:
+    kubernetes.io/os: 'windows'
+    kubernetes.io/arch: 'amd64'
+    beta.kubernetes.io/osversion: '1903'


question: can this match a set instead of a specific value? How to handle if a given Pod could run on multiple OS/os.version/arch instead of a single one?

PatrickLang · 2019-10-03T21:34:27Z

keps/sig-windows/20190424-windows-cri-containerd.md

+    kubernetes.io/os: 'windows'
+    kubernetes.io/arch: 'amd64'
+    beta.kubernetes.io/osversion: '1903'
+    capability.microsoft.com/isolation: 'hyperv'


add comment - that's a proposed new label not an existing one

clarify - checking for the hyperv capability belongs in a label because not all cpu types / VM sizes may support Hyper-V

this may need to be a more generic name like supportsVirtualizationInstructions

PatrickLang · 2019-10-16T19:32:21Z

I'm closing this in favor of #1302 which has more details around the use cases and is being reviewed with the appropriate SIGs.

Starting section on CRI multiarch/version support

6f56ace

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 23, 2019

k8s-ci-robot requested review from mattfarina and michmike September 23, 2019 19:11

feiskyer reviewed Sep 24, 2019

View reviewed changes

jterry75 reviewed Sep 24, 2019

View reviewed changes

PatrickLang added 2 commits September 30, 2019 16:33

Add ContainerD tracking issue & status

6cd1b9f

Adding 3 options for proceeding with RuntimeClass

Fix headings

ee135f9

k8s-ci-robot assigned tallclair Oct 1, 2019

tallclair reviewed Oct 1, 2019

View reviewed changes

Fixing some whitespace, adding more details in examples

45a46d4

PatrickLang commented Oct 3, 2019

View reviewed changes

PatrickLang closed this Oct 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starting section on CRI multiarch/version support #1254

Starting section on CRI multiarch/version support #1254

PatrickLang commented Sep 23, 2019 •

edited

Loading

k8s-ci-robot commented Sep 23, 2019

feiskyer Sep 24, 2019 •

edited

Loading

PatrickLang Oct 1, 2019

jterry75 Sep 24, 2019

PatrickLang commented Sep 30, 2019

PatrickLang commented Oct 1, 2019

tallclair left a comment

tallclair Oct 1, 2019

tallclair Oct 1, 2019

jterry75 Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019 •

edited

Loading

ddebroy Oct 9, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019 •

edited

Loading

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang Oct 3, 2019

PatrickLang commented Oct 16, 2019


		###### Details of Multi-arch images

		Here's one example of a container image that has a multi-arch manifest with entries for Linux & Windows, multiple CPU architectures, and multiple Windows os versions.

Starting section on CRI multiarch/version support #1254

Starting section on CRI multiarch/version support #1254

Conversation

PatrickLang commented Sep 23, 2019 • edited Loading

k8s-ci-robot commented Sep 23, 2019

feiskyer Sep 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang commented Sep 30, 2019

PatrickLang commented Oct 1, 2019

tallclair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang Oct 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang Oct 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang commented Oct 16, 2019

PatrickLang commented Sep 23, 2019 •

edited

Loading

feiskyer Sep 24, 2019 •

edited

Loading

PatrickLang Oct 3, 2019 •

edited

Loading

PatrickLang Oct 3, 2019 •

edited

Loading