chore: Hyperv and upstream Containerd package support #3688

jsturtevant · 2020-08-10T23:30:09Z

Reason for Change:

Containerd 1.4rc1 is now available. Hyperv support has progressed but is still in experimental support.

This pr enables:

using builds from containerd upstream builds
Configuration support for hyperv using RuntimeClass
doc updates

Better support for configuring Containerd will be done in a separate PR after a discussion in issue #3687

Issue Fixed:

Fixes #2827

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Notes:

jsturtevant · 2020-08-10T23:31:50Z

/cc @AbelHu @marosset @ksubrmnn

codecov · 2020-08-11T00:03:37Z

Codecov Report

Merging #3688 into master will increase coverage by 0.00%.
The diff coverage is 84.54%.

@@           Coverage Diff            @@
##           master    #3688    +/-   ##
========================================
  Coverage   73.14%   73.15%            
========================================
  Files         147      147            
  Lines       25165    25301   +136     
========================================
+ Hits        18408    18508   +100     
- Misses       5623     5655    +32     
- Partials     1134     1138     +4

Impacted Files	Coverage Δ
pkg/api/common/const.go	`40.00% <ø> (ø)`
pkg/api/vlabs/types.go	`71.83% <ø> (ø)`
pkg/engine/engine.go	`85.64% <0.00%> (-0.31%)`	⬇️
pkg/engine/templates_generated.go	`53.42% <38.88%> (-0.20%)`	⬇️
pkg/api/vlabs/validate.go	`80.00% <92.50%> (+0.50%)`	⬆️
pkg/api/addons.go	`97.79% <100.00%> (+0.02%)`	⬆️
pkg/api/converterfromapi.go	`94.17% <100.00%> (+0.07%)`	⬆️
pkg/api/convertertoapi.go	`93.68% <100.00%> (+0.07%)`	⬆️
pkg/api/k8s_versions.go	`100.00% <100.00%> (ø)`
pkg/api/types.go	`94.31% <100.00%> (+0.06%)`	⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f321785...3ca765c. Read the comment docs.

docs/topics/features.md

mboersma · 2020-08-11T15:31:49Z

scripts/build-windows-containerd.sh

@@ -31,41 +31,20 @@ cp bin/ctr.exe /output
 #cp bin/containerd.exe /output # missing CRI plugin, so build from containerd/cri
 cd \$GOPATH
 cd src/github.com/containerd
-git clone https://github.com/containerd/cri.git
+git clone https://github.com/jterry75/cri.git


Why are we using someone's fork of ContainerD?

This the fork the Container Platform team maintains for making the updates required for hyperv. They are in active conversation with the Containerd team to integrate these changes into the upstream. This was the same process they had for the initial Containerd Support that is now in the 1.4 release.

Update docs/topics/features.md Co-authored-by: Matt Boersma <Matt.Boersma@microsoft.com>

marosset · 2020-08-11T18:28:34Z

I think we probably want to add a new property in the api model that sets if the default runtime class in the containerd config uses hyper-v isolation or not.
I can't think of a good way to run upstream test passes for both process isolation and hyper-v isolation containers without this.
Thoughts?

parts/k8s/windowscontainerdfunc.ps1

jsturtevant · 2020-08-11T19:13:15Z

Agreed I created issue #3687 with a possible idea. As of right now this will allow running both based on the configuration file that is present in the hyperv package. This is not ideal but wanted to get thoughts on #3687 and get something to unlock the failing tests before modifying the api model to far.

Thinking on it a bit more the model change might be something like:

"windowsProfile": {
        "windowsSku": "Datacenter-Core-2004-with-Containers-smalldisk",
        "imageVersion": "latest",
        "defaultHandler": process/hyperv
          "hypervisorHandlers":[
            "17763",
            "18362",
            "18363",
            "19041"
          ]

This would allow user to specify:

process isolated default and no hyperv handlers,
process isolated default with configurable hypervisor handlers
hyperv default, process isolated, and configurable hypervisor handlers.

These fields would only be valid if using containerd.

marosset · 2020-08-12T16:08:44Z

I think we need to have at least have a switch for the default runtime handler / isolation type for this to get checked in to master.
It looks like in the code the in this PR the switch to use hyper-v vs process isolation is the existence of a config file which will currently always exist.

We can use #3687 to figure out how to handle more customizing the non-default runtime classes defined in the containerd config.

mainred · 2020-08-13T00:23:47Z

examples/windows/kubernetes-hyperv.json

-          }
+        "networkPlugin": "azure",
+        "containerRuntime": "containerd",
+        "windowsContainerdURL": "https://k8swin.blob.core.windows.net/k8s-windows/containerd/containerplat-aks-test-0.0.8.zip"


May I regard this package url as the _official url_for 20H1 so far?

yes this is the latest package

These are the official bits but we will probably need to host them along with our other artifacts for production use.
k8swin.blob.core.windows.net is just hosted in a single zone.

I am going to set up a different url for prod use. Will follow back once I have it

Sounds good.
That should not block this PR IMO

marosset · 2020-08-13T18:53:39Z

/lgtm

acs-bot · 2020-08-13T18:56:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsturtevant, marosset

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [marosset]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* chore: Hyperv and upstream Containerd package support (#3688) * fix: fix issue where installing a different version of containerd vs what was pre-installed on VHD failed silently (#3743) * feat: collect hyperv logs * fix: do not log secret in provisioning script Co-authored-by: Mark Rossetti <marosset@microsoft.com>

* hyperv and upstream support * update docs and scripts for hyperv Co-authored-by: Matt Boersma <Matt.Boersma@microsoft.com>

jsturtevant added 7 commits August 7, 2020 15:27

use containerds config command

d7337a2

hyperv support

783735f

update docs and scripts for hyperv

27ca8fa

updating links and containerd urls

29b47ea

updates to limitations

719d149

fix fomatting

d704084

should use custom kubenet url for hyperv

7a8f2ca

acs-bot added the size/XL label Aug 10, 2020

jsturtevant added the windows label Aug 10, 2020

acs-bot requested review from AbelHu, ksubrmnn and marosset August 10, 2020 23:31

switch branches in correct location

78d6aec

mboersma reviewed Aug 11, 2020

View reviewed changes

currently switching on toml for hyperv (see open issue for more details)

97ae5a5

Update docs/topics/features.md Co-authored-by: Matt Boersma <Matt.Boersma@microsoft.com>

jsturtevant force-pushed the hyperv branch from 64df4f2 to 97ae5a5 Compare August 11, 2020 16:22

marosset reviewed Aug 11, 2020

View reviewed changes

parts/k8s/windowscontainerdfunc.ps1 Outdated Show resolved Hide resolved

jsturtevant mentioned this pull request Aug 11, 2020

Use multi os hyperv aks-engine package kubernetes/test-infra#18791

Merged

customized handlers is building and deploying

40e168b

mainred reviewed Aug 13, 2020

View reviewed changes

jsturtevant added 2 commits August 12, 2020 19:58

add tests

cfbade1

validate no duplicates

3127dd4

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 13, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

ae979cc

jsturtevant added 2 commits August 12, 2020 20:47

copy paste errors

ef6e038

handle process isolated with hypervisors properly

f4c7c85

jsturtevant and others added 2 commits August 13, 2020 11:29

fix formating

e76c5ff

include generated code

3ca765c

acs-bot assigned marosset Aug 13, 2020

acs-bot added the lgtm label Aug 13, 2020

marosset approved these changes Aug 13, 2020

View reviewed changes

acs-bot added the approved label Aug 13, 2020

jsturtevant mentioned this pull request Aug 13, 2020

Hyperv Containerd configuration #3687

Closed

jsturtevant merged commit 046db34 into Azure:master Aug 13, 2020

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 17, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

e13fcf4

jsturtevant mentioned this pull request Aug 18, 2020

docs: update limitations in dualstack docs #3727

Merged

4 tasks

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

570c152

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

c819f08

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

5f31fac

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

533fb15

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support Azure#3688

7ab5dd7

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

aefb1ce

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Aug 20, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

e784b5e

AbelHu added a commit that referenced this pull request Aug 24, 2020

chore: Hyperv and upstream Containerd package support (#3688)

7c44ed0

fmotrifork mentioned this pull request Aug 28, 2020

aks-engine 0.55.0 fishworks/fish-food#911

Merged

jsturtevant mentioned this pull request Sep 3, 2020

chore: add 1.19 job templates and remove agent_preprovision_extension kubernetes-sigs/windows-testing#196

Merged

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Sep 14, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

6673f16

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Sep 27, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

f8a07d1

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Sep 30, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

5085b00

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Sep 30, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

687aab5

AbelHu added a commit to AbelHu/aks-engine that referenced this pull request Oct 20, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

3c099d0

AbelHu added a commit that referenced this pull request Oct 23, 2020

chore: Hyperv and upstream Containerd package support (#3688)

c5b18d2

penggu pushed a commit to penggu/aks-engine that referenced this pull request Oct 28, 2020

chore: Hyperv and upstream Containerd package support (Azure#3688)

d66b096

* hyperv and upstream support * update docs and scripts for hyperv Co-authored-by: Matt Boersma <Matt.Boersma@microsoft.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Hyperv and upstream Containerd package support #3688

chore: Hyperv and upstream Containerd package support #3688

jsturtevant commented Aug 10, 2020 •

edited

Loading

jsturtevant commented Aug 10, 2020

codecov bot commented Aug 11, 2020 •

edited

Loading

mboersma Aug 11, 2020

jsturtevant Aug 11, 2020

marosset commented Aug 11, 2020

jsturtevant commented Aug 11, 2020 •

edited

Loading

marosset commented Aug 12, 2020

mainred Aug 13, 2020

jsturtevant Aug 13, 2020

marosset Aug 13, 2020

jsturtevant Aug 13, 2020

marosset Aug 13, 2020

marosset commented Aug 13, 2020

acs-bot commented Aug 13, 2020

chore: Hyperv and upstream Containerd package support #3688

chore: Hyperv and upstream Containerd package support #3688

Conversation

jsturtevant commented Aug 10, 2020 • edited Loading

jsturtevant commented Aug 10, 2020

codecov bot commented Aug 11, 2020 • edited Loading

Codecov Report

mboersma Aug 11, 2020

Choose a reason for hiding this comment

jsturtevant Aug 11, 2020

Choose a reason for hiding this comment

marosset commented Aug 11, 2020

jsturtevant commented Aug 11, 2020 • edited Loading

marosset commented Aug 12, 2020

mainred Aug 13, 2020

Choose a reason for hiding this comment

jsturtevant Aug 13, 2020

Choose a reason for hiding this comment

marosset Aug 13, 2020

Choose a reason for hiding this comment

jsturtevant Aug 13, 2020

Choose a reason for hiding this comment

marosset Aug 13, 2020

Choose a reason for hiding this comment

marosset commented Aug 13, 2020

acs-bot commented Aug 13, 2020

jsturtevant commented Aug 10, 2020 •

edited

Loading

codecov bot commented Aug 11, 2020 •

edited

Loading

jsturtevant commented Aug 11, 2020 •

edited

Loading