Skip to content
This repository has been archived by the owner on Oct 24, 2023. It is now read-only.

Commit

Permalink
feat: Experimental support for Windows+ContainerD (#1322)
Browse files Browse the repository at this point in the history
  • Loading branch information
@marosset
marosset committed Mar 5, 2020
1 parent 1bbc94a commit b068aa7
Show file tree
Hide file tree
Showing 22 changed files with 1,244 additions and 96 deletions.
8 changes: 6 additions & 2 deletions docs/topics/clusterdefinitions.md
View file
@@ -1,5 +1,7 @@
# Cluster Definitions



## Cluster Defintions for apiVersion "vlabs"

Here are the cluster definitions for apiVersion "vlabs":
Expand Down Expand Up @@ -41,13 +43,15 @@ $ aks-engine get-versions
| apiServerConfig | no | Configure various runtime configuration for apiserver. See `apiServerConfig` [below](#feat-apiserver-config) |
| cloudControllerManagerConfig | no | Configure various runtime configuration for cloud-controller-manager. See `cloudControllerManagerConfig` [below](#feat-cloud-controller-manager-config) |
| clusterSubnet | no | The IP subnet used for allocating IP addresses for pod network interfaces. The subnet must be in the VNET address space. With Azure CNI enabled, the default value is 10.240.0.0/12. Without Azure CNI, the default value is 10.244.0.0/16. |
| containerRuntime | no | The container runtime to use as a backend. The default is `docker`. The other options are `kata-containers`, and `containerd` |
| containerRuntime | no | The container runtime to use as a backend. The default is `docker`. The other options are `kata-containers`, and `containerd`. Windows support for `containerd` is **Experimental** - see [Windows ContainerD](features.md#windows-containerd) |
| controllerManagerConfig | no | Configure various runtime configuration for controller-manager. See `controllerManagerConfig` [below](#feat-controller-manager-config) |
| customWindowsPackageURL | no | Configure custom windows Kubernetes release package URL for deployment on Windows. The format of this file is a zip file with multiple items (binaries, cni, infra container) in it. This setting will be deprecated in a future release of aks-engine where the binaries will be pulled in the format of Kubernetes releases that only contain the kubernetes binaries. |
| WindowsNodeBinariesURL | no | Windows Kubernetes Node binaries can be provided in the format of Kubernetes release (example: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#node-binaries-1). This setting allows overriding the binaries for custom builds. |
| WindowsContainerdURL | no (for development only) | **Experimental** - see [Windows ContainerD](features.md#windows-containerd) |
| WindowsSdnPluginURL | no (for development only) | **Experimental** - see [Windows ContainerD](features.md#windows-containerd) |
| dnsServiceIP | no | IP address for coredns or kube-dns to listen on. If specified must be in the range of `serviceCidr` |
| mobyVersion | no (for development only) | Enables an explicit moby version, e.g. `3.0.3`. Default is `3.0.5`. This `kubernetesConfig` property is for development only, and applies only to cluster creation: `aks-engine upgrade` will always statically set `mobyVersion` to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of moby. |
| containerdVersion | no (for development only) | Enables an explicit containerd version, e.g. `1.1.4`. Default is `1.1.5`. This `kubernetesConfig` property is for development only, and applies only to cluster creation: `aks-engine upgrade` will always statically set `containerdVersion` to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of containerd. |
| containerdVersion | no (for development only) | Enables an explicit containerd version, e.g. `1.1.4`. Default is `1.1.5`. This `kubernetesConfig` property is for development only, and applies only to cluster creation: `aks-engine upgrade` will always statically set `containerdVersion` to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of containerd. This value is currently ignored for Windows. |
| dockerBridgeSubnet | no | The specific IP and subnet used for allocating IP addresses for the docker bridge network created on the kubernetes master and agents. Default value is 172.17.0.1/16. This value is used to configure the docker daemon using the [--bip flag](https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0) |
| enableAggregatedAPIs | no | Enable [Kubernetes Aggregated APIs](https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/). enableRbac must be set to true to use aggregated APIs. Aggregated API functionality is required by [Service Catalog](https://github.com/kubernetes-incubator/service-catalog/blob/master/README.md). (boolean - default is true) |
| enableDataEncryptionAtRest | no | Enable [kubernetes data encryption at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).This is currently an alpha feature. (boolean - default == false) |
Expand Down
41 changes: 36 additions & 5 deletions docs/topics/features.md
View file
Expand Up @@ -2,17 +2,17 @@

|Feature|Status|API Version|Example|Description|
|---|---|---|---|---|
|Managed Disks|Beta|`vlabs`|[kubernetes-vmas.json](../../examples/disks-managed/kubernetes-vmas.json)|[Description](#feat-managed-disks)|
|Antrea Network Policy|Alpha|`vlabs`|[kubernetes-antrea.json](../../examples/networkpolicy/kubernetes-antrea.json)|[Description](#feat-antrea)|
|Azure Key Vault Encryption|Alpha|`vlabs`|[kubernetes-keyvault-encryption.json](../../examples/kubernetes-config/kubernetes-keyvault-encryption.json)|[Description](#feat-keyvault-encryption)|
|Calico Network Policy|Alpha|`vlabs`|[kubernetes-calico.json](../../examples/networkpolicy/kubernetes-calico-azure.json)|[Description](#feat-calico)|
|Cilium Network Policy|Alpha|`vlabs`|[kubernetes-cilium.json](../../examples/networkpolicy/kubernetes-cilium.json)|[Description](#feat-cilium)|
|Antrea Network Policy|Alpha|`vlabs`|[kubernetes-antrea.json](../../examples/networkpolicy/kubernetes-antrea.json)|[Description](#feat-antrea)|
|ContainerD Runtime for Windows|Experimental|`vlabs`|[kubernetes-hybrid.containerd.json](../../examples/windows/kubernetes-hybrid.containerd.json)|[Description](#windows-containerd)|
|Custom VNET|Beta|`vlabs`|[kubernetesvnet-azure-cni.json](../../examples/vnet/kubernetesvnet-azure-cni.json)|[Description](#feat-custom-vnet)|
|Ephemeral OS Disks|Experimental|`vlabs`|[ephmeral-disk.json](../../examples/disks-ephemeral/ephemeral-disks.json)|[Description](#ephemeral-os-disks)|
|Kata Containers Runtime|Alpha|`vlabs`|[kubernetes-kata-containers.json](../../examples/kubernetes-kata-containers.json)|[Description](#feat-kata-containers)|
|Managed Disks|Beta|`vlabs`|[kubernetes-vmas.json](../../examples/disks-managed/kubernetes-vmas.json)|[Description](#feat-managed-disks)|
|Private Cluster|Alpha|`vlabs`|[kubernetes-private-cluster.json](../../examples/kubernetes-config/kubernetes-private-cluster.json)|[Description](#feat-private-cluster)|
|Azure Key Vault Encryption|Alpha|`vlabs`|[kubernetes-keyvault-encryption.json](../../examples/kubernetes-config/kubernetes-keyvault-encryption.json)|[Description](#feat-keyvault-encryption)|
|Shared Image Gallery images|Alpha|`vlabs`|[custom-shared-image.json](../../examples/custom-shared-image.json)|[Description](#feat-shared-image-gallery)|
|Ephemeral OS Disks|Experimental|`vlabs`|[ephmeral-disk.json](../../examples/disks-ephemeral/ephemeral-disks.json)|[Description](#ephemeral-os-disks)|


<a name="feat-kubernetes-msi"></a>

Expand Down Expand Up @@ -543,3 +543,34 @@ We are investigating possible risks & mitigations for when VMs are deprovisioned


[Ephemeral OS Disks]: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/ephemeral-os-disks


## Windows ContainerD

> This feature is currently experimental, and has open issues.

Kubernetes 1.18 introduces alpha support for the ContainerD runtime on Windows Server 2019. This is still a work-in-progress tracked in [kubernetes/enhancements#1001](https://github.com/kubernetes/enhancements/issues/1001). This feature in AKS-Engine is for testing the in-development versions of ContainerD and Kubernetes, and is not for production use. Be sure to review [open issues](https://github.com/azure/aks-engine/issues?q=containerd+label%3Awindows+is%3Aopen) if you want to test or contribute to this effort.

Currently it only supports the `kubenet` networking model, and requires URLs to custom ContainerD and CNI plugin builds.

### Deploying multi-OS clusters with ContainerD

If you want to test or develop with Windows & ContainerD in AKS-Engine, see this sample
[kubernetes-hybrid.containerd.json](../../examples/windows/kubernetes-hybrid.containerd.json)

These parameters are all required.

```json
"kubernetesConfig": {
"networkPlugin": "kubenet",
"containerRuntime": "containerd",
"windowsContainerdURL": "...",
"windowsSdnPluginURL": "..."
}
```

### Building ContainerD

As of March 3, 2020, the ContainerD and network plugin repos don't have public builds available. This repo has a script that will build them from source and create two ZIP files: [build-windows-containerd.sh](../../scripts/build-windows-containerd.sh)

Upload these ZIP files to a location that your cluster will be able to reach, then put those URLs in `windowsContainerdURL` and `windowsSdnPluginURL` in the AKS-Engine apimodel shown above.
55 changes: 55 additions & 0 deletions examples/windows/kubernetes-hybrid.containerd.json
View file
@@ -0,0 +1,55 @@
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.18",
"kubernetesConfig": {
"networkPlugin": "kubenet",
"containerRuntime": "containerd",
"windowsContainerdURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cri-containerd.zip",
"windowsSdnPluginURL": "https://aksenginee2etestimages.blob.core.windows.net/test-content/windows-cni-containerd.zip"
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "linuxpool1",
"count": 2,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet"
},
{
"name": "windowspool2",
"count": 2,
"vmSize": "Standard_D2s_v3",
"availabilityProfile": "AvailabilitySet",
"osType": "Windows"
}
],
"windowsProfile": {
"adminUsername": "azureuser",
"adminPassword": "replacepassword1234$",
"enableAutomaticUpdates": false,
"sshEnabled": true
},
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}
11 changes: 11 additions & 0 deletions parts/k8s/kuberneteswindowsfunctions.ps1
View file
Expand Up @@ -220,3 +220,14 @@ function Register-NodeResetScriptTask {
$definition = New-ScheduledTask -Action $action -Principal $principal -Trigger $trigger -Description "k8s-restart-job"
Register-ScheduledTask -TaskName "k8s-restart-job" -InputObject $definition
}

function Assert-FileExists {
Param(
[Parameter(Mandatory=$true,Position=0)][string]
$Filename
)

if (-Not (Test-Path $Filename)) {
throw "$Filename does not exist"
}
}

0 comments on commit b068aa7

Please sign in to comment.