This repository has been archived by the owner on Oct 24, 2023. It is now read-only.
feat: azure arc addon #3634
Merged
Merged
feat: azure arc addon #3634
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
97e13b0
feat: azure arc addon
jadarsie efde8f8
renamed addon
jadarsie 5d0f3c8
data model
jadarsie a42a602
product name update
jadarsie cf73cf9
e2e improvements
jadarsie 3da8ee6
examples
jadarsie be9fd14
fixed cluster config
jadarsie 25f7f71
add arc-onboarding to everything.json
jadarsie b394ca0
Merge remote-tracking branch 'upstream/master' into arc-addon
jadarsie 13f1682
Account.ResourceGroup
jadarsie 5c9792a
CreateGroup does not set Account.ResourceGroup
jadarsie f222154
fix
jadarsie File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Azure Arc enabled Kubernetes | ||
|
|
||
| You can attach and configure Kubernetes clusters by using [Azure Arc enabled Kubernetes](https://docs.microsoft.com/azure/azure-arc/kubernetes/overview). | ||
| When a Kubernetes cluster is attached to Azure Arc, it will appear in the Azure portal. It will have an Azure Resource Manager ID and a managed identity. | ||
| Clusters are attached to standard Azure subscriptions, are located in a resource group, and can receive tags just like any other Azure resource. | ||
|
|
||
| To connect a Kubernetes cluster to Azure, the cluster administrator needs to deploy agents. These agents run in a Kubernetes namespace named `azure-arc` and are standard Kubernetes deployments. The agents are responsible for connectivity to Azure, collecting Azure Arc logs and metrics, and watching for configuration requests. | ||
|
|
||
| You can deploy the Azure Arc agents either as part of the cluster creation process (by including the `azure-arc-onboarding` addon spec in your input `apimodel.json`) or manually using [azure-cli](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/connect-cluster). | ||
|
|
||
| ## Azure Arc enabled Kubernetes Addon | ||
|
|
||
| The `azure-arc-onboarding` addon creates a Kubernetes job (in namespace `azure-arc-onboarding`) in charge of deploying the Azure Arc agents. | ||
| The following information is required in order to successfully onboard the new cluster. | ||
|
|
||
| | Name | Required | Description | | ||
| | ---------------- | -------- | -------------------------------------------------------------------------------------- | | ||
| | location | yes | Azure region where the `connectedCluster` ARM resource will be created | | ||
| | subscriptionID | yes | Subscription ID where the `connectedCluster` ARM resource will be created | | ||
| | tenantID | yes | Tenant ID that owns the specified Subscription | | ||
| | resourceGroup | yes | Existing resource group name where the `connectedCluster` ARM resource will be created | | ||
| | clusterName | yes | Unique cluster friendly name | | ||
| | clientID | yes | Service principal ID with permissions to create resources in target subscription/group | | ||
| | clientSecret | yes | Service principal secret | | ||
|
|
||
| Example: | ||
|
|
||
| ```json | ||
| { | ||
| "apiVersion": "vlabs", | ||
| "properties": { | ||
| "orchestratorProfile": { | ||
| "kubernetesConfig": { | ||
| "addons": [ | ||
| { | ||
| "name": "azure-arc-onboarding", | ||
| "enabled": true, | ||
| "config": { | ||
| "tenantID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "subscriptionID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "resourceGroup": "connectedClusters", | ||
| "clusterName": "clusterName", | ||
| "clientID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "clientSecret": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "location": "eastus" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Validation / Troubleshooting | ||
|
|
||
| To make sure that the onboarding process succeded, you can either look for the new `connectedCluster` resource in the Azure portal | ||
| (ARM ID: `/subscriptions/{subscriptionID}/resourceGroups/{resourceGroup}/providers/Microsoft.Kubernetes/connectedClusters/{clusterName}`) | ||
| or check the status of the agent pods in the `azure-arc` namespace. | ||
|
|
||
| ```bash | ||
| kubectl get pods -n azure-arc | ||
| ``` | ||
|
|
||
| If you notice something wrong, the first troubleshooting step would be to inspect the logs produced by the onboarding process | ||
|
|
||
| ```bash | ||
| kubectl logs -l job-name=azure-arc-onboarding -n azure-arc-onboarding | ||
| ``` | ||
|
|
||
| #### Frequent issues | ||
|
|
||
| Potential issues you may find by inspecting the job logs include: | ||
|
|
||
| - Target resource group does not exit | ||
| - Cluster name is not unique | ||
| - Invalid service principal credentials | ||
| - Service principal does not have enough permissions to create resources in target subscription or resource group | ||
| - Azure Arc is not available in the desired Azure region | ||
|
|
||
| ### Clean up | ||
|
|
||
| You are free to delete the resources created in namespace `azure-arc` created by job `azure-arc-onboarding`. | ||
|
|
||
| However, you won't be able to permanently delete the resources created in namespace `azure-arc-onboarding` | ||
| until file `arc-onboarding.yaml` is moved out of directory `/etc/kubernetes/addons` (control plane nodes' file system) | ||
| as `addon-manager` will re-create the resources in namespace `azure-arc-onboarding`. | ||
|
|
||
| ### Addon Reconfiguration | ||
|
|
||
| There are two different ways to reconfigure the `azure-arc-onboarding` addon the cluster is deployed. | ||
|
|
||
| The safer and recommended approach is to update, on every control plane node, | ||
| the secret resource declared in the addon manifest (`/etc/kubernetes/addons/arc-onboarding.yaml`) | ||
| and re-trigger the onboarding process by deleting the `azure-arc-onboarding` namespace. | ||
|
|
||
| A faster and more fragile alternative is to edit the secret using kubectl | ||
| (`kubectl edit secret azure-arc-onboarding -n azure-arc-onboarding`) and | ||
| and re-trigger the onboarding process by deleting the onboarding job | ||
| (`kubectl delete job azure-arc-onboarding -n azure-arc-onboarding`). | ||
| Keep in mind that your changes will be lost if the secret resource is deleted at any point in the future | ||
| as `addon-manager` will recreate it using the data in `arc-onboarding.yaml`. | ||
|
|
||
| More information on how to edit a Kubernetes secret can be found [here](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually). | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| { | ||
| "apiVersion": "vlabs", | ||
| "properties": { | ||
| "orchestratorProfile": { | ||
| "orchestratorType": "Kubernetes", | ||
| "kubernetesConfig": { | ||
| "useManagedIdentity": true, | ||
| "addons": [ | ||
| { | ||
| "name": "azure-arc-onboarding", | ||
| "enabled": true, | ||
| "config": { | ||
| "tenantID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "subscriptionID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "resourceGroup": "connectedClusters", | ||
| "clusterName": "clusterName", | ||
| "clientID": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "clientSecret": "88e66958-71dd-48b9-8fed-99e13b5c0a59", | ||
| "location": "eastus" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "masterProfile": { | ||
| "count": 1, | ||
| "dnsPrefix": "", | ||
| "vmSize": "Standard_DS2_v2" | ||
| }, | ||
| "agentPoolProfiles": [ | ||
| { | ||
| "name": "agentpool", | ||
| "count": 1, | ||
| "vmSize": "Standard_DS2_v2", | ||
| "availabilityProfile": "VirtualMachineScaleSets", | ||
| "storageProfile": "ManagedDisks" | ||
| } | ||
| ], | ||
| "linuxProfile": { | ||
| "adminUsername": "azureuser", | ||
| "ssh": { | ||
| "publicKeys": [ | ||
| { | ||
| "keyData": "" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: azure-arc-onboarding | ||
| labels: | ||
| addonmanager.kubernetes.io/mode: "EnsureExists" | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: azure-arc-onboarding | ||
| namespace: azure-arc-onboarding | ||
| labels: | ||
| addonmanager.kubernetes.io/mode: "EnsureExists" | ||
| data: | ||
| TENANT_ID: {{ContainerConfigBase64 "tenantID"}} | ||
| SUBSCRIPTION_ID: {{ContainerConfigBase64 "subscriptionID"}} | ||
| RESOURCE_GROUP: {{ContainerConfigBase64 "resourceGroup"}} | ||
| CONNECTED_CLUSTER: {{ContainerConfigBase64 "clusterName"}} | ||
| LOCATION: {{ContainerConfigBase64 "location"}} | ||
| CLIENT_ID: {{ContainerConfigBase64 "clientID"}} | ||
| CLIENT_SECRET: {{ContainerConfigBase64 "clientSecret"}} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: azure-arc-onboarding | ||
| namespace: azure-arc-onboarding | ||
| labels: | ||
| addonmanager.kubernetes.io/mode: "EnsureExists" | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRoleBinding | ||
| metadata: | ||
| name: azure-arc-onboarding | ||
| labels: | ||
| addonmanager.kubernetes.io/mode: "EnsureExists" | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: cluster-admin | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: azure-arc-onboarding | ||
| namespace: azure-arc-onboarding | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: azure-arc-onboarding | ||
| namespace: azure-arc-onboarding | ||
| labels: | ||
| addonmanager.kubernetes.io/mode: "EnsureExists" | ||
| spec: | ||
| template: | ||
| spec: | ||
| serviceAccountName: azure-arc-onboarding | ||
| nodeSelector: | ||
| kubernetes.io/arch: amd64 | ||
| kubernetes.io/os: linux | ||
| containers: | ||
| - name: azure-arc-onboarding | ||
| image: {{ContainerImage "azure-arc-onboarding"}} | ||
| env: | ||
| - name: TENANT_ID | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: TENANT_ID | ||
| - name: SUBSCRIPTION_ID | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: SUBSCRIPTION_ID | ||
| - name: RESOURCE_GROUP | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: RESOURCE_GROUP | ||
| - name: CONNECTED_CLUSTER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: CONNECTED_CLUSTER | ||
| - name: LOCATION | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: LOCATION | ||
| - name: CLIENT_ID | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: CLIENT_ID | ||
| - name: CLIENT_SECRET | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: azure-arc-onboarding | ||
| key: CLIENT_SECRET | ||
| restartPolicy: Never | ||
| backoffLimit: 4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's enclose this example JSON in something like this:
{ ... "properties": { "orchestratorProfile": { "kubernetesConfig": { "addons": [ ..., <your example addons configuration here>, ... ] } }, ... }