Enable GitOps config as an AKS add-on

[seanmck]

This issue tracks enabling of GitOps config as an AKS add-on. This add-on will make it easy for AKS clusters to participate in GitOps-based deployments alongside Azure Arc-connected clusters:

https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/use-gitops-connected-cluster

[ghost]

Hi seanmck, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[asubmani]

How would this affect AKS clusters that use Flux (v1 or V2) ? We have deployed Flux in its own namespace. If this is available as an update to an existing cluster than all it takes is to remove the current deployment and enable the managed Flux.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[asubmani]

how often would the gitops toolkit being used be updated? I am interested in knowing the support lifecycle policy of updating these with upstream versions of the gitops tool (flux argo CD etc)

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ernesto1596]

Is there any update for this feature or ETA?

[pl4nty]

This feature is described as available in the Azure Arc documentation and Azure Policy, but sourceControlConfiguration deployments to my Microsoft.ContainerService/managedClusters via policy remediation and az k8s-configuration are stuck at "complianceState": "Pending". Is this expected behaviour? If so, would connecting the cluster to Azure Arc be a suitable workaround?

Edit: connected the cluster to Arc successfully

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[ghost]

@Azure/aks-pm issue needs labels

[garrardkitchen]

Is there an ETA update on this feature? 🙏

[thepaulmacca]

Is there an ETA update on this feature? 🙏

I'm keen to know this as well. A bit concerning there's been no update for a while. Nothing stopping you from enabling it now, but would be good to know if that's advised or not at this time

[garrardkitchen]

Is there an ETA update on this feature? 🙏

I'm keen to know this as well. A bit concerning there's been no update for a while. Nothing stopping you from enabling it now, but would be good to know if that's advised or not at this time

I had attempted to add feature but sadly not available in UKSouth 🥲

[thepaulmacca]

I had attempted to add feature but sadly not available in UKSouth 🥲

Ah right ok, i was using it in West Europe. This is a good article to read if you haven't already https://pixelrobots.co.uk/2021/07/first-look-gitops-on-azure-kubernetes-service-using-the-aks-addon-preview/

[PixelRobots]

The official docs can be found at https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2. For some reason they are under arc and not AKS which is a bit confusing for customers. Would be good to separate them out.

[thepaulmacca]

The official docs can be found at https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2. For some reason they are under arc and not AKS which is a bit confusing for customers. Would be good to separate them out.

Agree with this

[rossmobi]

Will GitOps be enabled as an add-on rather than an extension?

Automatically installing as an add-on enables some enterprise-style deployment flows which we otherwise can't have if we need to install the extension e.g. using Azure Policy to DINE a fluxConfiguration item which will ensure mandatory components are installed on all AKS clusters.

[PixelRobots]

From what I know the AKS addon for GitOps is no longer a thing. It was just for flux v1. The extension is the new and only way to install it as it needs to integrate with ARM and addons cannot do that.

[rossmobi]

@PixelRobots

From what I know the AKS addon for GitOps is no longer a thing. It was just for flux v1.

Yeah I understand this to be true for v1

The extension is the new and only way to install it as it needs to integrate with ARM and addons cannot do that.

Is that true?

Add-ons like Azure Policy or OMS Agent are defined in the body of a request for Microsoft.ContainerService/managedClusters, meaning they are processed by Azure RM and are very much enforceable with Azure Policy for example. And so far, I've only seen extensions installed by use of Azure CLI, completely outside the scope of an Azure RM request.

So to my knowledge, the opposite of what you wrote is true; add-ons are integrated with ARM and extensions are not. No?

Happy to be enlightened if I've misunderstood :)

[Agger1995]

@rossdotpink
Both AKS extensions and AKS add-ons can be defined in ARM templates.
For our project, in which we will be using GitOps for deploying our workloads, we have defined our cluster as IaC in Bicep where we "install" the Flux component as an AKS extension, scoped to the AKS cluster resource.
In Bicep it looks like this:

resource fluxExtension 'Microsoft.KubernetesConfiguration/extensions@2021-09-01' = {
  scope: aksCluster
  name: 'flux'
  properties: {
    extensionType: 'Microsoft.Flux'
    ... # Omitted for brevity
  }
}

There is limited documentation available on how to configure this. However, I got a lot from looking at how it is implemented in the AKS Secure Baseline.
See https://github.com/mspnp/aks-baseline/blob/main/cluster-stamp.json#L1015-L1085

[PixelRobots]

Hi, sorry for the confusion. When I say integrated with ARM I am not meaning ARM templates or Bicep.

Extensions can be installed via ARM and Bicep as @Agger1995 has shown above.

What I was referring to can be read about in the docs. https://docs.microsoft.com/en-us/azure/aks/integrations#extensions

"Cluster extensions build on top of certain Helm charts and provide an Azure Resource Manager-driven experience for installation and lifecycle management of different Azure capabilities on top of your Kubernetes cluster."

This allows for the GitOps section to show in the Azure portal from what I understand.

Hope that clears some stuff up for you.

[rossmobi]

@Agger1995 Thanks! I did not know I could do that, I will give that a go. Totally agree about the documentation, but it's getting better :)

@PixelRobots Thanks :) main point I was missing is that extensions can indeed be installed as part of ARM/Bicep request, rather than just AZ CLI. Also, thank you for the Pixel Robots website; I knew I recognised it from somewhere, some of the first material I found on GitOps Flux on AKS was on your website.

[PixelRobots]

Any time. glad it helped.

AKS addons and AKS Cluster Extensions can get a bit confusing especially when it comes to updating them. Check out my blog post on that to ensure you keep your clusters updated. https://pixelrobots.co.uk/2022/02/azure-kubernetes-service-aks-addons-and-extensions-who-is-responsible/

[davidkarlsen]

When will this be available in Norwayeast?