Add managed identity support of Azure Event Hub provider in notification-controller

[dipti-pai]

Today, notification-controller integration with Azure event hub provider supports JWT tokens and SAS string for authentication. The existing approaches have drawbacks.

• JWT token is valid only for one hour and has to be refreshed by the user by fetching a new token using Azure APIs and setting it in a Kubernetes secret.
• SAS strings are not secure and cannot be used in production.

Because of these drawbacks, the proposal is to add support for Managed Identity for Azure Event Hub.

• The existing azure-event-hubs-go sdk used in notification controller is outdated and does not support azidentity APIs. The newer azeventhubs sdk is available with managed identity support.
• Existing authentication mechanisms (JWT and SAS) continue to work with the newer azeventhubs SDK.
• If token is specified in the .spec.secretRef of the Provider object, use the ProducerClient of the SDK to specify a TokenCredential built using the JWT token.
• If address contains a SAS connection string as specified in the .spec.secretRef of the Provider object, use the ProducerClient of the SDK to connect using SAS string.
• If no token OR SAS connection string is detected, use managed identity.

To use notification-controller with Workload identity,

• User would create a managed identity and grant it Azure Event Hub Sender role on the Azure Event Hub.
• User would create a federated credential to allow notification-controller pod running in the Kubernetes cluster to send events to Azure Event Hub.
• The deployments and service account for notification-controller would be updated to use workload identity.
• The Provider spec would no longer need a .spec.secretRef, it would reference the azure event hub namespace and event hub name as shown below.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: azure
spec:
  address: <azure-event-hub-name>
  channel: <azure-event-hub-namespace>
  type: azureeventhub

[matheuscscp]

Love workload identity ❤

I'd like to register here a recent discussion around this subject we had in the flux dev meetings.

This proposal is proposing single-tenant workload identity, i.e. the managed identity is associated with notification-controller rather than with the Provider object. I'd like to propose holding back the single-tenant implementation here since it's not ideal in terms of security (e.g. for multi-tenant setups) and because there are no users depending on this feature yet (since it's not implemented). I propose starting directly with a multi-tenant approach. We've been already discussing how to implement this and it should be relatively easy with help from this library:

https://github.com/Azure/azure-sdk-for-go/blob/sdk/azidentity/v1.8.2/sdk/azidentity/client_assertion_credential.go#L53

This function receives a callback that must return a Kubernetes ServiceAccount token. We can introduce the field spec.serviceAccountName in the Provider API and use it to create a token for the ServiceAccount with the audience of the Azure Security Token Service, which is api://AzureADTokenExchange. Then we extract the Azure tenant and client IDs from the ServiceAccount annotations like this:

https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet#service-account-annotations

The access scope for Azure Event Hubs is:

https://eventhubs.azure.net//.default

With these four inputs, scope, SA token, tenant ID and client ID, we should be able to call the function proposed above to get a JWT access token from the Azure Security Token Service and use it to publish messages to an Azure Event Hub exactly the way we do right now through the JWT method, with the difference now that we generated the token ourselves from the involved identities rather than loading it from a Secret.

This solution allows each Provider object to have its own managed identity ❤

I did a quick test and this approach works:

duration = 23h59m58.999746709s
access_token = eyJ0eXAi...<redacted>...FO6DJG8w
>>> successfully sent event to azure event hub

This is in line with the the RFC we are drafting here:

https://github.com/fluxcd/flux2/blob/rfc-multi-tenant-workload-identity/rfcs/0010-multi-tenant-workload-identity/README.md

PR Link: fluxcd/flux2#5209

[BigGold1310]

@stefanprodan: We need Azure Managed Identity support for Azure Event Hub, as the JWT Secret approach doesn't work for our infrastructure requirements.

I'd like to contribute by implementing this feature in the notification-controller. I can proceed with either:

1. A single-tenancy implementation (consistent with Flux's current managed identity approach)
2. A multi-tenancy implementation

Please advise which implementation path you'd prefer so I can begin work promptly. I'm ready to submit a PR in the coming weeks.

[matheuscscp]

@BigGold1310 We are already working on this, see fluxcd/flux2#5209

[BigGold1310]

@matheuscscp Thanks for the reference. I've seen the proposal. I'd like to know which implementation is now preferred in the Azure Event Hubs case here. Based on that I would start with a PR containing an implementation proposal. Feel free to guide me on that.

[dipti-pai]

@BigGold1310 This is on the Flux Roadmap for v2.6 release and on my radar to implement. You can use this issue for tracking the progress. Thanks for offering to help, we will keep you posted if we need help.