feat(scaler/prometheus): support authentication for Google Managed Prometheus

[nettoclaudio]

This PR adds Google's authentication in order to access metrics from Google Managed Prometheus.

Checklist

• When introducing a new scaler, I agree with the scaling governance policy
• I have verified that my change is according to the deprecations & breaking changes policy
• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4674

Relates to kedacore/keda-docs#1153

[github-actions]

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

• Our contribution guide

[JorTurFer]

Looking good! ❤️
Could you add an e2e test too? You can add any required service/permission using the testing-infrastructure repository. We manage the infrestructure using terraform 😄

[nettoclaudio]

Sure, I do. Although I think I'll need an extra hand to configure it correctly. 😬

Firstly, can you clarify some doubts about the infrastructure?

1. As far as I understood in the GCP module of testing-infrastructure, we just configure IAM to access stuff from the AKS cluster. All the e2e tests run there (AKS), right? Is there some chance to run a GKE cluster as well? If not, we will need configure a Prometheus (e.g. Prometheus Agent) to export to metrics to Google Managed Prometheus. (read more)

2. Just double checking. In the GCP module I see a few resources configuring the workload identity pools/providers. Is that a federation between Azure and GCP IAM? If so, the resulting service account key in the ouput output.e2e_user_credentials has admin permission in the e2e's GCP project? If a just create a Deployment in the AKS cluster, they will be able to successfully autheticate in the GCP using workload identity?

[JorTurFer]

Sure, you can ask whatever you need 😄 (we have a Slack channel in Kubernetes workspace for development stuff if you prefer that channel to be faster during the development)

As far as I understood in the GCP module of testing-infrastructure, we just configure IAM to access stuff from the AKS cluster. All the e2e tests run there (AKS), right? Is there some chance to run a GKE cluster as well? If not, we will need configure a Prometheus (e.g. Prometheus Agent) to export to metrics to Google Managed Prometheus. (read more)

We prefer to run all the test in AKS if it's possible, otherwise we'd need to manage multiple clusters in different platforms for main and also for PRs (there are 2 different clusters). If we need to export metrics, we could use opentelemetry collector to export the metrics to there using the googlemanagedprometheusexporter but I don't think that's necessary. I mean, as this test won't test the prometheus scaler behavior itself but the authentication, we could just use an scalar query in the ScaledObject and just update it updating the scaled object. We are already doing it for some tests: https://github.com/kedacore/keda/blob/main/tests/scalers/azure/azure_log_analytics/azure_log_analytics_test.go#L119

Just double checking. In the GCP module I see a few resources configuring the workload identity pools/providers. Is that a federation between Azure and GCP IAM? If so, the resulting service account key in the ouput output.e2e_user_credentials has admin permission in the e2e's GCP project? If a just create a Deployment in the AKS cluster, they will be able to successfully autheticate in the GCP using workload identity?

The federation between KEDA and GCP is already done. KEDA can access to GCP resources using workload identity federation (for example, this). I think that for this tests you need to:

• create the managed prometheus resource
• assign the required roles to the already existing account (this maybe it's not needed because the account is already owner)
• export the required variables like the prometheus url as GH secret
• as we mount all the secrets as envs during e2e tests, you can read them directly from there during the e2e tests and setup the tests
• execute the test

As I said, the federation is already done , the mutating webhook is already there, and everything should be working for just adding the extra required info for the new tests

[nettoclaudio]

Got it... thank you so much, Jorge.

I mean, as this test won't test the prometheus scaler behavior itself but the authentication, we could just use an scalar query in the ScaledObject and just update it updating the scaled object.

I absolutely agree. Whaterever rather than unauthenticated status code should be enough in this test case. So gimme a bunch of minutes trying to create it.

[JorTurFer]

Sure, let me know if I can help with something 😄

[JorTurFer]

/run-e2e gcp
Update: You can check the progress here