Remove dependency for kube-rbac-proxy and add secure-metrics feature #3833

super-harsh · 2024-02-28T04:18:34Z

What this PR does / why we need it:

This PR removes dependency for kube-rbac-proxy for securely serving metrics.
Instead, we have a secure-metrics feature flag in ASO using which users can toggle to enable metrics via HTTPs.

Bonus: Have added pprof handlers as well.

If applicable:

this PR contains documentation
this PR contains tests
this PR contains YAML Samples

…flag

codecov-commenter · 2024-02-28T05:36:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.40%. Comparing base (1650813) to head (99969f3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3833      +/-   ##
==========================================
- Coverage   53.42%   53.40%   -0.03%     
==========================================
  Files        1521     1521              
  Lines      546189   546189              
==========================================
- Hits       291781   291665     -116     
- Misses     209457   209573     +116     
  Partials    44951    44951

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

theunrepentantgeek

Looks good, but needs a couple minor tweaks. Approving so you can merge when they're done.

theunrepentantgeek · 2024-02-28T19:33:38Z

docs/hugo/content/guide/metrics.md

@@ -7,7 +7,7 @@ The metrics exposed fall into two groups: Azure based metrics, and reconciler me

 ## Toggling the metrics

-By default, metrics for ASOv2 are turned on and can be toggled by the following options:
+By default, secure metrics for ASOv2 are turned on and can be toggled by the following options:


Are they really turned on, given the option below says default: false ?

theunrepentantgeek · 2024-02-28T19:35:03Z

docs/hugo/content/guide/metrics.md

+
+- Use the settings below in your deployment: 
+
+- #### ASOv2 Helm Chart


I don't think this bullet point needs to be a heading - and if it were a heading, it would be 3rd level, not 4th.

Suggest changing to plain text.

theunrepentantgeek · 2024-02-28T19:35:50Z

scripts/v2/generate-helm-manifest.sh

+#grep -E $KUBE_RBAC_PROXY "$GEN_FILES_DIR"/*_deployment_* > /dev/null # Ensure that what we're about to try to replace actually exists (if it doesn't we want to fail)
+#sed -i "s@$KUBE_RBAC_PROXY.*@{{.Values.image.kubeRBACProxy}}@g" "$GEN_FILES_DIR"/*_deployment_*


Commented out - either restore or delete.

theunrepentantgeek · 2024-02-28T19:37:00Z

v2/charts/azure-service-operator/values.yaml

@@ -92,6 +92,9 @@ image:
 # 'address' field defines the metrics binding address on which metrics
 metrics:
  enable: true
+  # secureMetrics controls whether metrics should be served via 'http' or 'https'.
+  # Flagging secureMetrics as true would use https
+  secureMetrics: false


The comment in this yaml file reinforces my earlier question about whether secure metrics are turned on by default.

v2/config/rbac/auth_proxy_service.yaml

v2/charts/azure-service-operator/values.yaml

docs/hugo/content/guide/metrics.md

v2/cmd/controller/app/flags.go

matthchr · 2024-03-05T22:38:06Z

v2/cmd/controller/app/setup.go

+			SecureServing:  true,
+			FilterProvider: filters.WithAuthenticationAndAuthorization,
+			// Note that pprof endpoints are meant to be sensitive and shouldn't be exposed publicly.
+			ExtraHandlers: map[string]http.Handler{


We probably should have a specific cmdline flag that enables this and it should default to off.

This is what apiserver does via the --profiling argument, though it defaults it to true.

I'd argue we should have both metrics.secure and metrics.profiling flags in Helm, and two cmdline args in ASO to control these two bits.

Secure metrics shouldn't (IMO) require that you expose pprof.

v2/config/manager/manager_metrics_service.yaml

v2/config/rbac/crd_manager_role.yaml

matthchr · 2024-03-06T23:46:58Z

v2/cmd/controller/app/setup.go

+func getMetricsOpts(flags Flags) server.Options {
+	var metricsOptions server.Options
+
+	if !flags.SecureMetrics {


This code doesn't seem right, because it doesn't return here, it falls through and then overwrites the metricsOptions below?

I think you want something like:

var metricsOptions if secure { // secure } else { // insecure } if profiling { // profiling } return

Or if you want to only enable profiling if secure, put that bit into the secure bit, and/or (maybe better) add a check that secure is not off with profiling on and if so return an error and crash the pod.

Ah good catch. I missed return there. I'll update

matthchr · 2024-03-06T23:47:40Z

v2/charts/azure-service-operator/values.yaml

+  # Flagging secure as 'true' would use https
+  # Refer to https://azure.github.io/azure-service-operator/guide/metrics/ for more information
+  secure: true
+  # profiling endpoints are only enabled when serving metrics securely


minor: Say what profiling is (it enables /pprof endpoints), in addition to its restrictions. Though, see below about this restriction because I am not sure it is needed?

matthchr · 2024-03-06T23:48:36Z

v2/cmd/controller/app/flags.go

@@ -54,6 +60,9 @@ func ParseFlags(args []string) (Flags, error) {

 	// default here for 'MetricsAddr' is set to "0", which sets metrics to be disabled if 'metrics-addr' flag is omitted.
 	flagSet.StringVar(&metricsAddr, "metrics-addr", "0", "The address the metric endpoint binds to.")
+	flagSet.BoolVar(&secureMetrics, "secure-metrics", true, "Enable secure metrics. This secures the pprof and metrics endpoints via Kubernetes RBAC and HTTPS")
+	flagSet.BoolVar(&profilingMetrics, "profiling-metrics", true, "Enable pprof metrics, only enabled in conjunction with secure-metrics. This will enable serving pprof metrics endpoints")


Default should probably be false

matthchr

Approved with one minor style comment

matthchr · 2024-03-07T21:28:46Z

docs/hugo/content/guide/metrics.md

+
+Follow the steps below to scrape metrics securely.
+
+### ASOv2 Helm Chart


very minor: Could use one of the table-selector thingies here, like we do for Windows versus Linux in the installation instructions?

Remove dependency for kube-rbac-proxy and add secure-metrics feature …

7938491

…flag

super-harsh requested review from davefellows, theunrepentantgeek, matthchr and babbageclunk as code owners February 28, 2024 04:18

super-harsh added 2 commits February 28, 2024 17:19

Update values.yaml

ec4b7fb

Update kustomize.yaml

3df85ba

theunrepentantgeek assigned super-harsh Feb 28, 2024

theunrepentantgeek approved these changes Feb 28, 2024

View reviewed changes

matthchr reviewed Feb 28, 2024

View reviewed changes

v2/config/rbac/auth_proxy_service.yaml Show resolved Hide resolved

v2/charts/azure-service-operator/values.yaml Outdated Show resolved Hide resolved

docs/hugo/content/guide/metrics.md Show resolved Hide resolved

super-harsh added 2 commits March 4, 2024 10:39

Update to default enable secureMetrics; Update role and rolebindings

f0dc11a

Add metrics auth filter

93964d0

matthchr reviewed Mar 5, 2024

View reviewed changes

super-harsh added 3 commits March 6, 2024 16:12

Add seperate arg for profiling-metrics

b15c3f7

Refactor

e544203

Merge branch 'main' into remove/rbac-proxy

a04af18

super-harsh mentioned this pull request Mar 6, 2024

Tracking issue for 2.7.0 release #3846

Closed

fix typo

5283762

matthchr reviewed Mar 6, 2024

View reviewed changes

super-harsh added 2 commits March 7, 2024 13:19

Resolve comments

8f243f3

gofmt file

99969f3

matthchr approved these changes Mar 7, 2024

View reviewed changes

Add tabpane for securemetrics docs

752e319

super-harsh added this pull request to the merge queue Mar 8, 2024

Merged via the queue into main with commit 034c9e3 Mar 8, 2024
9 checks passed

super-harsh deleted the remove/rbac-proxy branch March 8, 2024 03:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove dependency for kube-rbac-proxy and add secure-metrics feature #3833

Remove dependency for kube-rbac-proxy and add secure-metrics feature #3833

super-harsh commented Feb 28, 2024 •

edited

codecov-commenter commented Feb 28, 2024 •

edited

theunrepentantgeek left a comment

theunrepentantgeek Feb 28, 2024

theunrepentantgeek Feb 28, 2024

theunrepentantgeek Feb 28, 2024

theunrepentantgeek Feb 28, 2024

matthchr Mar 5, 2024

matthchr Mar 6, 2024

matthchr Mar 6, 2024

super-harsh Mar 7, 2024

matthchr Mar 6, 2024

matthchr Mar 6, 2024

matthchr left a comment

matthchr Mar 7, 2024


		- Use the settings below in your deployment:

		- #### ASOv2 Helm Chart

		#grep -E $KUBE_RBAC_PROXY "$GEN_FILES_DIR"/_deployment_ > /dev/null # Ensure that what we're about to try to replace actually exists (if it doesn't we want to fail)
		#sed -i "s@$KUBE_RBAC_PROXY.@{{.Values.image.kubeRBACProxy}}@g" "$GEN_FILES_DIR"/_deployment_*


		Follow the steps below to scrape metrics securely.

		### ASOv2 Helm Chart

Remove dependency for kube-rbac-proxy and add secure-metrics feature #3833

Remove dependency for kube-rbac-proxy and add secure-metrics feature #3833

Conversation

super-harsh commented Feb 28, 2024 • edited

codecov-commenter commented Feb 28, 2024 • edited

Codecov Report

theunrepentantgeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

super-harsh commented Feb 28, 2024 •

edited

codecov-commenter commented Feb 28, 2024 •

edited