Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

⚠️ Implement secure diagnostics (metrics, pprof, log level changes) #9264

Merged
merged 1 commit into from Sep 15, 2023
Merged

⚠️ Implement secure diagnostics (metrics, pprof, log level changes) #9264

merged 1 commit into from Sep 15, 2023

Conversation

sbueringer
Copy link
Member

@sbueringer sbueringer commented Aug 22, 2023 •
edited

Signed-off-by: Stefan Büringer buringerst@vmware.com

What this PR does / why we need it:

Changes to flags

  • deprecate --metrics-bind-addr flag
  • add new flag --diagnostics-address (default: :8443)
  • add new flag --insecure-diagnostics (default: false)

With the following behavior

  • If --metrics-bind-addr=<addr> is set:
    • metrics are served on http without authentication/authorization (as today)
  • If --diagnostics-address=<addr> --insecure-diagnostics is set
    • same behavior as --metrics-bind-addr
  • If --diagnostics-address is set
    • metrics are served on https with authentication/authorization
    • in addition pprof endpoints and an endpoint to change the log level are served (also protected)

Default behavior:

  • controller binary:
    • without any flags set: secure serving on :8443
    • with --metrics-bind-addr set: insecure serving on the specified address (as before)
  • published component YAMLs:
    • without any env vars set: secure serving on :8443
    • with CAPI_DIAGNOSTICS_ADDRESS=localhost:8080 & CAPI_INSECURE_DIAGNOSTICS=true: insecure serving on localhost:8080

This should allow a smooth transition as folks can set either the env vars for clusterctl or --metrics-bind-addr directly on the binary to preserve the current behavior

It is now possible to easily and securely expose metrics in production. Additionally, the pprof endpoint can now be also always enabled and it is possible to change log levels dynamically.

This should make it a lot easier to debug Cluster API in production.

Open TODOs:

  • doc: mention in migration guide for providers
  • doc: user guide
  • fix e2e metrics collection

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #9289

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 22, 2023
@sbueringer sbueringer changed the title [WIP] Enable secure metrics serving [WIP] ✨ Enable secure metrics serving Aug 22, 2023
@sbueringer sbueringer mentioned this pull request Aug 23, 2023
Merged
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 23, 2023
@sbueringer sbueringer mentioned this pull request Aug 23, 2023
Closed
@sbueringer sbueringer force-pushed the pr-auth-metrics branch 2 times, most recently from 8a2de8c to d74ee1f Compare August 23, 2023 13:30
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2023
@sbueringer sbueringer changed the title [WIP] ✨ Enable secure metrics serving [WIP] ✨ Implement secure diagnostics (metrics, pprof, log level changes) Sep 8, 2023
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 8, 2023
@sbueringer sbueringer changed the title [WIP] ✨ Implement secure diagnostics (metrics, pprof, log level changes) ✨ Implement secure diagnostics (metrics, pprof, log level changes) Sep 8, 2023
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 8, 2023
@sbueringer
Copy link
Member Author

This PR is now ready for review. I will finish the open todos once we have consensus on the changes
/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 8, 2023
@sbueringer sbueringer added area/metrics Issues or PRs related to metrics area/logging Issues or PRs related to logging labels Sep 8, 2023
@k8s-ci-robot k8s-ci-robot removed do-not-merge/needs-area PR is missing an area label labels Sep 8, 2023
killianmuldoon
killianmuldoon reviewed Sep 14, 2023
View reviewed changes
Copy link
Contributor

@killianmuldoon killianmuldoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great work.

Code looks good to me, haven't had time to test it locally yet though. Will look to set it up over the next couple of days. Happy to approve given others have been able to test it.

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini, killianmuldoon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [fabriziopandini,killianmuldoon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@killianmuldoon
Copy link
Contributor

Oh and thanks for the docs! They'll come in useful 😄

@sbueringer
Copy link
Member Author

cc @bavarianbidi Just fyi given previous work/interest on metrics. Just in case CAPI or CR metrics are still interesting for you

chrischdi
chrischdi reviewed Sep 15, 2023
View reviewed changes
Copy link
Member

@chrischdi chrischdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one nit

controlplane/kubeadm/main.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 15, 2023
@sbueringer
Copy link
Member Author

@chrischdi PTAL :)

@sbueringer
Implement secure diagnostics (metrics, pprof, log level changes)
500fc1f 
Signed-off-by: Stefan Büringer buringerst@vmware.com
chrischdi
chrischdi reviewed Sep 15, 2023
View reviewed changes
Copy link
Member

@chrischdi chrischdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 15, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 83d875ee147eb33e2020c2672e73dff6a7e90bff

@sbueringer
Copy link
Member Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 15, 2023
@k8s-ci-robot k8s-ci-robot merged commit 44e996c into kubernetes-sigs:main Sep 15, 2023
21 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.6 milestone Sep 15, 2023
@sbueringer sbueringer deleted the pr-auth-metrics branch September 15, 2023 10:48
@sbueringer
Copy link
Member Author

Thx everyone for the reviews!!

@suleymanakbas91 suleymanakbas91 mentioned this pull request Sep 25, 2023
Merged
@mboersma mboersma mentioned this pull request Nov 6, 2023
Merged
4 tasks
@furkatgofurov7 furkatgofurov7 mentioned this pull request Nov 30, 2023
Merged
@kashifest kashifest mentioned this pull request Dec 5, 2023
Merged
@damdo damdo mentioned this pull request Jan 18, 2024
Merged
@dlipovetsky dlipovetsky mentioned this pull request Jan 25, 2024
Closed
@jiayiwang7 jiayiwang7 mentioned this pull request Mar 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chrischdi chrischdi chrischdi left review comments

@fabriziopandini fabriziopandini Awaiting requested review from fabriziopandini

@g-gaston g-gaston Awaiting requested review from g-gaston

@killianmuldoon killianmuldoon Awaiting requested review from killianmuldoon

Assignees

@chrischdi chrischdi

@fabriziopandini fabriziopandini

@g-gaston g-gaston

@killianmuldoon killianmuldoon

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/logging Issues or PRs related to logging area/metrics Issues or PRs related to metrics cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
v1.6
Development

Successfully merging this pull request may close these issues.

Secure diagnostics (metrics, pprof, log level changes)
6 participants
@sbueringer @chrischdi @fabriziopandini @k8s-ci-robot @g-gaston @killianmuldoon