Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prometheus for seed monitoring #4560

Merged
merged 1 commit into from
Sep 17, 2021
Merged

Conversation

wyb1
Copy link
Contributor

@wyb1 wyb1 commented Aug 26, 2021

How to categorize this PR?

/area monitoring
/kind enhancement

What this PR does / why we need it:

Add a new Prometheus that will monitor the components running in the extension-* and garden namespaces. All extensions will have cadvisor metrics monitored by default. If extensions are exposing custom metrics they will be scraped if the following annotations are added to the pods:

prometheus.io/scrape=true
prometheus.io/port=<port>
  • Add VPA for the prometheus
  • Add Prometheus as a datasource to grafana
  • Add documentation for prometheus instances
  • Add unit tests for new network policies

/cc @MartinWeindel

Which issue(s) this PR fixes:
Fixes #2946

Special notes for your reviewer:

In the long term this new Prometheus could eventually replace the aggregate prometheus. We will need some experience with how this Prometheus handles the load of the cadvisor metrics.

Release note:

Add a new Prometheus that operators can use to monitor extensions and components running in the garden namespace

istvanballok
istvanballok previously approved these changes Aug 26, 2021
Copy link
Contributor

@istvanballok istvanballok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@timebertt
Copy link
Member

Nice! Looking forward to using it :)

Probably it's a good idea to directly build this new component as a golang component instead of a helm chart. Otherwise, we will refactor it sooner or later (see #2754).

@gardener-robot
Copy link

@wyb1 You need rebase this pull request with latest master branch. Please check.

@timebertt
Copy link
Member

/assign

@gardener-robot gardener-robot added needs/review kind/api-change API change with impact on API users size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 6, 2021
@wyb1
Copy link
Contributor Author

wyb1 commented Sep 6, 2021

Rebased. Will work on refactoring this PR as a golang component.

@timebertt
Copy link
Member

As discussed offline, let's not put effort into refactoring this PR into a golang component and proceed with it in the current state.

At a later point, we rather want to evaluate whether and how we can/want to extract monitoring and logging components out of the gardener/gardener repository into dedicated repositories that can be developed, released and deployed individually like gardener extensions.

Copy link
Member

@timebertt timebertt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I like it! :)
Two suggestions from my side, rest the looks promising already.

docs/monitoring/README.md Show resolved Hide resolved
@rfranzke
Copy link
Member

/author-action

@gardener-robot
Copy link

@wyb1 The pull request was assigned to you under author-action. Please unassign yourself when you are done. Thank you.

- Add VPA for the prometheus
- Add Prometheus as a datasource to grafana
- Add documentation for prometheus instances
- Add unit tests for new network policies

Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>
Copy link
Member

@timebertt timebertt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, looks great now!
/lgtm

@timebertt
Copy link
Member

@vpnachev @rfranzke do you plan to take another look?

@rfranzke
Copy link
Member

/assign @vpnachev for review and milestone

@vpnachev vpnachev added this to the v1.32 milestone Sep 17, 2021
@vpnachev vpnachev merged commit 9895fdb into gardener:master Sep 17, 2021
@wyb1 wyb1 deleted the seed-monitoring branch September 20, 2021 07:05
@gardener-robot gardener-robot added area/monitoring Monitoring (including availability monitoring and alerting) related kind/enhancement Enhancement, improvement, extension labels Sep 20, 2021
istvanballok added a commit to istvanballok/gardener that referenced this pull request Dec 10, 2021
The "extensions" and "garden" scrape configuration scrapes the pods that
are marked with an annotation in the garden and extension-.+ namespaces.

See gardener#4560 for more details.

Previously, only the pod labels were kept as metric labels, the pod name
and the namespace were missing.

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>
rfranzke pushed a commit that referenced this pull request Dec 10, 2021
The "extensions" and "garden" scrape configuration scrapes the pods that
are marked with an annotation in the garden and extension-.+ namespaces.

See #4560 for more details.

Previously, only the pod labels were kept as metric labels, the pod name
and the namespace were missing.

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
krgostev pushed a commit to krgostev/gardener that referenced this pull request Apr 21, 2022
The "extensions" and "garden" scrape configuration scrapes the pods that
are marked with an annotation in the garden and extension-.+ namespaces.

See gardener#4560 for more details.

Previously, only the pod labels were kept as metric labels, the pod name
and the namespace were missing.

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
krgostev pushed a commit to krgostev/gardener that referenced this pull request Jul 5, 2022
The "extensions" and "garden" scrape configuration scrapes the pods that
are marked with an annotation in the garden and extension-.+ namespaces.

See gardener#4560 for more details.

Previously, only the pod labels were kept as metric labels, the pod name
and the namespace were missing.

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/monitoring Monitoring (including availability monitoring and alerting) related kind/api-change API change with impact on API users kind/enhancement Enhancement, improvement, extension size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add custom monitoring for extensions
9 participants