Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mgr: perf counters node exporter #45220

Merged
merged 24 commits into from May 10, 2022
Merged

mgr: perf counters node exporter #45220

merged 24 commits into from May 10, 2022

Conversation

avanthakkar
Copy link
Contributor

@avanthakkar avanthakkar commented Mar 2, 2022 •
edited

The number of perf counters keeps growing in each ceph version and mgr's performance is fragile when moving big payloads of data around because of subinterpreters. A way to reduce this load is by moving perf counters away from the mgr.

update: This PR will be the first of many which will be merge onto the feature branch feature-exporter. This PR is intended to have all features marked as done in Next steps except TLS support.

Next steps:

  • List socket files and make use of them
  • Create http server to expose GET /metrics for prometheus
  • Load perf counters from all daemons and transform to prometheus format
  • Containarize
  • Introduce a exporter container for cephadm
  • MetricCollector discovery in cephadm in order to configure prometheus to scrape all MetricCollector endpoints (mgr/cephadm: Adding prometheus service discovery endpoints #45425)
  • Ensure rook support
  • Posiblity to expose perf counters so that modules can request through http/whatever
  • Security: HTTPS / TLS 🆕
  • Security: HTTP Basic Auth 🆕
  • Ops: configuration (via ceph.conf + mon config opts) 🆕
  • Ops: application log (errors, warnings, etc.) 🆕
  • Ops: server log (HTTP requests) 🆕
  • Performance: HTTP compression 🆕
  • Performance: memory/cpu footprint, soak testing (mem-leak) 🆕
  • Documentation 🆕
  • Tests: Integration tests with teuthology
  • Tests: Add unit testing

Proposal for node-exporter:
Screenshot from 2022-03-02 21-30-43

Signed-off-by: Pere Diaz Bou pdiazbou@redhat.com
Co-authored-by: Avan Thakkar athakkar@redhat.com

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@avanthakkar avanthakkar changed the title mgr: perf counters exporter [[WIP] mgr: perf counters exporter Mar 2, 2022
@avanthakkar avanthakkar changed the title [[WIP] mgr: perf counters exporter [[WIP] mgr: perf counters node exporter Mar 2, 2022
@pereman2 pereman2 added this to In progress in Dashboard via automation Mar 2, 2022
cbodley
cbodley reviewed Mar 11, 2022
View reviewed changes
Copy link
Contributor

@cbodley cbodley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the beast stuff in http_connection looks good, nice work so far

src/msg/msg_types.h Outdated Show resolved Hide resolved
src/ceph_exporter.cc Outdated Show resolved Hide resolved
src/mgr/DaemonMetricCollector.cc Outdated Show resolved Hide resolved
pereman2 and others added 16 commits April 27, 2022 15:41
@pereman2 @avanthakkar
exporter: exporter dir and http_server.h
b4d3d57 
* move all related files to the exporter to src/exporter
* move http_server logic to a new file under src/exporter/http_server.*

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
@avanthakkar
exporter: add ceph-exporter package
d28d48c 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
mgr/cephadm: adding ceph exporter container
db50570 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@pereman2 @avanthakkar
exporter: quote label values
f3f160e 
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
@avanthakkar
exporter: ignore mgr perf counters + change port to 9926 + cleanups
9217029 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter: make keep-alive connection based on client
0dea32c 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter: fix metrics names + scrape all exporter endpoints in cephadm
33c4b82 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@pereman2 @avanthakkar
exporter: Cleanups on DaemonMetricCollector and google styling
b34d925 
1. Formatted all exporter files to google c++ styling.
2. Refactored send_requests a bit to increase readabilty.
3. Removed unused functions + variables.
4. Moved fs::path socketdir to a static const char* SOCKETDIR.

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
@avanthakkar
exporter: fix lint errors + remove exporter entity
d42ea08 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter: update package description
de0877d 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter: cephadm tox fixes
6b4182c 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter-cephadm: increase number of docker.io occurences
69d7c0b 
This is just a temporary change, as in future once the feature branch is merged in master the exporter
would be included in default master ceph image
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@pereman2 @avanthakkar
exporter: cleanups
b1ca06a 
- DaemonMetricCollector singleton with static variable
- some iterator references
- DaemonMetricCollector chagne timer to steady_timer with chrono
- AdminClientSocket handle error when send_request.
- update_sockests now clears the list of sockets before adding new ones.
- use string_body instead of dynamic_body in http::response

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
@avanthakkar
exporter: addressing cephadm cleanups/suggesstions
4af3d71 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
exporter: fix run-tox-cephadm/mgr
39d9cd1 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@avanthakkar
expoter: addressing http server fixes
a2b7e7f 
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
@pereman2
exporter: move ceph_exporter.cc to src/exporter
4b97824 
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
@pereman2
exporter: guard metrics string
79efb79 
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Dashboard automation moved this from Review in progress to Reviewer approved May 6, 2022
@pereman2 pereman2 merged commit 60460a3 into ceph:feature-exporter May 10, 2022
11 of 12 checks passed
@pereman2 pereman2 deleted the exporter-poc branch May 10, 2022 11:53
Dashboard automation moved this from Reviewer approved to Done May 10, 2022
@pereman2
Copy link
Contributor

We'll keep track of the whole progress in #46213.

@cbodley cbodley mentioned this pull request Jun 1, 2022
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pereman2 pereman2 pereman2 approved these changes

@neha-ojha neha-ojha Awaiting requested review from neha-ojha

@pcuzner pcuzner Awaiting requested review from pcuzner

@jdurgin jdurgin Awaiting requested review from jdurgin

@rkachach rkachach Awaiting requested review from rkachach

@epuertat epuertat Awaiting requested review from epuertat

@Sarthak0702 Sarthak0702 Awaiting requested review from Sarthak0702 Sarthak0702 was automatically assigned from ceph/dashboard

@cbodley cbodley Awaiting requested review from cbodley

@adk3798 adk3798 Awaiting requested review from adk3798

Assignees
No one assigned
Labels
bluestore build/ops ceph-volume cephadm cephfs Ceph File System CI Continuous Integration common core crimson dashboard documentation mgr mon monitoring orchestrator performance pybind rbd rgw rook tests
Projects
Ceph-Dashboard
Archived in project
Dashboard
  
Done
Milestone
No milestone
9 participants
@avanthakkar @mattbenjamin @yuvalif @pereman2 @cbodley @epuertat @adk3798 @rzarzynski @nizamial09