New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

luminous: mgr: Sync the prometheus module #23216

Merged
merged 9 commits into from Sep 12, 2018

Conversation

Projects
None yet
5 participants
@b-ranto
Contributor

b-ranto commented Jul 24, 2018

This back-ports the latest PR #22732 that fixed the way we expose the metrics. It also contains a patch that fixes ceph-mgr restarts. The patch is known to expose a deficiency in the ceph-mgr error handling in luminous and can make the rados:mgr tests fail randomly.

b-ranto added some commits Apr 30, 2018

prometheus: Fix prometheus shutdown/restart
The prometheus serve method uses block which it should not as it can
cause several problems when shutting down (restarting) ceph-mgr. As a
result the restart of the ceph-mgr module will block until time-out if
the prometheus module is enabled. This fixes the problem by using a
simple simple shutdown_event instead of block() and stopping the server
afterwards.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit a662caa)
prometheus: Fix metric resets
This patch changes the way we reset metrics when collecting data in the
prometheus exporter module. With this patch, the exported metrics at any
point in time should align with a freshly started ceph-mgr module.

The patch also introduces locking mechanism to serialize the requests
and to avoid overwriting the metrics during multiple scrapes happenning
at the same time.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit f6c4db8)

Conflicts:
	src/pybind/mgr/prometheus/module.py: PG_STATES are not in the
	mgr_module.py file in luminous
prometheus: Make the cache timeout configurable
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 84241a2)

Conflicts:
	src/pybind/mgr/prometheus/module.py: There is no file_sd_config
prometheus: Use instance instead of inst variable
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 0869351)
prometheus: Optimize metrics formatting
It is faster to use join for join multiple strings than the regular
string concatenation since it does not create a new string on each
iteration.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit d395868)
prometheus: Remove the Metrics class
The class was introduced to account for the 'pending' mechanism but that
is no longer being used by the code so we no longer need it since it
serves a purpose of a simple dictionary, now.

This simplifies the code and brings a bit of performance improvements
since the Metric classes in the collect method are created only if they
did not exist before.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit c00dd55)

Conflicts:
	src/pybind/mgr/prometheus/module.py: Fixed pattern mismatch
prometheus: Format metrics in the collect function
This allows us to clear metrics upon formatting in the collect function.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 2583672)
prometheus: Reset the time the data was captured
This was forgotten during the previous patches.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 83565c3)
prometheus: Set the response header for cached response
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 9130ab4)

@b-ranto b-ranto requested a review from jan--f Jul 27, 2018

@jan--f

jan--f approved these changes Aug 17, 2018

@liewegas liewegas added this to the luminous milestone Sep 10, 2018

@liewegas liewegas changed the title from [luminous] Sync the prometheus module to luminous: Sync the prometheus module Sep 10, 2018

@yuriw

This comment has been minimized.

Contributor

yuriw commented Sep 11, 2018

@yuriw yuriw merged commit 38e4d25 into ceph:luminous Sep 12, 2018

4 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details

@b-ranto b-ranto deleted the b-ranto:wip-mgr-prom-bp-luminous branch Sep 12, 2018

@smithfarm smithfarm changed the title from luminous: Sync the prometheus module to luminous: mgr: Sync the prometheus module Oct 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment