Introduce controleval_metrics.py tool to generate metrics in Prometheus format#11040
Conversation
marcusburghardt
added
enhancement
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
Mab879
left a comment
Mab879
left a comment
There was a problem hiding this comment.
Thanks for the PR!
I do have a couple comments.
marcusburghardt
force-pushed
the
controleval_prometheus
branch
from
3d91259 to
3141909
Compare
|
The CI failures look legit. Seems that we are not using requirements.txt in some of the tests. |
Mab879
left a comment
Mab879
left a comment
There was a problem hiding this comment.
The script looks good, but the CI failures need attention.
|
/packit build |
mildas
left a comment
mildas
left a comment
There was a problem hiding this comment.
As prometheus_client is not a project requirement and we don't want it to be, the problematic ctests should be run only when the prometheus_client is installed on the system.
In the TF tests, it makes sense to install the dependency and test it. However, for customers and contributors who don't want it, in the current state it will cause failing ctests and they will be forced to install the dependency on their system just to make it pass.
|
/packit retest-failed |
|
Discussing with @mildas , we concluded it would be better to import Update: It is done by the last two commits. |
Mab879
left a comment
Mab879
left a comment
There was a problem hiding this comment.
The script looks good.
Please include a conditional ctest that will run the Prometheus format if prometheus_client is installed.
@Mab879 I wasn't planning to include tests for this script for now. Once this PR is merged I will propose a new PR to integrate it with GitHub Pages and then implement all necessary tests at once. If you have objections I can include the test in this PR tomorrow. |
That will work for me. @mildas Is everything okay on this PR? |
Moved and split code in specific functions to make them more usable in other functions or subcommands.
This commit creates the subcmd, its options and a very basic function which will evolve in the next commits.
The goals is to collect prometheus metrics from all control files used by the informed products. Therefore, the function iterates with the list of products, locate all profiles files and search for controls used in profile files. If a control is found, a set is incremented to further usage. Once all relevant control files are identified, stats are collected from each specific level in a policy. This commit cares of collectiong the necessary information. Next commits will implement the Prometheus format.
Once the data necessary for the metrics is collected, a Prometheus registr is created and populated with metrics for all policies used by informed products. Finally the registry is exported in a txt file in Prometheus format in order to be easily consumed by Prometheus jobs.
Shorter lines with better readability in functions.
The initial behaviour was expecting to save the metrics in a file by using the write_to_textfile function from prometheus_client. There was no specific function to redirect the output to stdout. However, it was found another funcion which returns the metrics encoded in utf-8. This function was used to make the stdout as default.
This script depends on controleval.py but is intended to generate metrics to be consumed by other tools. In this initial version, it is generating metrics for Prometheus but other tools can be included in the future, if necessary. These functions were initially included in controleval.py but the drawnback was that prometheus_client module was required by the script even when calling the stats sub-command and making the module conditional would impact the readbility.
These functions were moved to controleval_metrics.py.
If the prometheus-client module is present, the controleval_metrics.py will be tested.
marcusburghardt
force-pushed
the
controleval_prometheus
branch
from
9698ec9 to
98ad1a0
Compare
marcusburghardt
changed the title
|
Code Climate has analyzed commit 98ad1a0 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 53.8% (0.0% change). View more on Code Climate. |
mildas
left a comment
mildas
left a comment
There was a problem hiding this comment.
Checked the code and performed testing, LGTM
4 participants
Description:
In the previous state of
controleval.pywe can easily collect stats about control files.This is pretty useful to check the current state.
However, it would be also very useful to have metrics showing the history of policies so we can easily see how it is progressing, maintained, etc. To achieve this, the information should be periodically collected and ideally stored in a time series database.
Prometheus is a pretty popular and straightforward tool to collect metrics in a time series database which can be easily integrated with Grafana to show nice dashboards, like it is done in the CommunityMon project.
This PR extends the
controleval.pytool to use the already collected metrics and export them in the Prometheus format.Rationale:
Review Hints:
Please, check the commits and their respective commit messages as they are ordered.
To see how the metrics are exported, here is an example for policies used by rhel9 and rhel8: