Replace metricsmap-bpf-prom-sync with Prometheus Collector pattern #27370

carnerito · 2023-08-09T07:51:12Z

Previously, the 'metricsmap-bpf-prom-sync' controller was responsible for periodically collecting cilium_datapath drop
and forward metrics.
This PR introduces the metricsmapCollector, which implements the Prometheus Collector interface within the
metricsmap package. As a result, the aforementioned controller has been removed.
The metricsmapCollector is registered within the global Prometheus registry during the initialization of metricsmap.
The metrics.Register function has been modified to propagate errors from the registry.Register function instead of
simply overriding it with nil.
The metricsmapCollector comprises two metrics maps: forwardedMetricsMap and droppedMetricsMap.
These maps are populated within a callback function passed to the IterateWithCallback function.
This approach serves two primary purposes:

Separation of Map Iteration and Metric Update: By separating the iteration over the BPF map and the updating of
Prometheus metrics, the implementation ensures that no partial metrics are exposed in case of map iteration failure.
Normalization of exposed Metrics: Unlike the statement in the bpf/lib/metrics.h comments, which suggests exposing
only one reason label for forwarded metrics, the eBPF code exposes multiple reasons. Through testing, it was found
that reasons 0 and 3 were exposed which triggered this error in prometheus client:
It occurs "collected metric %s %s was collected before with the same name and label values" error in the lastest version prometheus/client_golang#242

Furthermore, cilium command has an indirect dependency on metricsmap package. To prevent metrics map initialization
from happening each time cilium command is executed, metricsmap is converted to hive Cell and injected in cilium daemon
Infrastructure module. For details see this comment: #27370 (comment).

Fixes: #27058

Signed-off-by: Boris Petrovic carnerito.b@gmail.com

maintainer-s-little-helper · 2023-08-09T08:18:50Z

Commit 971a40e9b4830fd094a9fb641dd97b0832be9a8e does not contain "Signed-off-by".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

maintainer-s-little-helper · 2023-08-09T09:29:08Z

Commit 971a40e9b4830fd094a9fb641dd97b0832be9a8e does not contain "Signed-off-by".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

dylandreimerink · 2023-08-14T14:51:24Z

Hi @carnerito, thank you for this contribution. Could you sign off on your commit? https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

carnerito · 2023-08-14T15:09:35Z

@dylandreimerink Can I squash all commits and force-push?

Update
Ignore this, I didn't know that I can amend specific commit not just latest one.

zacharysarah

@carnerito Thanks for the updates! Some minor stylistic changes are required, with strong encouragement to fix some of the inherited language touched by this PR.

Documentation/bpf/toolchain.rst

Documentation/community/governance/commit_access.rst

Documentation/community/roadmap.rst

Documentation/configuration/argocd-issues.rst

Documentation/glossary.rst

Documentation/installation/cni-chaining-generic-veth.rst

pkg/maps/metricsmap/metricsmap.go

Previously, the 'metricsmap-bpf-prom-sync' controller was responsible for periodically collecting cilium_datapath drop and forward metrics. This commit introduces the metricsmapCollector, which implements the Prometheus Collector interface within the metricsmap package. As a result, the aforementioned controller has been removed. The metricsmapCollector is registered within the global Prometheus registry during the initialization of metricsmap. The metrics.Register function has been modified to propagate errors from the registry.Register function instead of simply overriding it with nil. The metricsmapCollector comprises two metrics maps: forwardedMetricsMap and droppedMetricsMap. These maps are populated within a callback function passed to the IterateWithCallback function. This approach serves two primary purposes: 1. Separation of Map Iteration and Metric Update: By separating the iteration over the BPF map and the updating of Prometheus metrics, the implementation ensures that no partial metrics are exposed in case of map iteration failure. 2. Normalization of exposed Metrics: Unlike the statement in the bpf/lib/metrics.h comments, which suggests exposing only one reason label for forwarded metrics, the eBPF code exposes multiple reasons. Through testing, it was found that reasons 0 and 3 were exposed which triggered this error in prometheus client: prometheus/client_golang#242 Furthermore, cilium command has indirect dependency on metricsmap package. To prevent metrics map initialization to happen each time cilium command is executed, metricsmap is converted to hive Cell and injected in cilium daemon Infrastructure module. For details see this comment: cilium#27370 (comment). Fixes: cilium#27058 Signed-off-by: Boris Petrovic <carnerito.b@gmail.com>

christarazi · 2023-10-30T17:53:57Z

I've rebased the PR to re-run the tests.

christarazi · 2023-10-30T17:54:01Z

/test

Previously, the 'metricsmap-bpf-prom-sync' controller was responsible for periodically collecting cilium_datapath drop and forward metrics. This commit introduces the metricsmapCollector, which implements the Prometheus Collector interface within the metricsmap package. As a result, the aforementioned controller has been removed. The metricsmapCollector is registered within the global Prometheus registry during the initialization of metricsmap. The metrics.Register function has been modified to propagate errors from the registry.Register function instead of simply overriding it with nil. The metricsmapCollector comprises two metrics maps: forwardedMetricsMap and droppedMetricsMap. These maps are populated within a callback function passed to the IterateWithCallback function. This approach serves two primary purposes: 1. Separation of Map Iteration and Metric Update: By separating the iteration over the BPF map and the updating of Prometheus metrics, the implementation ensures that no partial metrics are exposed in case of map iteration failure. 2. Normalization of exposed Metrics: Unlike the statement in the bpf/lib/metrics.h comments, which suggests exposing only one reason label for forwarded metrics, the eBPF code exposes multiple reasons. Through testing, it was found that reasons 0 and 3 were exposed which triggered this error in prometheus client: prometheus/client_golang#242 Furthermore, cilium command has indirect dependency on metricsmap package. To prevent metrics map initialization to happen each time cilium command is executed, metricsmap is converted to hive Cell and injected in cilium daemon Infrastructure module. For details see this comment: cilium#27370 (comment). Fixes: cilium#27058 Signed-off-by: Boris Petrovic <carnerito.b@gmail.com>

aanm · 2023-10-31T08:56:50Z

/test

carnerito · 2023-10-31T11:44:30Z

@aanm Cilium build was failing because of unused import. It's fixed now, could you please rerun tests?

aanm · 2023-10-31T11:50:59Z

@carnerito Thanks, can you add the last commit as part of 8e741df? We try that each commit is in a "buildable state"

carnerito · 2023-10-31T11:53:36Z

@aanm Just to be sure, do you want me to squash all commits and force-push?

aanm · 2023-10-31T12:21:29Z

@aanm Just to be sure, do you want me to squash all commits and force-push?

@carnerito I didn't look at all commits but now that you mentioned it, yes please squash all commits together and force-push. Thank you!

Previously, the 'metricsmap-bpf-prom-sync' controller was responsible for periodically collecting cilium_datapath drop and forward metrics. This commit introduces the metricsmapCollector, which implements the Prometheus Collector interface within the metricsmap package. As a result, the aforementioned controller has been removed. The metricsmapCollector is registered within the global Prometheus registry during the initialization of metricsmap. The metrics.Register function has been modified to propagate errors from the registry.Register function instead of simply overriding it with nil. The metricsmapCollector comprises two metrics maps: forwardedMetricsMap and droppedMetricsMap. These maps are populated within a callback function passed to the IterateWithCallback function. This approach serves two primary purposes: 1. Separation of Map Iteration and Metric Update: By separating the iteration over the BPF map and the updating of Prometheus metrics, the implementation ensures that no partial metrics are exposed in case of map iteration failure. 2. Normalization of exposed Metrics: Unlike the statement in the bpf/lib/metrics.h comments, which suggests exposing only one reason label for forwarded metrics, the eBPF code exposes multiple reasons. Through testing, it was found that reasons 0 and 3 were exposed which triggered this error in prometheus client: prometheus/client_golang#242 Furthermore, cilium command has indirect dependency on metricsmap package. To prevent metrics map initialization to happen each time cilium command is executed, metricsmap is converted to hive Cell and injected in cilium daemon Infrastructure module. For details see this comment: cilium#27370 (comment). Fixes: cilium#27058 Signed-off-by: Boris Petrovic <carnerito.b@gmail.com>

carnerito · 2023-10-31T12:26:11Z

@aanm Done

aanm · 2023-10-31T13:08:07Z

/test

derailed

@carnerito Thank you for this PR and updates Boris!!

joestringer · 2023-11-01T23:03:52Z

Required tests are passing. Seems to be mainly waiting on @cilium/sig-foundations review due to the cells.go change.

joestringer · 2023-11-01T23:04:39Z

Removing my review, I was only involved here as triager trying to keep PRs moving.

pchaigno · 2023-12-05T11:16:02Z

Is it me or this pull request involuntary renamed a bunch of metrics? For example, what used to be called cilium_drop_count_total is now called cilium_datapath_drop_count_total. Isn't that a breaking change?

carnerito · 2023-12-05T11:18:18Z

@pchaigno Yes, it's fixed in #29226

pchaigno · 2023-12-05T11:20:15Z

Ah, perfect! Thanks! The fix simply didn't reach my PR yet 🙂

carnerito requested review from a team as code owners August 9, 2023 07:51

carnerito requested review from derailed and dylandreimerink August 9, 2023 07:51

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Aug 9, 2023

github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Aug 9, 2023

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Aug 9, 2023

dylandreimerink approved these changes Aug 14, 2023

View reviewed changes

carnerito force-pushed the metricsmap-collector branch from 7de9995 to 8fb7280 Compare August 14, 2023 19:13

carnerito requested review from a team as code owners August 14, 2023 19:13

carnerito requested review from aditighag, zacharysarah, joamaki and kaworu August 14, 2023 19:13

maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Aug 14, 2023

zacharysarah requested changes Aug 14, 2023

View reviewed changes

carnerito force-pushed the metricsmap-collector branch 2 times, most recently from 42c4cde to d1988ae Compare August 15, 2023 06:27

dylandreimerink requested review from zacharysarah and removed request for a team August 16, 2023 08:47

aanm requested review from joestringer and christarazi October 30, 2023 08:31

aanm removed the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Oct 30, 2023

christarazi force-pushed the metricsmap-collector branch from 1a47832 to 42e3b7b Compare October 30, 2023 17:53

aanm force-pushed the metricsmap-collector branch from 42e3b7b to ec983f9 Compare October 31, 2023 08:56

carnerito force-pushed the metricsmap-collector branch from b5f07e9 to 6711c88 Compare October 31, 2023 12:25

derailed approved these changes Oct 31, 2023

View reviewed changes

joestringer removed their request for review November 1, 2023 23:04

christarazi approved these changes Nov 2, 2023

View reviewed changes

dylandreimerink approved these changes Nov 2, 2023

View reviewed changes

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 2, 2023

aanm merged commit f8e9472 into cilium:main Nov 2, 2023
57 of 62 checks passed

pchaigno mentioned this pull request Dec 5, 2023

Fix Cilium Datapath Prometheus metric names #29226

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace metricsmap-bpf-prom-sync with Prometheus Collector pattern #27370

Replace metricsmap-bpf-prom-sync with Prometheus Collector pattern #27370

carnerito commented Aug 9, 2023 •

edited

Loading

maintainer-s-little-helper bot commented Aug 9, 2023

maintainer-s-little-helper bot commented Aug 9, 2023

dylandreimerink commented Aug 14, 2023

carnerito commented Aug 14, 2023 •

edited

Loading

zacharysarah left a comment

christarazi commented Oct 30, 2023

christarazi commented Oct 30, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

derailed left a comment

joestringer commented Nov 1, 2023

joestringer commented Nov 1, 2023

pchaigno commented Dec 5, 2023

carnerito commented Dec 5, 2023

pchaigno commented Dec 5, 2023

Replace metricsmap-bpf-prom-sync with Prometheus Collector pattern #27370

Replace metricsmap-bpf-prom-sync with Prometheus Collector pattern #27370

Conversation

carnerito commented Aug 9, 2023 • edited Loading

maintainer-s-little-helper bot commented Aug 9, 2023

maintainer-s-little-helper bot commented Aug 9, 2023

dylandreimerink commented Aug 14, 2023

carnerito commented Aug 14, 2023 • edited Loading

zacharysarah left a comment

Choose a reason for hiding this comment

christarazi commented Oct 30, 2023

christarazi commented Oct 30, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

carnerito commented Oct 31, 2023

aanm commented Oct 31, 2023

derailed left a comment

Choose a reason for hiding this comment

joestringer commented Nov 1, 2023

joestringer commented Nov 1, 2023

pchaigno commented Dec 5, 2023

carnerito commented Dec 5, 2023

pchaigno commented Dec 5, 2023

carnerito commented Aug 9, 2023 •

edited

Loading

carnerito commented Aug 14, 2023 •

edited

Loading