Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing services_events_total metric #26511

Closed
1 task done
whites11 opened this issue Jun 27, 2023 · 3 comments · Fixed by #26719
Closed
1 task done

Missing services_events_total metric #26511

whites11 opened this issue Jun 27, 2023 · 3 comments · Fixed by #26719
Assignees
@christarazi
Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. sig/agent Cilium agent related.

Comments

@whites11
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

I deployed cilium 1.13.4 on kubernetes.
I am scraping both the operator and the agents with prometheus.
I deployed the default grafana dashboard.

I don't have data for one chart, turns out the cilium_services_events_total is not exposed by either components.

Looking at the code it looks like the metric should always be exposed

Cilium Version

1.13.4

Kernel Version

5.15.111-flatcar

Kubernetes Version

v1.24.15

Sysdump

No response

Relevant log output

# curl -s localhost:9962/metrics|grep services_events_total
#
# curl -s localhost:9963/metrics|grep services_events_total
#



### Anything else?

it's not about the metric being without data, the description is not there in the exporter outputs.

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct
@whites11 whites11 added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels Jun 27, 2023
@ldelossa
Copy link
Contributor

@whites11 can you please provide a sysdump when you're experiencing this issue.

@ldelossa ldelossa added sig/agent Cilium agent related. and removed needs/triage This issue requires triaging to establish severity and next steps. labels Jun 28, 2023
@whites11
Copy link
Author

unfortunately, sysdump is too big for github.
you can download it from here for https://drive.google.com/file/d/1dEnnqBT2zRzhXacYMzcpkwTdY04dk5mb/view?usp=sharing for 24h

@christarazi
Copy link
Member

I tested latest Cilium v1.12 and v1.11. Both of them also are missing the metric. Seems like this has been a bug for a long time.

@christarazi christarazi mentioned this issue Jul 7, 2023
Merged
@christarazi christarazi self-assigned this Jul 7, 2023
christarazi added a commit to christarazi/cilium that referenced this issue Jul 20, 2023
@christarazi
services: Fix metric from not publishing
722fcea 
When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: cilium#26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
dylandreimerink pushed a commit that referenced this issue Jul 21, 2023
@christarazi @dylandreimerink
services: Fix metric from not publishing
b86dab8 
When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: #26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
nbusseneau pushed a commit that referenced this issue Jul 24, 2023
@christarazi @nbusseneau
services: Fix metric from not publishing
e209fa1 
[ upstream commit b86dab8 ]

When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: #26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
nbusseneau pushed a commit that referenced this issue Jul 24, 2023
@christarazi @nbusseneau
services: Fix metric from not publishing
d55c513 
[ upstream commit b86dab8 ]

When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: #26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
aanm pushed a commit that referenced this issue Jul 25, 2023
@christarazi @aanm
services: Fix metric from not publishing
82648d2 
[ upstream commit b86dab8 ]

When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: #26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
youngnick pushed a commit that referenced this issue Jul 26, 2023
@christarazi @youngnick
services: Fix metric from not publishing
37f7ced 
[ upstream commit b86dab8 ]

When the metric variable is defined as a global variable (within the
`var` scope at the package level), then it will be instantiated as a
NoOp metric. Once the metrics package is initialized, then all the
metrics variables will transition from NoOp metrics to a real metric
type. This problem occurred because the global variables instantiation
happened before the metrics package initialization.

This commit fixes it by using the metrics variable after the metrics
package has been initialized. We can assume it's been initialized when
the code executed is production ("live") code.

Fixes: #26511
Fixes: 978b27c ("Metrics: Add services metrics")
Signed-off-by: Chris Tarazi <chris@isovalent.com>
Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@christarazi christarazi

Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. sig/agent Cilium agent related.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

services: Fix metric from not publishing christarazi/cilium
3 participants
@whites11 @christarazi @ldelossa