Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics-generator should take sampling rate into account #1515

Open
kvrhdn opened this issue Jun 27, 2022 · 1 comment
Open

Metrics-generator should take sampling rate into account #1515

kvrhdn opened this issue Jun 27, 2022 · 1 comment
Labels
component/metrics-generator keepalive Label to exempt Issues / PRs from stale workflow

Comments

@kvrhdn
Copy link
Member

kvrhdn commented Jun 27, 2022

Is your feature request related to a problem? Please describe.
Traces are often downsampled before they are ingested by Tempo. Since the metrics-generator emits metrics based upon ingest volume, these metrics will also be downsampled.

For example, traces_spanmetrics_calls_total emits how often a specific operation happend. If traces are downsampled by constant sampling percentage 50%, the metric will be 50% lower as well than reality. This is likely misleading as you have to highly aware of downsampling happening in the trace pipeline.

With more involved down sampling setups (tail sampling, per operation sampling) it becomes impossible to calculate the actual value.

Describe the solution you'd like
The metrics-generator should pick up the sampling rate used and adjust the emitted metrics. Ideally, it extracts the sampling rate from metadata attached to ingested spans.

Describe alternatives you've considered

Additional context

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

@github-actions github-actions bot added the stale Used for stale issues / PRs label Nov 14, 2022
@joe-elliott joe-elliott added keepalive Label to exempt Issues / PRs from stale workflow and removed stale Used for stale issues / PRs labels Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/metrics-generator keepalive Label to exempt Issues / PRs from stale workflow
Projects
None yet
Development

No branches or pull requests

2 participants