You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Traces are often downsampled before they are ingested by Tempo. Since the metrics-generator emits metrics based upon ingest volume, these metrics will also be downsampled.
For example, traces_spanmetrics_calls_total emits how often a specific operation happend. If traces are downsampled by constant sampling percentage 50%, the metric will be 50% lower as well than reality. This is likely misleading as you have to highly aware of downsampling happening in the trace pipeline.
With more involved down sampling setups (tail sampling, per operation sampling) it becomes impossible to calculate the actual value.
Describe the solution you'd like
The metrics-generator should pick up the sampling rate used and adjust the emitted metrics. Ideally, it extracts the sampling rate from metadata attached to ingested spans.
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered:
This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.
Is your feature request related to a problem? Please describe.
Traces are often downsampled before they are ingested by Tempo. Since the metrics-generator emits metrics based upon ingest volume, these metrics will also be downsampled.
For example,
traces_spanmetrics_calls_total
emits how often a specific operation happend. If traces are downsampled by constant sampling percentage 50%, the metric will be 50% lower as well than reality. This is likely misleading as you have to highly aware of downsampling happening in the trace pipeline.With more involved down sampling setups (tail sampling, per operation sampling) it becomes impossible to calculate the actual value.
Describe the solution you'd like
The metrics-generator should pick up the sampling rate used and adjust the emitted metrics. Ideally, it extracts the sampling rate from metadata attached to ingested spans.
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: