Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TraceQL Metrics] Unexpected error when rating by rootName/rootServiceName #3429

Closed
joe-elliott opened this issue Feb 26, 2024 · 1 comment · Fixed by grafana/grafana#83619 or #3440

Comments

@joe-elliott
Copy link
Member

image

Reproduced with the same images as here:
#3428

If I include a filter in the query it works:

image

@mdisibio
Copy link
Contributor

Confirmed this is a Tempo-side bug: it's an issue with the series where rootName=empty string "". I didn't realize it but when we set labels here it treats empty as missing and removes it. Need to think a bit about how to solve this - how should we represent blanks? The concept doesn't exist in metrics.

Ran into a related scenario with nils which is another way Tempo differs. {} | rate() by(.foo) doesn't only meter spans with a .foo. It counts all spans and includes a <nil> series. Perhaps we should do similar and return <empty> ?

To muddy the waters a bit more, this is only really necessary when all labels are empty or nil. See the comment here describing the ways we can drop nil labels and produce good "prometheus-like" series.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants