Add filter to reject/skip ineligible spans

[andreivaloshyn]

Is your feature request related to a problem? Please describe.
We have more than 50% of ineligible spans which belong to Prometheus and because of that it's hard to search and also it takes extra resources to process unwanted spans

Describe the solution you'd like
It will be great to have the ability to filter spans by resources to be able to reject certain spans.
For example, spans which have

{
  "value": "Prometheus/2.39.1",
  "key": "user_agent"
}

should be rejected/skipped by the distributor.

Describe alternatives you've considered
We tried to implement Istio Envoy filtering but faced with strange behavior in this case.

Additional context

[joe-elliott]

We could consider this in Tempo, but this is likely already possible using Grafana Agent or the OpenTelemetry Collector. Have you looked at doing this in your pipeline?

[andreivaloshyn]

We could consider this in Tempo, but this is likely already possible using Grafana Agent or the OpenTelemetry Collector. Have you looked at doing this in your pipeline?

@joe-elliott thank you for your response.
Well, we as a team thought about this. OpenTelemetry Collector and Grafana Agent aren't good options for us as well because its additional component to deploy and maintain.
So it will be great to have this option as a built-in feature in Tempo.

[joe-elliott]

Currently you can configure a opentelemetry receivers on the distributor. I suppose one option would be to allow the configuration of additional processors there as well. I would also not be opposed to this feature in the distributors if someone is willing to work on it, but it would have to be quite well thought out and should be discussed with the team.

Overall, our recommendation would be to use a tracing pipeline to do this kind of work. However, if it were PR'ed to Tempo cleanly in a way that did not introduce latency or major future code maintenance we would consider it.

One final thought. Technically you could run an OTel Collector sidecar that did this work. Does this reduce your complexity concerns?

[andreivaloshyn]

Well, by sidecar you mean additional container for each pod we should collect traces from? We have thousands of pods and and cost saving focus this year, so my team will not be happy to add one more sidecar :(

UPD. hmm, as I see sidecar in this particular case is not a separate container

[joe-elliott]

No, I just mean a sidecar on Tempo. You could deploy the OTEL collector right next to Tempo that does buffering/batching and other manipulations. I understand your hesitance to add a new piece of software, but a trace pipeline is a very powerful tool. It can manipulate trace data, generate metrics, batch spans (which reduces Tempo TCO), etc.

[andreivaloshyn]

Well, I deployed OTel via k8s operator as a deployment. With this setup I can "cut" redundant spans.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.