[Search] Caching issue with searching last traces - no new traces are displayed

[MurzNN]

Describe the bug
I have a problem that the search query for the last traces in Tempo by Grafana UI sometimes got cached or load-balanced, because I'm stopping to see new traces.

After several clicks on "Run query" I see the changing list of traces, but it contains old traces, without new ones, and seems there are 2 cached results, changing each other.

And in the same time I can search my new traces by trace_id, but they are still missing in the list of last traces!

Restarting the Tempo container helps to temporarily resolve the issue, and all last traces appear. But the problem returns after some time again.

To Reproduce
Steps to reproduce the behavior:

1. I have configured Grafana and Tempo locally with Docker, using the setup like this https://github.com/grafana/tempo/blob/main/example/docker-compose/local/docker-compose.yaml
2. Push some traces from your app to Tempo.
3. Open in Grafana UI Explore > Tempo > Search > Run query
4. See the last traces.
5. Continue adding more traces, and tracking them by clicking "Run query" again and again.
6. After some time you will stop seeing fresh traces. Seems the query becomes cached, and even 2 or 3 cached versions, because after repeatedly clicking "Run query" you will start seeing two different cached lists of traces, but without new traces.

Expected behavior
The result of Search should always display all last traces.

Environment:

• Infrastructure: local
• Deployment tool: Docker Compose

Additional Context

[mapno]

Hi! Tempo does not necessarily return the most recent traces in the requested time range. It simply returns the first matching traces it finds. You’ve searched with no parameters so it will return 20 (default) random traces in the time period.

You can try by increasing the number of results with query_frontend.search.default_result_limit = 20 or by reducing the search window to the last X minutes.

[MurzNN]

Thank you for so surprising information!
So, is there any way to build a search query to get all recent traces, ordered by timestamp desc with limit?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

[anarcher]

I can't find the latest traces either, and am seeing random results. Is there any room for improvement here?

[joe-elliott]

I can't find the latest traces either, and am seeing random results. Is there any room for improvement here?

Yes. The original decision to return the first N results that matched was due to the poor efficiency of searching the old v2 backend. I do believe with the newer backend we can start moving toward deterministic results, but this is not on our roadmap at the moment.