Tempo leaves behind data files for blocks with no metadata

[zalegrala]

Describe the bug

In some circumstances, a data.parquet file is the only object in a block path, which means this block shows up in the list, but the metadata is available. As of #2678 tempo now deletes the tenant index when the tenant is found to have zero blocks, but because these paths still show up, the tenant is not completely deleted which causes index failures to occur repeatedly for these tenants and results in additional calls to the backend that aren't helpful.

The index deletion I believe only uncovered this issue, which was dormant due to the index being left in place prior to #2678.

To Reproduce

Running r106 we see this in environments where a tenant index was deleted due to no blocks being found. My hunch is that this has something to do with unclean shutdown, but have no data to back this up.

Expected behavior
Block meta is either reconstructed from the data, or the data is removed.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: jsonnet

[mdisibio]

Adding that this situation will cause repeated errors like the following until all files are cleaned up: msg="failed to write tenant index" tenant=<id> err="storage: object doesn't exist". This is because it sees no more blocks and tries to delete the tenant index, but it's already been deleted. The delete call at https://github.com/grafana/tempo/blob/main/tempodb/backend/raw.go#L94 isn't handling the does not exist error gracefully.

In this situation it should not be considered a failure and propagate up. It's falsely triggering the TempoTenantIndexFailure. I think this should be an easy fix by ignoring backend.ErrDoesNotExist.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.