compact:debug meta files are never deleted

[Shadi]

Thanos, Prometheus and Golang version used:

Thanos: quay.io/thanos/thanos:v0.15.0
Prometheus: quay.io/prometheus/prometheus:v2.22.0

Object Storage Provider: Dell ECS

What happened:
We have a preview thanos deployment that we use with our preview clusters, I was looking at one of the S3 buckets that the sidecars upload to and found out that the $(bucket)/debug folder has a lot of files, much more than the number of objects in the s3 bucket, the retention in the compactor for this bucket is set to 3 months for the 3 retention flags, but looking at some of the meta files they are for blocks that are more than 1 year old now, for example:

"ulid": "01DPJDWX5QWGN99VA2CJMRRAR1",
	"minTime": 1567159200000,
	"maxTime": 1567180800000,
	"stats": {
		"numSamples": 11395469,
		"numSeries": 16123,
		"numChunks": 95231
	},
	"compaction": {
		"level": 38571,
		"sources": [
			"01DKH9RY2JTXYRNGRZ2TW4H4SW",
			"01DKHGMN9DP36NF94JCX31KAA3",
			"01DKHQGCJ1AY7K7VQQX3AB316M"
		],
		"parents": [
			{
				"ulid": "01DPJDW8ETX65KVVXHANAG0KNQ",
				"minTime": 1567159200000,
				"maxTime": 1567180800000
			}
		]
	}

that is from Friday, August 30, 2019, which lead me to believe that the debug meta files might not get deleted, I looked a little bit at the code and I can see the files get created on upload:
https://github.com/thanos-io/thanos/blob/master/pkg/block/block.go#L94

but only the meta file in the block directory is deleted, DebugMetas not used anywhere in the Delete function inside that package:
https://github.com/thanos-io/thanos/blob/master/pkg/block/block.go#L167

What you expected to happen:
Debug meta file is deleted when the block is outside of retention window

How to reproduce it (as minimally and precisely as possible):
Set compactor retention to a value smaller than the current value, after the block object gets deleted the debug meta file will still be there.

[bwplotka]

That's correct, what do you propose? (:

This is really for debug, but was not proven to be very useful. Let's discuss and decide what to do 🤔

[Shadi]

I think the deletion call can mirror the Upload call and delete the file after deleting the meta file that is part of the block directory.

[GiedriusS]

Maybe let's bring this up on the next community call? (:

[Shadi]

Hi @GiedriusS, I don't know if the question is directed at me or someone else, I'm fine with that :) its a small issue and in worst case the files can be cleaned up offline by a separate tool if they become an issue.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[Shadi]

Still reproducible

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[jphx]

I've observed this problem also. A fix would be appreciated.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[GiedriusS]

We don't upload them anymore hence closing. #5110