-
Notifications
You must be signed in to change notification settings - Fork 2k
Compactor is stuck, S3 storage constantly growing #7176
Replies: 2 comments · 7 replies
-
Can you check the compactor log or metric and see if there it is hanging? |
Beta Was this translation helpful? Give feedback.
All reactions
-
Compactor is not halted: Logs sometimes show these errors, but don't think they are related:
Any other metric i can check? |
Beta Was this translation helpful? Give feedback.
All reactions
-
Maybe you could check through |
Beta Was this translation helpful? Give feedback.
All reactions
-
Maybe you could upload the whole output? The most important parts are missing from the screenshot |
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes, sure. here it is:
|
Beta Was this translation helpful? Give feedback.
All reactions
-
Seems like it is not stuck as it is busy downloading files and decoding data. Maybe bumping |
Beta Was this translation helpful? Give feedback.
All reactions
-
I can try doing that, are there any recommendations for setting it ? |
Beta Was this translation helpful? Give feedback.
All reactions
-
tried bumping Another thing noticed is that logs are showing Issue is that blocks are never deleted. What could be the reason for that? S3 has around 150 blocks marked for deletion, that seems to stay there indefinitely |
Beta Was this translation helpful? Give feedback.
-
Hello there,
I'm encountering an issue with the Thanos compactor that appears to be stuck, leading to continuous growth in S3 storage.
Bit of context:
Several months ago, we did a migration from one S3 bucket to another. This was achieved using Thanos tools (
bucket tools replicate
) . During this time the compactor was turned off, and turned on after fully switching buckets.After the migration, we noticed that the compactor wasn't completing its compactions and failed to delete blocks marked for deletion. To address this, we took action, which seemed to alleviate the issue temporarily (storage decreased by 40 TB, returning to previous levels, and the todo compactions metric began to shrink, eventually reaching 0).
What we did was :
compact.concurrency
from 1 to 2However, we're now facing the same challenges once more. The todo compaction/downsampling metrics aren't decreasing, and S3 storage continues to grow. Compactor is not halted.
Here's some of our compactor configuration (Running in K8s):
Any insights or assistance you can provide would be greatly appreciated. Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions