Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thanos compactor shutting down do to timeout whilest trying to get no-compact-mark.json #7195

Open
TiferetJackson opened this issue Mar 7, 2024 · 0 comments

Comments

@TiferetJackson
Copy link

Thanos, Prometheus and Golang version used:
Thanos version: thanosio/thanos:main-2022-12-13-e58a3f2 from docker hub
Prometheus version: v2.31.1

Object Storage Provider: S3

What happened:
Thanos compactor shuts down do to internal server error.
This happens do to the compactor trying to get XXXXX/no-compact-mark.json and recieving an I/O timeout.
Upon investigating this issue, there really is no 'no-compact-mark.json' file in the specified bucket.

What you expected to happen:
Either for compactor to skip current bucket while compacting or for compactor to compact current bucket.
Or event just a way to tell compactor to ignore or skip errors and continue deleting and compacting instead of shutting down.

How to reproduce it (as minimally and precisely as possible):
I'm not quite sure why compactor determined that the specified bucket should not be compacted,
but in order to reproduce create a bucket with a no-compact-mark.json and than delete that file.

Full logs to relevant components:
err="syncing metas: filter blocks marked for no compaction: get file: XXXXXX/no-compact-mark.json: Get "S3 Route/bucket name/XXXXX/no-compact-mark.json" dial tcp : i/o timeout."

Anything else we need to know:
We upload the compactor as a deployment to the Openshift 4 environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant