Downsampling stuck at certains blocks

[ggabijaa]

Compactor in K8s finishes compaction and gets stuck at the downsampling stage and pod get restarted every few hours.
Here are some logs before compactor pod get restared.

Firstly it spams this, for all the chunks inside the block:

{"caller":"block.go:68","err":"opening file: open /data/downsample/01HWxxx17Q/chunks/000303: no such file or directory","level":"info","msg":"failed to calculate hash when downloading; re-downloading","relPath":"chunks/000303","ts":"2024-05-06T11:05:09.174209453Z"}
...

And after following logs pod get restarted:

{"caller":"intrumentation.go:67","level":"warn","msg":"changing probe status","reason":"error executing compaction: first pass of downsampling failed: downsampling to 5 min: download block 01HWxxx17Q: context canceled","status":"not-ready","ts":"2024-05-06T11:14:36.888076299Z"}
{"caller":"http.go:91","component":"compact","err":"error executing compaction: first pass of downsampling failed: downsampling to 5 min: download block 01HWxxx17Q: context canceled","level":"info","msg":"internal server is shutting down","service":"http/server","ts":"2024-05-06T11:14:36.888136051Z"}
{"caller":"http.go:110","component":"compact","err":"error executing compaction: first pass of downsampling failed: downsampling to 5 min: download block 01HWxxx17Q: context canceled","level":"info","msg":"internal server is shutdown gracefully","service":"http/server","ts":"2024-05-06T11:14:36.888662889Z"}
{"caller":"intrumentation.go:81","level":"info","msg":"changing probe status","reason":"error executing compaction: first pass of downsampling failed: downsampling to 5 min: download block 01HWxxx17Q: context canceled","status":"not-healthy","ts":"2024-05-06T11:14:36.888725155Z"}
{"caller":"runutil.go:100","err":"BaseFetcher: iter bucket: context canceled","level":"error","msg":"function failed. Retrying in next tick","ts":"2024-05-06T11:14:36.888902277Z"}
{"caller":"compact.go:568","err":"syncing metas: BaseFetcher: iter bucket: context canceled","level":"error","msg":"retriable error","ts":"2024-05-06T11:14:36.888931582Z"}
{"caller":"compact.go:597","err":"BaseFetcher: iter bucket: Get \"http://mys3storage.com/bucket/?delimiter=&encoding-type=url&prefix=\": context canceled","level":"error","msg":"retriable error","ts":"2024-05-06T11:14:36.889199835Z"}
{"caller":"main.go:161","err":"downsampling to 5 min: download block 01HWxxx17Q: context canceled\nfirst pass of downsampling failed\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:445\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:481\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:480\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nerror executing compaction\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:508\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:480\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650","level":"error","ts":"2024-05-06T11:14:36.889289223Z"}

When checking inside the pod /data/downsample folder - i can see block 01HWxxx17Q and all it's chunks.
Compactor is not halted.

What could be the issue with the compactor?

[GiedriusS]

Could it be that someone is shutting down the compactor? If your blocks are huge then it takes a long time, unfortunately. I always check the goroutines dump in /debug/pprof to see if it is not stuck.

[ggabijaa]

nope, the only time it's shut down is when it gets restarted like this. At other cases it gets restarted when downloading blocks. Right now i see blocks are being compacted. So there was some huge blocks that made it very slow

[BouchaaraAdil]

the same here, the compactor is stuck downloading some large blocks fails and start again downloading
kinda aloop.

my case is thanos fails to download blocks with level 3: 2d in order to compact/downsample them to 14d

can thanos continue downloading blocks after being failed downloading one of the list start from a previous state