Thanos Receive container not starting w/ v0.32.x

[grimz-ly]

Object Storage Provider:

minIO

What happened:

Changed version of container to v0.32.0 from v0.31.0 and the receive container no longer starts with following output

ts=2023-10-12T21:38:00.31138432Z caller=factory.go:53 level=info name=receive component=receive msg="loading bucket configuration"
ts=2023-10-12T21:38:00.311810444Z caller=receive.go:750 level=info name=receive component=receive msg="default tenant data dir already present, not attempting to migrate storage"
ts=2023-10-12T21:38:00.312061938Z caller=receive.go:277 level=debug name=receive component=receive msg="setting up TSDB"
ts=2023-10-12T21:38:00.312107869Z caller=receive.go:599 level=debug name=receive component=receive msg="removing storage lock files if any"
ts=2023-10-12T21:38:00.312209623Z caller=main.go:135 level=error name=receive err="open /data/remote-write-receive-data: permission denied\nremove storage lock files\nmain.startTSDBAndUpload\n\t/app/cmd/thanos/receive.go:601\nmain.runReceive\n\t/app/cmd/thanos/receive.go:279\nmain.registerReceive.func1\n\t/app/cmd/thanos/receive.go:99\nmain.main\n\t/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\npreparing receive command failed\nmain.main\n\t/app/cmd/thanos/main.go:135\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

What you expected to happen:

receive container to start and receive incoming metrics

Anything else we need to know:

changing image version back to v0.31.0 allows container to start as expected.

[GiedriusS]

It's not running as root anymore for security purposes so you'll have to make adjustments: #6107. I think this is unfixable from our end.

[grimz-ly]

Thank you for this quick reply and I apologize for not noticing this in the release notes. It only seemed to affect the content in /data of the receive container in which the following fixed the permissions to allow 1001 user access.

setfacl -R -m u:1001:rwX receive

[GiedriusS]

👍 glad that you got it figured out, hopefully that command will be useful to others in the future 😄 closing the issue to keep the issue tracker tidy

[zawadaa]

That change kick me too. Prometheus in docker run as nobody so...
Any way, isn't that user thanos an overhead? Simple user nobody is in every linux distro.
Use that and Dockerfile will be simpler.