[bitnami/thanos] cache configs not properly loaded on store gateway

[fad3t]

Name and Version

bitnami/thanos 15.1.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Upgraded Thanos from v12 to v15.

Are you using any custom parameters or values?

  [..]
  storegateway:
    enabled: true
    logLevel: "warn"
    logFormat: "json"
    args:
    - store
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --data-dir=/data
    - --objstore.config-file=/conf/objstore.yml
    - --store.caching-bucket.config-file=/conf/cache/bucket-cache.yml
  indexCacheConfig: |
    type: "REDIS"
    config:
      addr: "prometheus-redis-master.prometheus.svc.cluster.local:6379"
      db: 0
      dial_timeout: "8s"
      read_timeout: "6s"
      write_timeout: "6s"
      cache_size: "1GB"
  bucketCacheConfig: |
    type: "REDIS"
    config:
      addr: "prometheus-redis-master.prometheus.svc.cluster.local:6379"
      db: 1
      dial_timeout: "8s"
      read_timeout: "6s"
      write_timeout: "6s"
      cache_size: "1GB"

What is the expected behavior?

Thanos should be running.

What do you see instead?

The storegateway keeps crashing with the following log messages:

ts=2024-04-19T08:52:04.344759326Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-04-19T08:52:04.345866254Z caller=main.go:135 level=error err="open /conf/bucket-cache.yml: no such file or directory\nloading file /conf/bucket-cache.yml for store.caching-bucket.config-file\ngithub.com/efficientgo/tools/extkingpin.(*PathOrContent).Content\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/pkg/mod/github.com/efficientgo/tools/extkingpin@v0.0.0-20220817170617-6c25e3b627dd/pathorcontent.go:79\nmain.runStore\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/store.go:304\nmain.registerStore.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/store.go:237\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:133\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nget caching bucket configuration\nmain.runStore\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/store.go:306\nmain.registerStore.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/store.go:237\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:133\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\npreparing store command failed\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:135\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650"

Additional information

The prometheus-thanos-objstore-secret used to contain 3 values: the indexCacheConfig, the bucketCacheConfig and the objStoreConfig. Only the last item remains, the other 2 moved into a configMap, but I think its configuration is wrong: it's only created when storegateway.config is set (not my case) but also it b64-encodes the value, which is probably a leftover from when it was stored in a secret..?

[carrodher]

Thank you for opening this issue and submitting the associated Pull Request. Our team will review and provide feedback. Once the PR is merged, the issue will automatically close.

Your contribution is greatly appreciated!