New deployment against Vast S3 failing startup checks

[jwhitelbl]

Describe the bug

We're attempting a new mimir deployment against an S3 endpoint running on a Vast cluster (https://www.vastdata.com/). The various pods that interface with S3 buckets are failing with "context deadline exceeded" for various S3 endpoint location checks (location, sanity-check-at-startup).

To Reproduce

deploy mimir with a values.yaml similar to:

minio:
  enabled: false
global:
  # -- Definitions to set up nginx resolver
  dnsService: rke2-coredns-rke2-coredns
  dnsNamespace: kube-system
  
mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          endpoint: [redacted fqdn]
          secret_access_key: "${AWS_SECRET_ACCESS_KEY}" # This is a secret injected via an environment variable
          access_key_id: "${AWS_ACCESS_KEY_ID}" # This is a secret injected via an environment variable
    blocks_storage:
      s3:
        bucket_name: mimir-blocks
    alertmanager_storage:
      s3:
        bucket_name: mimir-alertmanager
    ruler_storage:
      s3:
        bucket_name: mimir-ruler
    
nginx:
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - host: [redacted fqdn]
        paths:
          - path: /
            pathType: Prefix
    tls:
      # empty, disabled

AWS_ACCESS_KEY_ID="[redacted]" AWS_SECRET_ACCESS_KEY="[redacted]" helm install mimir grafana/mimir-distributed -f ./mimir.yaml -n lgtm

logs from mimi-alertmanager pod:

ts=2024-07-17T17:54:56.841449751Z caller=main.go:225 level=info msg="Starting application" version="(version=2.13.0, branch=HEAD, revision=4775ec1)"
ts=2024-07-17T17:54:56.856834142Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2024-07-17T17:54:56.863044087Z caller=modules.go:925 level=info msg="Starting Alertmanager in classic mode"
ts=2024-07-17T17:54:56.863059687Z caller=featurecontrol.go:97 level=warn msg="Classic mode enabled"
ts=2024-07-17T17:54:56.86391108Z caller=memberlist_client.go:435 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-alertmanager-0-186c4544
ts=2024-07-17T17:54:56.864773512Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2024-07-17T17:54:56.864778632Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2024-07-17T17:54:56.864826781Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2024-07-17T17:54:56.865688804Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2024-07-17T17:54:56.865696094Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2024-07-17T17:54:56.866726053Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=12 to_join=8
ts=2024-07-17T17:54:56.874668674Z caller=memberlist_client.go:561 level=info msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=9.074978ms
ts=2024-07-17T17:54:56.874694523Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.lgtm.svc.cluster.local.:7946
ts=2024-07-17T17:54:56.884243282Z caller=memberlist_client.go:580 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=10 elapsed_time=9.541629ms
ts=2024-07-17T17:55:00.825299613Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.1.231:7946"
ts=2024-07-17T17:55:02.826984173Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.240:7946"
ts=2024-07-17T17:55:02.828006802Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.150:7946"
ts=2024-07-17T17:55:02.828096891Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.241:7946"
ts=2024-07-17T17:55:04.571571994Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.237:7946"
ts=2024-07-17T17:55:05.04206799Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.240:7946"
ts=2024-07-17T17:55:06.573391211Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.240:7946"
ts=2024-07-17T17:55:06.573768044Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.151:7946"
ts=2024-07-17T17:55:06.57445421Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.1.231:7946"
ts=2024-07-17T17:55:07.043034224Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.237:7946"
ts=2024-07-17T17:55:07.043691481Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.1.228:7946"
ts=2024-07-17T17:55:07.044148312Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.151:7946"
ts=2024-07-17T17:55:11.866756287Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.241:7946"
ts=2024-07-17T17:55:13.868684923Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.151:7946"
ts=2024-07-17T17:55:13.868724112Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.239:7946"
ts=2024-07-17T17:55:13.869541926Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.147:7946"
ts=2024-07-17T17:55:14.571681171Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.237:7946"
ts=2024-07-17T17:55:16.572995769Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.240:7946"
ts=2024-07-17T17:55:16.573792663Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.150:7946"
ts=2024-07-17T17:55:16.574529118Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.151:7946"
ts=2024-07-17T17:55:16.867062726Z caller=log.go:245 level=info msg="Suspect mimir-alertmanager-0-dcee8f62 has failed, no acks received"
ts=2024-07-17T17:55:25.467199005Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.147:7946"
ts=2024-07-17T17:55:27.327737189Z caller=log.go:245 level=info msg="Marking mimir-ingester-zone-b-0-82f8d366 as failed, suspect timeout reached (2 peer confirmations)"
ts=2024-07-17T17:55:27.468834196Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.0.150:7946"
ts=2024-07-17T17:55:27.468960444Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.2.240:7946"
ts=2024-07-17T17:55:27.469432184Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-alertmanager-0-dcee8f62' from=10.42.1.232:7946"
ts=2024-07-17T17:55:27.866348927Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"https://[redacted]/mimir-alertmanager/?location=\": context deadline exceeded"
ts=2024-07-17T17:55:28.086365981Z caller=log.go:245 level=info msg="Marking mimir-alertmanager-0-dcee8f62 as failed, suspect timeout reached (2 peer confirmations)"
ts=2024-07-17T17:55:46.866651163Z caller=log.go:245 level=info msg="Suspect mimir-ingester-zone-c-0-93a5628c has failed, no acks received"
ts=2024-07-17T17:56:00.578188681Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"https://[redacted]/mimir-alertmanager/?location=\": context deadline exceeded"
ts=2024-07-17T17:56:33.861880276Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"https://[redacted]/mimir-alertmanager/?location=\": context deadline exceeded"
ts=2024-07-17T17:57:09.085379198Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"https://[redacted]/mimir-alertmanager/?location=\": context deadline exceeded"
ts=2024-07-17T17:57:45.210654494Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"https://[redacted]/mimir-alertmanager/sanity-check-at-startup\": context deadline exceeded"

Expected behavior

The pods connect to the existing S3 buckets and enter a ready state

Environment

RKE2 v1.27.2+rke2r1
Vast release-5.0.0-sp10-1313052
S3 endpoint using https and a valid ca-signed certificate
helm v3.13.2
mimir-distributed-5.4.0 2.13.0

Additional Context

The Vast S3 endpoint should be as compatible as minio. No known caveats but we're absolutely willing to play around!

[aknuds1]

Converting to discussion, due to lack of evidence of Mimir bug.

[jwhitelbl]

resolved, answering for posterity. This was a case of the $AWS_SECRET_ACCESS_KEY and $AWS_ACCESS_KEY_ID env var not making it to the installation. Moving those variables to the helm install command via a --set 'mimir.structuredConfig.common.storage.s3.secret_access_key=[redacted]' and --set 'mimir.structuredConfig.common.storage.s3.access_key_id=[redacted]' resolved the issue.