Upgrading to Operator helmchart 4.5.8 resulted in bucket connection issue

[rufreakde]

We updated the helm chart of the operator to latest version. 4.5.8 and everything got applied and was green

We use the following CR: (no helm chart)

---
# Source: tenant/templates/tenant.yaml
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio1
  namespace: up-monitoring-db
  ## Optionally pass labels to be applied to the statefulset pods
  labels:
    app: minio
  annotations:
    argocd.argoproj.io/sync-wave: "100"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true


spec:
  image: quay.io/minio/minio:RELEASE.2023-01-12T02-06-16Z
  imagePullPolicy: IfNotPresent

  ## Secret with credentials to be used by MinIO Tenant.

  credsSecret:
    name: minio1-secret # not used anymore but can be maintained?

  configuration:
    name: minio1-configuration # generated by operator

  pools:

    - servers: 1
      volumesPerServer: 4
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          storageClassName: st1
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 125Gi
      resources:

        requests:
          cpu: 100m
          memory: 128Mi


  mountPath: /export
  subPath: /data
  requestAutoCert: true
  features:
    bucketDNS: false
  buckets:
    - name: chunks
      objectLock: false
    - name: admin
      objectLock: false
    - name: ruler
      objectLock: false
  users:
    - name: minio-user
  podManagementPolicy: Parallel

  prometheusOperator: false

Expected Behavior

After update loki connection should not be disrupted bucketDNS is false. (we took care of the deprecated s3.bucketDNS)

Current Behavior

level=info ts=2023-03-01T14:29:32.755340999Z caller=main.go:103 msg="Starting Loki" version="(version=2.7.0, branch=HEAD, revision=1b627d880)"
level=info ts=2023-03-01T14:29:32.756032015Z caller=server.go:323 http=:3100 grpc=:9095 msg="server listening on addresses"
level=info ts=2023-03-01T14:29:32.757103436Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=loki-read-0-9ed00415
level=warn ts=2023-03-01T14:29:32.757712418Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
level=warn ts=2023-03-01T14:29:32.757743699Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
level=info ts=2023-03-01T14:29:32.758627931Z caller=table_manager.go:262 msg="query readiness setup completed" duration=5.146µs distinct_users_len=0
level=info ts=2023-03-01T14:29:32.758698262Z caller=shipper.go:127 msg="starting index shipper in RO mode"
level=info ts=2023-03-01T14:29:32.758719352Z caller=shipper_index_client.go:78 msg="starting boltdb shipper in RO mode"
level=info ts=2023-03-01T14:29:32.759392537Z caller=memberlist_client.go:543 msg="memberlist fast-join starting" nodes_found=1 to_join=4
level=info ts=2023-03-01T14:29:32.762421941Z caller=memberlist_client.go:563 msg="memberlist fast-join finished" joined_nodes=2 elapsed_time=3.050445ms
level=info ts=2023-03-01T14:29:32.762450817Z caller=memberlist_client.go:576 msg="joining memberlist cluster" join_members=loki-memberlist
level=info ts=2023-03-01T14:29:32.764173125Z caller=memberlist_client.go:595 msg="joining memberlist cluster succeeded" reached_nodes=2 elapsed_time=1.722516ms
BadRequest: Bad Request
    status code: 400, request id: , host id: 
failed to get s3 object
github.com/grafana/loki/pkg/storage/chunk/client/aws.(*S3ObjectClient).GetObject
    /src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:389
github.com/grafana/loki/pkg/storage/stores/indexshipper/storage.prefixedObjectClient.GetObject
    /src/loki/pkg/storage/stores/indexshipper/storage/prefixed_object_client.go:25
github.com/grafana/loki/pkg/storage/stores/indexshipper/storage.(*indexStorageClient).GetFile
    /src/loki/pkg/storage/stores/indexshipper/storage/client.go:128
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.(*deleteRequestsTable).init.func1
    /src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:70
github.com/grafana/loki/pkg/storage/stores/indexshipper/storage.DownloadFileFromStorage
    /src/loki/pkg/storage/stores/indexshipper/storage/util.go:48
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.(*deleteRequestsTable).init
    /src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:68
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.newDeleteRequestsTable
    /src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:50
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.NewDeleteStore
    /src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_store.go:63
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.(*Compactor).initDeletes
    /src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:276
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.(*Compactor).init
    /src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:260
github.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.NewCompactor
    /src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:230
github.com/grafana/loki/pkg/loki.(*Loki).initCompactor
    /src/loki/pkg/loki/modules.go:985
github.com/grafana/dskit/modules.(*Manager).initModule
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
    /src/loki/pkg/loki/loki.go:360
main.main
    /src/loki/cmd/loki/main.go:105
runtime.main
    /usr/local/go/src/runtime/proc.go:250
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1594
error initialising module: compactor
github.com/grafana/dskit/modules.(*Manager).initModule
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
    /src/loki/pkg/loki/loki.go:360
main.main
    /src/loki/cmd/loki/main.go:105
runtime.main
    /usr/local/go/src/runtime/proc.go:250
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1594
Stream closed EOF for up-monitoring/loki-read-0 (read)

minio operator log:

I0301 13:32:49.063446       1 main.go:77] Starting MinIO Operator
I0301 13:32:49.454222       1 main.go:176] caBundle on CRD updated
I0301 13:32:49.454605       1 main-controller.go:243] Setting up event handlers
I0301 13:32:49.454724       1 leaderelection.go:248] attempting to acquire leader lease up-monitoring-db/minio-operator-lock...
I0301 13:32:49.461983       1 leaderelection.go:258] successfully acquired lease up-monitoring-db/minio-operator-lock
I0301 13:32:49.462156       1 main-controller.go:500] minio-operator-5b9955c79f-vw8gq: I am the leader, applying leader labels on myself
I0301 13:32:49.462269       1 main-controller.go:409] Waiting for API to start
I0301 13:32:49.462322       1 main-controller.go:390] Starting console TLS certificate setup
I0301 13:32:49.462366       1 main-controller.go:404] Console TLS is not enabled
I0301 13:32:49.462428       1 main-controller.go:381] Starting HTTP Upgrade Tenant Image server
I0301 13:32:49.510019       1 main-controller.go:352] Using Kubernetes CSR Version: v1
I0301 13:32:49.515910       1 main-controller.go:356] Starting HTTPS API server
I0301 13:32:49.515951       1 main-controller.go:412] Waiting for Upgrade Server to start
I0301 13:32:49.515977       1 main-controller.go:415] Waiting for Console TLS
I0301 13:32:49.515982       1 main-controller.go:419] Starting Tenant controller
I0301 13:32:49.515985       1 main-controller.go:422] Waiting for informer caches to sync
I0301 13:32:49.516003       1 main-controller.go:427] Starting workers
I0301 13:32:49.542628       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:32:49.542628       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:32:51.644377       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:32:54.168424       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
E0301 13:32:54.541212       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': Put "https://minio.up-monitoring-db.svc.cluster.local/minio/admin/v3/add-user?accessKey=minio-generated-user": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:32:54.541285       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: Put "https://minio.up-monitoring-db.svc.cluster.local/minio/admin/v3/add-user?accessKey=minio-generated-user": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:32:55.035685       1 http_handlers.go:180] MINIO_ARGS value is /export{0...3}/data
I0301 13:32:55.442886       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
E0301 13:33:34.581726       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:33:34.581815       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
E0301 13:34:54.630018       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:34:54.630099       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
I0301 13:35:49.554778       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority
E0301 13:36:14.674196       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:36:14.674282       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
E0301 13:37:34.719004       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:37:34.719001       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
I0301 13:38:49.552718       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority
I0301 13:38:54.762035       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
E0301 13:38:54.762069       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
E0301 13:40:14.802082       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:40:14.802085       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
E0301 13:41:34.842013       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': context deadline exceeded
I0301 13:41:34.842035       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210230654", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
I0301 13:41:49.552343       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority
I0301 13:42:40.727389       1 main-controller.go:1098] Tenant 'minio1' MinIO updated successfully from: 2022-01-08T03:11:54Z, to: 2023-01-12T02-06-16Z successfully
I0301 13:42:40.734570       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210236438", FieldPath:""}): type: 'Normal' reason: 'PoolUpdated' Tenant pool ss-0 updated
E0301 13:42:40.739656       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': Operation cannot be fulfilled on statefulsets.apps "minio1-ss-0": the object has been modified; please apply your changes to the latest version and try again
I0301 13:42:40.777311       1 monitoring.go:172] 'up-monitoring-db/minio1' Failed to get storage info: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/admin/v3/storageinfo": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:42.309382       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:42.969863       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:42.977416       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:43.091935       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:43.161610       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:43.563076       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:42:52.151584       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:01.816689       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:06.177671       1 http_handlers.go:180] MINIO_ARGS value is /export{0...3}/data
I0301 13:43:07.034851       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
E0301 13:43:34.989557       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': Put "https://minio.up-monitoring-db.svc.cluster.local/minio/admin/v3/add-user?accessKey=minio-generated-user": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:34.989639       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210236499", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: Put "https://minio.up-monitoring-db.svc.cluster.local/minio/admin/v3/add-user?accessKey=minio-generated-user": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:35.100379       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:35.872930       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:36.115466       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:36.124240       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:36.695880       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:37.094985       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:37.295943       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:37.695534       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:39.338701       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:43:39.722357       1 http_handlers.go:180] MINIO_ARGS value is /export{0...3}/data
I0301 13:43:40.123533       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "https://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.68.238.198:443: connect: connection refused
I0301 13:44:35.103513       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210236978", FieldPath:""}): type: 'Normal' reason: 'UsersCreated' Users created
I0301 13:44:35.109090       1 status.go:54] Hit conflict issue, getting latest version of tenant
I0301 13:44:35.123471       1 helper.go:825] Successfully created bucket chunks
I0301 13:44:35.127928       1 helper.go:825] Successfully created bucket admin
I0301 13:44:35.132539       1 helper.go:825] Successfully created bucket ruler
I0301 13:44:35.137618       1 status.go:203] Hit conflict issue, getting latest version of tenant
I0301 13:44:35.147216       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"up-monitoring-db", Name:"minio1", UID:"f0db263f-e8f4-4bed-9ee9-3ce8cecf3521", APIVersion:"minio.min.io/v2", ResourceVersion:"210236978", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created
I0301 13:44:35.151905       1 status.go:54] Hit conflict issue, getting latest version of tenant

Possible Solution

?

Steps to Reproduce (for bugs)

1. apply secrets

---
Name:         minio-user
Namespace:    up-monitoring-db
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
CONSOLE_ACCESS_KEY:  XX bytes
CONSOLE_SECRET_KEY:  XX bytes
---
Name:         minio1-secret
Namespace:    up-monitoring-db
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
accesskey:  XX bytes
secretkey:  XX bytes

2. apply operator helm chart
3. apply minio tenant CR
4. the following secret gets generated correctly by operator

Name:         minio1-configuration
Namespace:    up-monitoring-db
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
config.env:  XX bytes

Context

Since we use minio to store our logg chunks it is a broken log storage right now.
Additionally why did it recreate buckets shouldnt it have reconnected to our existing PVC?

Regression

Your Environment

• Version used (minio-operator): minio/operator:v4.5.8
• Environment name and version (e.g. kubernetes v1.17.2): v1.24.9

[rufreakde]

Okay With this exact settings I just changed:

requestAutoCert: true
to
requestAutoCert: false

As far as I understand this should not break the connection.
I still see the minio1-tlssecret. And it is still mounted as volumeMount:

      name: minio1-tls
      - mountPath: /tmp/minio-config
 ---
  - name: minio1-tls
    projected:
      defaultMode: 420
      sources:
      - secret:
          items:
          - key: public.crt
            path: CAs/operator.crt
          name: operator-tls

So what changed? Do we loose the autorotation feature now?

[rufreakde]

Additionally I noticed something. Changing the "desired state" of the image in the tenant resource. The minio Operator is not updating the statefulset image accordingly.

The log shows:

I0303 10:01:43.468081       1 main.go:77] Starting MinIO Operator
I0303 10:01:44.325850       1 main.go:176] caBundle on CRD updated
I0303 10:01:44.326275       1 main-controller.go:243] Setting up event handlers
I0303 10:01:44.326437       1 leaderelection.go:248] attempting to acquire leader lease up-monitoring-db/minio-operator-lock...
I0303 10:01:44.350888       1 leaderelection.go:258] successfully acquired lease up-monitoring-db/minio-operator-lock
I0303 10:01:44.350971       1 main-controller.go:409] Waiting for API to start
I0303 10:01:44.350994       1 main-controller.go:390] Starting console TLS certificate setup
I0303 10:01:44.351006       1 main-controller.go:404] Console TLS is not enabled
I0303 10:01:44.351014       1 main-controller.go:500] minio-operator-6bb5b96f9f-6tcmt: I am the leader, applying leader labels on myself
I0303 10:01:44.351017       1 main-controller.go:381] Starting HTTP Upgrade Tenant Image server
I0303 10:01:44.380462       1 main-controller.go:352] Using Kubernetes CSR Version: v1
I0303 10:01:44.387320       1 main-controller.go:356] Starting HTTPS API server
I0303 10:01:44.387571       1 main-controller.go:412] Waiting for Upgrade Server to start
I0303 10:01:44.387585       1 main-controller.go:415] Waiting for Console TLS
I0303 10:01:44.387591       1 main-controller.go:419] Starting Tenant controller
I0303 10:01:44.387597       1 main-controller.go:422] Waiting for informer caches to sync
I0303 10:01:44.387610       1 main-controller.go:427] Starting workers
I0303 10:01:44.423042       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:01:49.376438       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:01:49.376469       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:02:09.408749       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:02:09.408790       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:02:29.212399       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:02:29.224189       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:02:29.237270       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:02:35.953128       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:02:36.443869       1 monitoring.go:129] 'up-monitoring-db/minio1' Failed to get cluster health: Get "http://minio.up-monitoring-db.svc.cluster.local/minio/health/cluster": dial tcp 100.71.127.4:80: connect: connection refused
I0303 10:02:36.805699       1 http_handlers.go:180] MINIO_ARGS value is /export{0...3}/data
I0303 10:02:49.443214       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:02:49.443241       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:03:49.475850       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:03:49.475892       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:04:44.458462       1 monitoring.go:172] 'up-monitoring-db/minio1' Failed to get storage info: Failed to parse server response (invalid character 'C' looking for beginning of value): Client sent an HTTP request to an HTTPS server.
I0303 10:04:49.506723       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:04:49.506755       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:05:49.543115       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:05:49.543153       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready
I0303 10:06:49.575986       1 main-controller.go:1025] up-monitoring-db/minio1 is not running can't update image online
E0303 10:06:49.576024       1 main-controller.go:618] error syncing 'up-monitoring-db/minio1': MinIO is not ready

It is always showing MiniO not ready even though the status looks green.

Name:         minio1-ss-0-0
...
Events:
  Type    Reason                  Age    From                     Message
  ----    ------                  ----   ----                     -------
  Normal  Scheduled               5m52s  default-scheduler        Successfully assigned up-monitoring-db/minio1-ss-0-0 to ip-10-250-11-144.eu-central-1.compute.internal
  Normal  SuccessfulAttachVolume  5m49s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pv-shoot--spcnatprov--stag-op-f844f0e2-f553-43ac-b870-fd0ca5e52890"
  Normal  SuccessfulAttachVolume  5m49s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pv-shoot--spcnatprov--stag-op-fb4aaaef-960f-4c68-9e82-2903e6b1656c"
  Normal  SuccessfulAttachVolume  5m49s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pv-shoot--spcnatprov--stag-op-da2a76a9-c2eb-4042-aa24-9fa1446c90ce"
  Normal  SuccessfulAttachVolume  5m48s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pv-shoot--spcnatprov--stag-op-b5280ace-8d1a-4cc1-8825-6b4f2fe7cf56"
  Normal  Pulled                  5m45s  kubelet                  Container image "minio/minio:RELEASE.2022-01-08T03-11-54Z" already present on machine
  Normal  Created                 5m45s  kubelet                  Created container minio
  Normal  Started                 5m45s  kubelet                  Started container minio

[rufreakde]

Okay deleting the tenant CRD and recreating it worked except for:

Why is ths one not working now it worked before? :(

EDIT:
Currently working solution:

---
Name:         minio1-configuration
Namespace:    up-monitoring-db
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
secretkey:   xx bytes
accesskey:  xx bytes
config.env:  xx bytes
---
---
# Source: tenant/templates/tenant.yaml
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio1
  namespace: up-monitoring-db
  ## Optionally pass labels to be applied to the statefulset pods
  labels:
    app: minio
  annotations:
    argocd.argoproj.io/sync-wave: "100"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true


spec:
  image: quay.io/minio/minio:RELEASE.2023-01-12T02-06-16Z
  imagePullPolicy: IfNotPresent

  ## Secret with credentials to be used by MinIO Tenant.

  configuration:
    name: minio1-configuration

  pools:

    - servers: 1
      volumesPerServer: 4
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          storageClassName: st1
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 125Gi
      resources:

        requests:
          cpu: 100m
          memory: 128Mi


  mountPath: /export
  subPath: /data
  requestAutoCert: false # problematic functionality
  features:
    bucketDNS: false
  buckets:
    - name: chunks
      objectLock: false
    - name: admin
      objectLock: false
    - name: ruler
      objectLock: false
  users:
    - name: minio-user
  podManagementPolicy: Parallel

  prometheusOperator: false

question:

Will this work longer than a month?

[rufreakde]

Okay I was playing arond with different settings and configurations and it also seems that the operator does not work correctly on desired state "changes" the state of the tenant Database is in some kind of inconsistent state on such changes because the Operator does only apply changes but is not cleaning up!

e.g.
changing from requestAutoCert: true to requestAutoCert: false is not removing the configmap there.

There are several other things as well. This looks to me like an oversight in the operator behaviour! Hence a bug!
@pjuarezd should I create a new issue for this?

On the other side in our case we strictly don't need TLS (as we can either set this up with ISTIO auto inject)
Is there an fully example how to setup Operator and Tenant without any TLS configuration? Since service meshes with auto TLS injection are widely spread this should be a rather common use case.

[rufreakde]

So how to handle istio TLS injection with minio?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[cniackz]

Hello @rufreakde first of all thank you for creating an issue in our Operator Repository.

There are many topics here to discuss but I am not sure what the issue is or what should we exactly address here.

1. For the original title, I would suggest to upgrade to latest Operator version available 5.0.9, there are multiple changes and there shoudn't be any issue on the upgrade with regards to the buckets. If it persist, please open new ticket with precise steps on how to reproduce.

2. Additionally why did it recreate buckets shouldnt it have reconnected to our existing PVC? PVs can be re-used but you have to be careful in how to deploy the tenant for that to happen. Same tenant name is expected and same pool name is expected. That is possible to achieve. If this is not the case even today for you, then please open a new issue with the exact steps to reproduce and I can help or guide in how to achieve that. By the way, buckets will not be generated again if already present because we make sure of it. So the fact that they got regenerated, it means we are using new PVs and that should not be if you plan on re-using the Volumes. Anyway, that is just a configuration issue on how tenant got deployed over new volumes.

3. Finally TLS can be disabled and you don't have to keep it on. To disable it simple do this as you saw:

  spec:
    requestAutoCert: false

If you need external certs, then that will require other fields. Having requestAutoCert: false means there will not be any self-signed certificate generated by the operator. Besides, Operator is no longer generating TLS on its own in the latest version as we already introduced many new changes like the sidecar. So please try latest Operator and if there is a bug, yes please open a new issue for this with precise steps to reproduce and thank you again!.