invalid Login whenever operator provisioned tenant console is accessed

[gelato]

Expected Behavior

Tenant console allows user access with credentials set up during provisioning using operator ui

Current Behavior

Always getting invalid Login when trying to access console. mc admin works though. I've tried deploying tenant with/without TLS, tried setting MINIO_BROWSER_REDIRECT_URL, my additional domains are also always set. I even tried creating new user through mc admin app and assigning consoleAdmin policy to it - still getting 401 from console (Failed to load resource: the server responded with a status of 401 ()). One thing I can tell - my public domain resolves to local addresses, but don't think that it matters.

Possible Solution

No clue

Steps to Reproduce (for bugs)

Deploy MinIO Operator with helm, deploy tenant with setting external domains and turning off service exposure

Context

I can't login to console

Regression

No clue

Your Environment

• Version used (minio-operator): 5.0.10
• Environment name and version (e.g. kubernetes v1.17.2): Kubernetes Server Version: v1.28.5
• Server type and version: ??
• Operating System and version (uname -a): Linux 6.6.6-76060606-generic #202312111032170230614322.04~d28ffec SMP PREEMPT_DYNAMIC Mon D x86_64 x86_64 x86_64 GNU/Linux
• Link to your deployment file: https://app.assembla.com/spaces/k8s/git-2/source/main/clusters/monolith/infrastructure/minio/values.yaml?_format=raw&disposition=inline

Ingresses from here: https://github.com/minio/operator/blob/master/docs/nginx-ingress.md

[gelato]

Figured it myself by deploying outdated version of tenant, which showed much more verbose error messages, instead of just "Invalid login". So my k8s cluster is self-hosted cluster witrh certmanager and enabled TLS. And for tenant to work properly i need to add secret names of my ingress's certificates to tenant yaml config after tenant is created. (there's a set of yaml parameters for that in tenant chart, but nothing in the webui)

[dharapvj]

We also faced the same issue..

Older minio version .. RELEASE.2023-05-27T05-56-19Z gave more verbose error..
Post "https://tenant_name.svc.k8s_domain_name/": x509: certificate signed by unknown authority

To fix this issue - we added below config in tenant helm values block.

minio-tenant:
  tenant:  
    certificate:
      externalCertSecret:
      - name: minio-TENANT_NAME-certificate
        type: kubernetes.io/tls

and we created this secret as below. Replace names appropriately.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: TENANT-tls
  namespace: TENANT
spec:
  dnsNames:
    - "*.TENANT.svc.cluster.local"
    - "*.TENANT-hl.TENANT.svc.cluster.local"
    - "*.TENANT.TENANT.svc.cluster.local"
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: ca-prod
  secretName: minio-TENANT_NAME-certificate

This fixed the issue.

@harshavardhana - can we re-open this issue to fix the error message in newer version of minio? New version minio gives only very vague invalid Login error message.

[aep]

Older minio version .. RELEASE.2023-05-27T05-56-19Z gave more verbose error..

thanks for the tip!

Post "https://minio.cc-aep.svc.cluster.local/": x509: certificate is valid for s3.aep.cc.ber1.minio.aep.cc.ber1.public, , not minio.cc-aep.svc.cluster.local

so the cert needs to be valid for minio.cc-aep.svc.cluster.local, but there's no way to do that in my case since we're using letsencrypt

[Starttoaster]

It definitely feels like a mistake to me that this doesn't work with LetsEncrypt issued certificates. We're hitting the same issue trying to deploy a tenant through the operator and accessing the console. Interestingly though, login only fails in the console -- using the mc command line utility the same credentials and same TLS certs works fine. @cniackz could this Issue be re-opened as it's still relevant with publicly trusted TLS certificates through LetsEncrypt, which is an extremely common cert issuer for Kubernetes infrastructure.

edit: We got this working by using Cloudflare as a proxy. So this may actually be a weird hairpin routing issue or similar. However, the console didn't make this clear at all because it just spits out "invalid Login". It would be helpful to have more debugging information to go on. The only reason we began suspecting a hairpin routing issue was because we enabled OIDC, and the OIDC integration in the console told us it was an i/o timeout, where regular user/pass authentication just spat out a 401 with no context.

[jpabbuehl]

@Starttoaster can you provide an example how to make this work with cloudflare? are you using cert-manager?
Thanks
Jean-Paul