same JWT not work for login operator console, get 401 unauthenticated for invalid credentials #1422

qwegas · 2023-01-29T02:29:00Z

just install and create a nodeport svc to login it

helm install minio-operator  operator-4.5.8.tgz -n minio-operator --create-namespace

apply

apiVersion: v1
kind: Service
metadata:
  name: console-nodeport
  namespace: minio-operator
spec:
  ports:
  - name: http
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app.kubernetes.io/instance: minio-operator-console
    app.kubernetes.io/name: operator
  type: NodePort

---

apiVersion: v1
kind: Secret
metadata:
  name: console-sa-secret
  namespace: minio-operator
  annotations:
    kubernetes.io/service-account.name: console-sa
type: kubernetes.io/service-account-token

get jwt

kubectl -n minio-operator  get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode

and then failed

console log

Serving operator at http://[::]:9090
ErrorWithContext:Get "https://dl.min.io/server/minio/release/linux-amd64/": dial tcp: lookup dl.min.io on 10.43.0.10:53: server misbehaving
ErrorWithContext:Get "https://dl.min.io/server/minio/release/linux-amd64/": dial tcp: lookup dl.min.io on 10.43.0.10:53: server misbehaving
ErrorWithContext:Get "https://dl.min.io/server/minio/release/linux-amd64/": dial tcp: lookup dl.min.io on 10.43.0.10:53: server misbehaving
ErrorWithContext:Get "https://dl.min.io/server/minio/release/linux-amd64/": dial tcp: lookup dl.min.io on 10.43.0.10:53: server misbehaving

Your Environment

Version used (minio-operator): 4.5.8
Environment name and version (e.g. kubernetes v1.17.2): 1.23.6
Server type and version:
Operating System and version (uname -a):
Link to your deployment file:

The text was updated successfully, but these errors were encountered:

qwegas · 2023-01-29T03:31:03Z

use 4.5.7 save my life

qwegas · 2023-01-29T07:13:44Z

same problem in 4.5.7
At the first time，it can login the operator console

and then install tenant by helm
maybe the login timeout and refresh to the login page.
use same jwt to login
get same error
even reinstall the helm chart can't fix the problem

z666k · 2023-03-17T07:40:32Z

I also get such a problem,my version :
k8s: 1.23.17
minio: 4.5.8
install style: kubectl minio init --console-tls

stale · 2023-06-17T22:56:23Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

yaohwu · 2023-07-04T08:34:12Z

got same 401 error after following https://min.io/docs/minio/kubernetes/upstream/operations/installation.html doc and creating a minio tenant.

Token maybe be invalid after sometime or creating a tenant or other actions that I don't know.

kubectl minio version
v5.0.6

Even using
kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r '.data.token' | base64 -d
to get the token but the 401 happened again.

yaohwu · 2023-07-04T09:01:35Z

got same error even after kubectl minio delete and kubectl minio init

yaohwu · 2023-07-04T09:16:13Z

Here is the logs

E: 2023/07/04 09:08:40 Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir /tmp/certs/CAs: read-only file system
Serving operator at http://[::]:9090
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value

From the key word,it maye be same with #1534.

yaohwu · 2023-07-04T10:03:54Z

Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir

apiVersion: v1
kind: Service
metadata:
  name: operator-minio-external
spec:
  type: NodePort
  ports:
  - nodePort: 31909
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app.kubernetes.io/name: operator

Using this to create a NodePort, get same error 401

asura-10 · 2023-07-17T16:48:39Z

Here is the logs

E: 2023/07/04 09:08:40 Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir /tmp/certs/CAs: read-only file system
Serving operator at http://[::]:9090
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value
Unable to validate the session token Anonymous: invalid character 'A' looking for beginning of value

From the key word,it maye be same with #1534.

To use Chrome's incognito mode， give a shot

allanrogerr · 2023-10-12T19:42:27Z

4.5.8 @qwegas This is not how a NodePort is defined. You are missing .spec.ports[].nodePort Please observe the following sample:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: minio-operator
    meta.helm.sh/release-namespace: minio-operator
  creationTimestamp: "0001-01-01T01:01:01Z"
  labels:
    app.kubernetes.io/instance: minio-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: operator
    app.kubernetes.io/version: v4.5.8
    helm.sh/chart: operator-4.5.8
  name: console
  namespace: minio-operator
  resourceVersion: "1333"
  uid: 1-2-3-4-5
spec:
  clusterIP: x.x.x.x
  clusterIPs:
  - x.x.x.x
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 31000
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app.kubernetes.io/instance: minio-operator-console
    app.kubernetes.io/name: operator
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

allanrogerr · 2023-10-12T19:43:18Z

The secret should also be autogenerated by the minio-operator. Please do not modify it.

allanrogerr · 2023-10-12T19:49:45Z

Here is a complete example:

Install k3s. Print versions.

curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - --snapshotter=fuse-overlayfs
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
k3s --version

Output

k3s version v1.27.6+k3s1 (bd04941a)
go version go1.20.8

sudo systemctl status k3s.service

Output

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Thu 2023-10-12 19:00:40 UTC; 26min ago

Install helm

wget https://get.helm.sh/helm-v3.6.3-linux-amd64.tar.gz
tar -zxvf helm-v3.6.3-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm version

Output

version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

Install minio-operator. Print version.

curl -O https://raw.githubusercontent.com/minio/operator/master/helm-releases/operator-4.5.8.tgz
tar -xvf operator-4.5.8.tg
helm install --namespace minio-operator --create-namespace minio-operator operator-4.5.8.tgz -f operator/values.yaml
helm inspect chart operator

Output

apiVersion: v2
appVersion: v4.5.8
description: A Helm chart for MinIO Operator
home: https://min.io
icon: https://min.io/resources/img/logo/MINIO_wordmark.png
keywords:
- storage
- object-storage
- S3
maintainers:
- email: dev@minio.io
  name: MinIO, Inc
name: operator
sources:
- https://github.com/minio/operator
type: application
version: 4.5.8

Validate minio-operator running

kubectl get pods -n minio-operator

Output

NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-7d988dd6b6-lckt4   1/1     Running   0          24m
console-5756b69c9f-8s8d8          1/1     Running   0          24m
minio-operator-7d988dd6b6-swvnj   1/1     Running   0          24m

Add Nodeport to service

kubectl patch service -n minio-operator console --type='merge' -p '{"spec":{"type": "NodePort", "ports":[{"name": "http", "port": 9090, "nodePort": 31000}]}}'
kubectl -n minio-operator get svc/console -o yaml

Output (redacted)

apiVersion: v1
kind: Service
...
spec:
  clusterIP: x.x.x.x
  clusterIPs:
  - x.x.x.x
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 31000
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app.kubernetes.io/instance: minio-operator-console
    app.kubernetes.io/name: operator
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Get jwt and login

kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode

allanrogerr · 2023-10-12T19:50:39Z

@yaohwu Your NodePort is not correctly defined as well. Please see above examples.

yaohwu · 2023-10-25T07:50:52Z

@yaohwu Your NodePort is not correctly defined as well. Please see above examples.

thanks and I will try it in the next few days

allanrogerr · 2023-11-29T11:46:31Z

@yaohwu Any luck?

yaohwu · 2023-11-30T11:16:44Z

Well I try the above examples and it works as of now. But as what I said before, I am worried about that it will be 401 after sometime. I will feedback if some issues happened

Token maybe be invalid after sometime or creating a tenant or other actions that I don't know.

allanrogerr · 2023-11-30T23:25:26Z

@yaohwu Does this work?

The minio-operator console's jwt will not change unless you recreate console-sa-secret for some reason. However, its purpose is only to provision the tenants. Once the tenants are created, you use the username and password combination (or openid/ldap) previously configured.

You should therefore directly access the MinIO tenant console using the following means:

Validate the tenant console service is up

kubectl --namespace <TENANT_NAMESPACE> get svc/<TENANT_NAME>-console

Add a NodePort (or Port Forward)

kubectl patch service -n <TENANT_NAMESPACE>/<TENANT_NAME>-console --type='merge' -p '{"spec":{"type": "NodePort", "ports":[{"name": "https", "port": 9443, "nodePort": 30000}]}}'

or

kubectl -n <TENANT_NAMESPACE> port-forward svc/<TENANT_NAME>-console 9043:9443 --address 0.0.0.0

Access the tenant console directly

qwegas added community triage labels Jan 29, 2023

qwegas closed this as completed Jan 29, 2023

qwegas reopened this Jan 29, 2023

qwegas changed the title ~~login operator console get 401 unauthenticated for invalid credentials~~ same JWT not work for login operator console, get 401 unauthenticated for invalid credentials Jan 29, 2023

stale bot added the stale label Jun 17, 2023

stale bot removed the stale label Jul 4, 2023

allanrogerr self-assigned this Oct 12, 2023

allanrogerr added the question Further information is requested label Nov 30, 2023

allanrogerr closed this as completed Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

same JWT not work for login operator console, get 401 unauthenticated for invalid credentials #1422

same JWT not work for login operator console, get 401 unauthenticated for invalid credentials #1422

qwegas commented Jan 29, 2023

qwegas commented Jan 29, 2023

qwegas commented Jan 29, 2023 •

edited

z666k commented Mar 17, 2023

stale bot commented Jun 17, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

asura-10 commented Jul 17, 2023

allanrogerr commented Oct 12, 2023

allanrogerr commented Oct 12, 2023

allanrogerr commented Oct 12, 2023

allanrogerr commented Oct 12, 2023

yaohwu commented Oct 25, 2023

allanrogerr commented Nov 29, 2023

yaohwu commented Nov 30, 2023

allanrogerr commented Nov 30, 2023

same JWT not work for login operator console, get 401 unauthenticated for invalid credentials #1422

same JWT not work for login operator console, get 401 unauthenticated for invalid credentials #1422

Comments

qwegas commented Jan 29, 2023

Your Environment

qwegas commented Jan 29, 2023

qwegas commented Jan 29, 2023 • edited

z666k commented Mar 17, 2023

stale bot commented Jun 17, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

yaohwu commented Jul 4, 2023

asura-10 commented Jul 17, 2023

allanrogerr commented Oct 12, 2023

allanrogerr commented Oct 12, 2023

allanrogerr commented Oct 12, 2023

Install k3s. Print versions.

Install helm

Install minio-operator. Print version.

Validate minio-operator running

Add Nodeport to service

Get jwt and login

allanrogerr commented Oct 12, 2023

yaohwu commented Oct 25, 2023

allanrogerr commented Nov 29, 2023

yaohwu commented Nov 30, 2023

allanrogerr commented Nov 30, 2023

qwegas commented Jan 29, 2023 •

edited