Portainer CE on k3s (Kubernetes 1.24+) fails with x509 errors due to missing CA in auto-mounted token — full working fix

[oliviermonney-crypto]

Ask a Question!

Summary

When running Portainer CE inside a k3s cluster (or any Kubernetes 1.24+ cluster), the UI fails to list namespaces, pods, services, ingresses, secrets, etc., and repeatedly shows TLS errors such as:

Warning

tls: failed to verify certificate: x509: certificate signed by unknown authority

This happens even when Portainer is running inside the cluster and using the default ServiceAccount.
After investigation, this appears to be a structural incompatibility between:

• Kubernetes 1.24+ (which removed legacy long-lived ServiceAccount tokens)
• k3s (which uses a self-signed cluster CA)
• Portainer CE (which expects a legacy-style ServiceAccount token bundle)

Below is the root cause and a solution.

Environment

• k3s (Kubernetes 1.24+)
• Portainer CE running inside the cluster
• Default ServiceAccount auto-mounted token (BoundServiceAccountTokenVolume)
• Self-signed cluster CA (default in k3s)

Symptoms

Portainer CE UI shows errors such as:

Warning

Unable to retrieve namespaces: tls: failed to verify certificate: x509: certificate signed by unknown authority
Unable to list pods across the cluster: tls: failed to verify certificate
Unable to get dashboard stats

The Portainer container logs show the same TLS validation failures.

Root Cause

1. Kubernetes 1.24+ removed legacy long-lived ServiceAccount tokens

BoundServiceAccountTokens are:

• short-lived (1 hour)
• auto-rotated
• mounted automatically

2. Portainer CE reads the token from:

/var/run/secrets/kubernetes.io/serviceaccount

It expects three files:

• token
• ca.crt
• namespace

But with Kubernetes 1.24+:

• the auto-mounted token does not include a CA bundle
• and the CA is required for TLS validation

3. Portainer CE has no UI to configure:

• a custom token
• a custom CA
• or a custom API certificate

These options exist in Portainer Business, but not in CE.

Solution Overview

• To make Portainer CE work correctly inside k3s, we must:
• Create a long-lived legacy ServiceAccount token manually
• Ensure the Secret contains both the token AND the CA
• Disable auto-mounting of the ephemeral token
• Mount our static Secret at the expected path
• Restart Portainer

This restores the legacy behavior Portainer CE expects.

Step-by-step

1. Create a long-lived ServiceAccount token

# portainer-sa-token.yaml
apiVersion: v1
kind: Secret
metadata:
  name: portainer-sa-token
  namespace: portainer
  annotations:
    kubernetes.io/service-account.name: portainer-sa
type: kubernetes.io/service-account-token

Apply:

kubectl apply -f portainer-sa-token.yaml

Extract the token:

kubectl get secret portainer-sa-token -n portainer -o jsonpath='{.data.token}' | base64 -d

2. Add the cluster CA to the Secret

Extract the k3s CA:

sudo cat /var/lib/rancher/k3s/server/tls/server-ca.crt | base64 -w0

Edit the Secret:

kubectl edit secret portainer-sa-token -n portainer

Add or replace:

data:
  ca.crt: <base64_of_CA>

This ensures the Secret contains the full legacy ServiceAccount bundle.

3. Disable auto-mounted ephemeral token

kubectl patch deployment portainer -n portainer --type=json -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/automountServiceAccountToken",
    "value": false
  }
]'

4. Mount the full Secret (token + CA + namespace)

The initial patch mounted only the token, which caused the TLS failure.
We must mount the entire Secret.

kubectl patch deployment portainer -n portainer --type=json -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
      "name": "static-token",
      "secret": {
        "secretName": "portainer-sa-token"
      }
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "static-token",
      "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
      "readOnly": true
    }
  }
]'

5. Restart Portainer

kubectl rollout restart deployment/portainer -n portainer
kubectl rollout status deployment/portainer -n portainer

Result

After applying this fix:
Portainer CE successfully lists namespaces
Pods, services, ingresses, secrets, PVCs all load correctly
No more TLS errors
No need to disable TLS verification
No need for Portainer Business features
/Optional improvements for maintainers
A minimal enhancement would be:

• Allow specifying a custom CA bundle
• Allow specifying a custom token
• Or automatically read the CA from the mounted Secret

This would make Portainer CE compatible with modern Kubernetes clusters without manual patching.
I hope this helps

[mg1986jp]

This is a common issue with k3s 1.24+ because Kubernetes stopped auto-mounting the service account token with the CA cert in the projected volume. The token is there but the CA bundle isn't where Portainer expects it.

The fix is to either:

1. Mount the k3s CA certificate explicitly into the Portainer deployment. In k3s, the CA is usually at /var/lib/rancher/k3s/server/tls/server-ca.crt on the host. You can create a ConfigMap from it and mount it into the Portainer pod.

2. Or set the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT env vars in the Portainer pod, and make sure the ServiceAccount has the CA bundle mounted at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. In k3s you might need to create a proper ServiceAccount with the annotation for the bound token.

3. Quickest workaround: add --set env[0].name=KUBERNETES_SERVICE_SKIP_TLS_VERIFY,env[0].value=true to your Helm install if you're OK with skipping TLS verification for the internal k3s API. Not great for production but works for testing.

Which approach are you going with?

[oliviermonney-crypto]

Thanks for the detailed explanation. I really appreciate you taking the time to look into this.

I’m running k3s, and in my case the issue wasn’t related to the CA bundle or the projected token volume, but simply that no kubernetes.io/service-account-token secret was created at all for the portainer-sa service account.

I understand that in some k3s setups the token may be auto‑generated, but in my environment it simply didn’t happen, so Portainer had no token and couldn’t access the Kubernetes API.
The fix for me was to manually create a bound token secret linked to the existing portainer-sa service account.
After applying this, k3s generated a valid token and CA bundle inside the secret, and Portainer immediately recovered API access without needing to mount the CA manually or disable TLS verification.

Since it took me a while to track down the root cause, I’m sharing the solution here in case it helps someone else.
Everything is working perfectly now on my side, so I’ll go ahead and close the issue.