[Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding] seen in Kubernetes container after K8s upgrade

[soumyalj]

Rancher versions:
rancher/server:v1.6.6-rc5

**Docker version: (docker version,docker info preferred)**v1.12.6

**Operating system and kernel: (cat /etc/os-release, uname -r preferred)**Ubuntu

**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)**DO

**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)**Single node

**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)**Kubernetes
Kubernetes version : v1.7.2

Steps to Reproduce:

1. Create a Kubernetes setup in rancher v1.5.7 with Kubernetes v1.5.4
2. Upgrade to v1.6.6-rc5 and upgrade all the infra stacks.
3. Upgrade Kubernetes to v1.7.2
   After Kubernetes upgrade, the following logs are seen in the Kubernetes system container

7/28/2017 10:49:57 AME0728 17:49:57.689726       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:02 AME0728 17:50:02.301758       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:03 AME0728 17:50:03.363587       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:07 AME0728 17:50:07.819415       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:09 AME0728 17:50:09.158002       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:13 AME0728 17:50:13.417460       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:14 AME0728 17:50:14.820881       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:18 AME0728 17:50:18.955695       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:20 AME0728 17:50:20.517262       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:24 AME0728 17:50:24.303228       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:26 AME0728 17:50:26.106003       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:29 AME0728 17:50:29.865332       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:31 AME0728 17:50:31.738074       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]
7/28/2017 10:50:35 AME0728 17:50:35.684330       1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Token has been invalidated, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]

[leodotcloud]

Similar error seen here too: #9508

[alena1108]

@leodotcloud the reason seems to be different. In 9508 it is a connectivity issue

[lkollenberger]

I think we have the same problem in a fresh installation of Rancher v1.6.10.

kube-dns is unavailable, and logs in the kubernetes container are full of these:

E1031 16:35:56.664114 1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [crypto/rsa: verification error, an error on the server ("illegal base64 data at input byte 36") has prevented the request from succeeding]]]

Hosts are Ubuntu Server 16.04, with Docker 1.12.6. I'm using a custom kubernetes template with portainer and with vxlan instead of ipsec.

Is there any workaround available?

EDIT: thanks to @superseb, who found a solution: this was caused by reusing a host with a previous installation of Rancher in it, I though that removing the containers was enough (I was wrong). Followed this, and cleaned /var/lib/rancher/, everything is up and running again.