Impact
An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where sensitive fields, like passwords, API keys and Rancher's service account token (used to provision clusters), were stored in plaintext directly on Kubernetes objects like Clusters
, for example cluster.management.cattle.io
. Anyone with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data.
The exposed credentials are visible in Rancher to authenticated Cluster Owners
, Cluster Members
, Project Owners
, Project Members
and User Base
on the endpoints:
/v1/management.cattle.io.catalogs
/v1/management.cattle.io.cluster
/v1/management.cattle.io.clustertemplates
/v1/management.cattle.io.notifiers
/v1/project.cattle.io.sourcecodeproviderconfig
/k8s/clusters/local/apis/management.cattle.io/v3/catalogs
/k8s/clusters/local/apis/management.cattle.io/v3/clusters
/k8s/clusters/local/apis/management.cattle.io/v3/clustertemplates
/k8s/clusters/local/apis/management.cattle.io/v3/notifiers
/k8s/clusters/local/apis/project.cattle.io/v3/sourcecodeproviderconfigs
Sensitive fields are now stripped from Clusters
and other objects and moved to a Secret
before the object is stored. The Secret
is retrieved when the credential is needed. For objects that existed before this security fix, a one-time migration happens on startup.
Important:
- The exposure of Rancher's
serviceAccountToken
allows any standard user to escalate its privileges to cluster administrator in Rancher.
- For the exposure of credentials not related to Rancher, the final impact severity for confidentiality, integrity and availability is dependent on the permissions that the leaked credentials have on their own services.
The fields that have been addressed by this security fix are:
Notifier.SMTPConfig.Password
Notifier.WechatConfig.Secret
Notifier.DingtalkConfig.Secret
Catalog.Spec.Password
SourceCodeProviderConfig.GithubPipelineConfig.ClientSecret
SourceCodeProviderConfig.GitlabPipelineConfig.ClientSecret
SourceCodeProviderConfig.BitbucketCloudPipelineConfig.ClientSecret
SourceCodeProviderConfig.BitbucketServerPipelineConfig.PrivateKey
Cluster.Spec.RancherKubernetesEngineConfig.BackupConfig.S3BackupConfig.SecretKey
Cluster.Spec.RancherKubernetesEngineConfig.PrivateRegistries.Password
Cluster.Spec.RancherKubernetesEngineConfig.Network.WeaveNetworkProvider.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.Global.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.VirtualCenter.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.OpenstackCloudProvider.Global.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientSecret
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientCertPassword
Cluster.Status.ServiceAccountToken
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.PrivateRegistries.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.Network.WeaveNetworkProvider.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.Global.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.VirtualCenter.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.OpenstackCloudProvider.Global.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientSecret
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientCertPassword
Patches
Patched versions include releases 2.5.16, 2.6.7 and later versions.
After upgrading to a patched version, it is important to check for the SecretsMigrated
condition on Clusters
, ClusterTemplates
, and Catalogs
to confirm when secrets have been fully migrated off of those objects and the objects scoped within them (Notifiers
and SourceCodeProviderConfigs
).
Workarounds
Limit access in Rancher to trusted users. There is not a direct mitigation besides upgrading to the patched Rancher versions.
Important:
- It is highly advised to rotate Rancher's
serviceAccountToken
. This rotation is not done by the version upgrade. Please see the helper script below.
- The local and downstream clusters should be checked for potential unrecognized services (pods), users and API keys.
- It is recommended to review for potential leaked credentials in this scenario, that are not directly related to Rancher, and to change them if deemed necessary.
The script available in rancherlabs/support-tools/rotate-tokens repository can be used as a helper to rotate the service account token (used to provision clusters). The script requires a valid Rancher API token, kubectl
access to the local
cluster and the jq
command.
Credits
We would like to recognize and appreciate Florian Struck (from Continum AG) and Marco Stuurman (from Shock Media B.V.) for the responsible disclosure of this security issue.
For more information
If you have any questions or comments about this advisory:
Impact
An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where sensitive fields, like passwords, API keys and Rancher's service account token (used to provision clusters), were stored in plaintext directly on Kubernetes objects like
Clusters
, for examplecluster.management.cattle.io
. Anyone with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data.The exposed credentials are visible in Rancher to authenticated
Cluster Owners
,Cluster Members
,Project Owners
,Project Members
andUser Base
on the endpoints:/v1/management.cattle.io.catalogs
/v1/management.cattle.io.cluster
/v1/management.cattle.io.clustertemplates
/v1/management.cattle.io.notifiers
/v1/project.cattle.io.sourcecodeproviderconfig
/k8s/clusters/local/apis/management.cattle.io/v3/catalogs
/k8s/clusters/local/apis/management.cattle.io/v3/clusters
/k8s/clusters/local/apis/management.cattle.io/v3/clustertemplates
/k8s/clusters/local/apis/management.cattle.io/v3/notifiers
/k8s/clusters/local/apis/project.cattle.io/v3/sourcecodeproviderconfigs
Sensitive fields are now stripped from
Clusters
and other objects and moved to aSecret
before the object is stored. TheSecret
is retrieved when the credential is needed. For objects that existed before this security fix, a one-time migration happens on startup.Important:
serviceAccountToken
allows any standard user to escalate its privileges to cluster administrator in Rancher.The fields that have been addressed by this security fix are:
Notifier.SMTPConfig.Password
Notifier.WechatConfig.Secret
Notifier.DingtalkConfig.Secret
Catalog.Spec.Password
SourceCodeProviderConfig.GithubPipelineConfig.ClientSecret
SourceCodeProviderConfig.GitlabPipelineConfig.ClientSecret
SourceCodeProviderConfig.BitbucketCloudPipelineConfig.ClientSecret
SourceCodeProviderConfig.BitbucketServerPipelineConfig.PrivateKey
Cluster.Spec.RancherKubernetesEngineConfig.BackupConfig.S3BackupConfig.SecretKey
Cluster.Spec.RancherKubernetesEngineConfig.PrivateRegistries.Password
Cluster.Spec.RancherKubernetesEngineConfig.Network.WeaveNetworkProvider.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.Global.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.VirtualCenter.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.OpenstackCloudProvider.Global.Password
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientSecret
Cluster.Spec.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientCertPassword
Cluster.Status.ServiceAccountToken
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.PrivateRegistries.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.Network.WeaveNetworkProvider.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.Global.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.VsphereCloudProvider.VirtualCenter.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.OpenstackCloudProvider.Global.Password
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientSecret
ClusterTemplate.Spec.ClusterConfig.RancherKubernetesEngineConfig.CloudProvider.AzureCloudProvider.AADClientCertPassword
Patches
Patched versions include releases 2.5.16, 2.6.7 and later versions.
After upgrading to a patched version, it is important to check for the
SecretsMigrated
condition onClusters
,ClusterTemplates
, andCatalogs
to confirm when secrets have been fully migrated off of those objects and the objects scoped within them (Notifiers
andSourceCodeProviderConfigs
).Workarounds
Limit access in Rancher to trusted users. There is not a direct mitigation besides upgrading to the patched Rancher versions.
Important:
serviceAccountToken
. This rotation is not done by the version upgrade. Please see the helper script below.The script available in rancherlabs/support-tools/rotate-tokens repository can be used as a helper to rotate the service account token (used to provision clusters). The script requires a valid Rancher API token,
kubectl
access to thelocal
cluster and thejq
command.Credits
We would like to recognize and appreciate Florian Struck (from Continum AG) and Marco Stuurman (from Shock Media B.V.) for the responsible disclosure of this security issue.
For more information
If you have any questions or comments about this advisory: