Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2019-11255: CSI volume snapshot, cloning and resizing features can result in unauthorized volume data access or mutation #85233

Closed
msau42 opened this issue Nov 13, 2019 · 5 comments
Assignees
Labels
area/security kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) sig/storage Categorizes an issue or PR as relevant to SIG Storage.

Comments

@msau42
Copy link
Member

msau42 commented Nov 13, 2019

Am I vulnerable?

CSI snapshot, cloning and resizing features are affected. Prior to Kubernetes 1.16, these features were all alpha and disabled by default. Starting in Kubernetes 1.16, CSI cloning and resizing features are beta and enabled by default.

These features also require CSI drivers to be installed in a Kubernetes cluster and the CSI driver also has to support those features. An unofficial list of CSI drivers and their supported features is available here, however it is best to check with the CSI driver vendor for the latest information.

Check if you have the following Kubernetes feature gates enabled:

VolumeSnapshotDataSource: alpha starting with K8s 1.12
ExpandCSIVolumes: alpha starting with K8s 1.14, beta starting with K8s 1.16
VolumePVCDataSource: alpha starting with K8s 1.15, beta starting with K8s 1.16

Check if you are using CSI drivers in your cluster. If so, the following command’s output will be non-empty:

$ kubectl get nodes -o jsonpath='{.items[*].metadata.annotations.csi\.volume\.kubernetes\.io\/nodeid}'
      {"my-csi-plugin":"kubernetes-minion-group-433q"}

Then, check the CSI driver’s pod specifications to see if they are using the following vulnerable versions of sidecars:

external-provisioner: v0.4.1-0.4.2, v1.0.0-1.0.1, v1.1.0-1.2.1, v1.3.0
external-snapshotter: v0.4.0-0.4.1, v1.0.0-1.0.1, v1.1.0-v1.2.1
external-resizer: v0.1.0-0.2.0

An example query:

$ kubectl get pods --all-namespaces -o jsonpath='{..image}' | tr ' ' $'\n' | grep "csi-provisioner\|csi-snapshotter\|csi-resizer"
      image: quay.io/k8scsi/csi-provisioner:v1.2.0

Note that the exact container image name may vary across CSI driver vendors. It is recommended to inspect the Pod specifications directly.

How do I mitigate the vulnerability?

As a short term mitigation, disable the VolumeSnapshotDataSource, ExpandCSIVolumes, and VolumePVCDataSource Kubernetes feature gates in kube-apiserver and kube-controller-manager. This will cause new PersistentVolumeClaims to be provisioned ignoring the DataSource and resizing requests will also be ignored. Note that this will cause new PVCs that are intended to be provisioned from a snapshot or clone to instead provision a blank disk.

Also, to disable taking volume snapshots, either remove the external-snapshotter sidecar from any CSI drivers or revoke the CSI driver’s RBAC permissions on the snapshot.storage.k8s.io API group.

Longer term, upgrade your CSI driver with patched versions of the affected sidecars. Fixes are available in the following sidecar versions:

external-provisioner:
v0.4.3
v1.0.2
v1.2.2
v1.3.1
v1.4.0

external-snapshotter:
v0.4.2
v1.0.2
v1.2.2

external-resizer
v0.3.0

Fixes for each of the sidecars can be tracked by:
kubernetes-csi/external-provisioner#380
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-resizer#63

How do I upgrade?

Check with your CSI driver vendor for upgrade instructions. No Kubernetes control plane or node upgrades are required unless the CSI driver is bundled into the Kubernetes distribution.

Vulnerability details

There are two different vulnerabilities impacting the same features.

When PersistentVolumeClaim and PersistentVolume objects are bound, they have bidirectional references to each other. When dereferencing a PersistentVolumeClaim to get a PersistentVolume, the impacted sidecar controllers were not validating that the PersistentVolume referenced back to the same PersistentVolumeClaim, potentially operating on unauthorized PersistentVolumes for snapshot, cloning and resizing operations.

A similar issue exists for VolumeSnapshot and VolumeSnapshotContent objects when creating a new PersistentVolumeClaim from a snapshot.

The second issue is related to the property that CSI volume and snapshot ids are only required to be unique within a single CSI driver. Impacted sidecar controllers were not validating that the requested source VolumeSnapshot or PersistentVolumeClaim specified were from the same driver processing the request, potentially operating on unauthorized volumes during snapshot, restore from snapshot, or cloning operations.

@msau42 msau42 added the kind/bug Categorizes issue or PR as related to a bug. label Nov 13, 2019
@msau42
Copy link
Member Author

msau42 commented Nov 13, 2019

/assign

@msau42
Copy link
Member Author

msau42 commented Nov 14, 2019

/area security
@kubernetes/sig-storage-bugs

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. area/security and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 14, 2019
@msau42
Copy link
Member Author

msau42 commented Nov 14, 2019

/close

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 14, 2019

@msau42: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Madhu-1 added a commit to Madhu-1/rook that referenced this issue Nov 18, 2019
Madhu-1 added a commit to Madhu-1/rook that referenced this issue Nov 19, 2019
mergify bot pushed a commit to rook/rook that referenced this issue Nov 19, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 49381b3)
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Updated container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
nizamial09-zz pushed a commit to nizamial09-zz/rook that referenced this issue Nov 25, 2019
zoetrope pushed a commit to cybozu-go/rook that referenced this issue Dec 26, 2019
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Jan 14, 2020
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Jan 21, 2020
Changes:
- Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)
- Update snapshotter RBAC policy
- Update resizer RBAC policy
- Updates external-provisioner RBAC policy for v1.3.1 image
- Update helm charts with updated RBAC policy

ref: kubernetes-sigs#411
kfyharukz pushed a commit to cybozu-go/rook that referenced this issue Jan 23, 2020
binoue pushed a commit to binoue/rook that referenced this issue Apr 10, 2020
@PushkarJ
Copy link
Member

PushkarJ commented Oct 14, 2022

/label official-cve-feed

(Related to kubernetes/sig-security#1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Development

No branches or pull requests

3 participants