Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2019-11255: CSI volume snapshot, cloning and resizing features can result in unauthorized volume data access or mutation #85233

Closed
msau42 opened this issue Nov 13, 2019 · 4 comments

Comments

@msau42
Copy link
Member

@msau42 msau42 commented Nov 13, 2019

Am I vulnerable?

CSI snapshot, cloning and resizing features are affected. Prior to Kubernetes 1.16, these features were all alpha and disabled by default. Starting in Kubernetes 1.16, CSI cloning and resizing features are beta and enabled by default.

These features also require CSI drivers to be installed in a Kubernetes cluster and the CSI driver also has to support those features. An unofficial list of CSI drivers and their supported features is available here, however it is best to check with the CSI driver vendor for the latest information.

Check if you have the following Kubernetes feature gates enabled:

VolumeSnapshotDataSource: alpha starting with K8s 1.12
ExpandCSIVolumes: alpha starting with K8s 1.14, beta starting with K8s 1.16
VolumePVCDataSource: alpha starting with K8s 1.15, beta starting with K8s 1.16

Check if you are using CSI drivers in your cluster. If so, the following command’s output will be non-empty:

$ kubectl get nodes -o jsonpath='{.items[*].metadata.annotations.csi\.volume\.kubernetes\.io\/nodeid}'
      {"my-csi-plugin":"kubernetes-minion-group-433q"}

Then, check the CSI driver’s pod specifications to see if they are using the following vulnerable versions of sidecars:

external-provisioner: v0.4.1-0.4.2, v1.0.0-1.0.1, v1.1.0-1.2.1, v1.3.0
external-snapshotter: v0.4.0-0.4.1, v1.0.0-1.0.1, v1.1.0-v1.2.1
external-resizer: v0.1.0-0.2.0

An example query:

$ kubectl get pods --all-namespaces -o jsonpath='{..image}' | tr ' ' $'\n' | grep "csi-provisioner\|csi-snapshotter\|csi-resizer"
      image: quay.io/k8scsi/csi-provisioner:v1.2.0

Note that the exact container image name may vary across CSI driver vendors. It is recommended to inspect the Pod specifications directly.

How do I mitigate the vulnerability?

As a short term mitigation, disable the VolumeSnapshotDataSource, ExpandCSIVolumes, and VolumePVCDataSource Kubernetes feature gates in kube-apiserver and kube-controller-manager. This will cause new PersistentVolumeClaims to be provisioned ignoring the DataSource and resizing requests will also be ignored. Note that this will cause new PVCs that are intended to be provisioned from a snapshot or clone to instead provision a blank disk.

Also, to disable taking volume snapshots, either remove the external-snapshotter sidecar from any CSI drivers or revoke the CSI driver’s RBAC permissions on the snapshot.storage.k8s.io API group.

Longer term, upgrade your CSI driver with patched versions of the affected sidecars. Fixes are available in the following sidecar versions:

external-provisioner:
v0.4.3
v1.0.2
v1.2.2
v1.3.1
v1.4.0

external-snapshotter:
v0.4.2
v1.0.2
v1.2.2

external-resizer
v0.3.0

Fixes for each of the sidecars can be tracked by:
kubernetes-csi/external-provisioner#380
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-resizer#63

How do I upgrade?

Check with your CSI driver vendor for upgrade instructions. No Kubernetes control plane or node upgrades are required unless the CSI driver is bundled into the Kubernetes distribution.

Vulnerability details

There are two different vulnerabilities impacting the same features.

When PersistentVolumeClaim and PersistentVolume objects are bound, they have bidirectional references to each other. When dereferencing a PersistentVolumeClaim to get a PersistentVolume, the impacted sidecar controllers were not validating that the PersistentVolume referenced back to the same PersistentVolumeClaim, potentially operating on unauthorized PersistentVolumes for snapshot, cloning and resizing operations.

A similar issue exists for VolumeSnapshot and VolumeSnapshotContent objects when creating a new PersistentVolumeClaim from a snapshot.

The second issue is related to the property that CSI volume and snapshot ids are only required to be unique within a single CSI driver. Impacted sidecar controllers were not validating that the requested source VolumeSnapshot or PersistentVolumeClaim specified were from the same driver processing the request, potentially operating on unauthorized volumes during snapshot, restore from snapshot, or cloning operations.

@msau42
Copy link
Member Author

@msau42 msau42 commented Nov 13, 2019

/assign

@msau42
Copy link
Member Author

@msau42 msau42 commented Nov 14, 2019

@msau42
Copy link
Member Author

@msau42 msau42 commented Nov 14, 2019

/close

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Nov 14, 2019

@msau42: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Madhu-1 added a commit to Madhu-1/rook that referenced this issue Nov 18, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Madhu-1 added a commit to Madhu-1/rook that referenced this issue Nov 19, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
mergify bot pushed a commit to rook/rook that referenced this issue Nov 19, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 49381b3)
egafford added a commit to egafford/rook that referenced this issue Nov 21, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Updated container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Nov 22, 2019
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
nizamial09-zz pushed a commit to nizamial09-zz/rook that referenced this issue Nov 25, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
zoetrope added a commit to cybozu-go/rook that referenced this issue Dec 26, 2019
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Jan 14, 2020
Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)

ref: kubernetes-sigs#411
jnaulty added a commit to jnaulty/aws-ebs-csi-driver that referenced this issue Jan 21, 2020
Changes:
- Update container image versions that have resolve the CVE according to
[kubernetes/kubernetes/issues/85233](kubernetes/kubernetes#85233)
- Update snapshotter RBAC policy
- Update resizer RBAC policy
- Updates external-provisioner RBAC policy for v1.3.1 image
- Update helm charts with updated RBAC policy

ref: kubernetes-sigs#411
kfyharukz added a commit to cybozu-go/rook that referenced this issue Jan 23, 2020
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
binoue added a commit to binoue/rook that referenced this issue Apr 10, 2020
update csi sidecar images to fix CVE-2019-11255

more info:
kubernetes/kubernetes#85233
kubernetes-csi/external-snapshotter#193
kubernetes-csi/external-provisioner#380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants