Added support for NVME Hyperdisks on Windows Nodes

[ivanLeont]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test

/kind feature

/kind flake

What this PR does / why we need it:

This PR fixes the ability to use hyperdisks/nvme drives on window's nodes. This works by having the CSI node query Google's API to extract the correct serial number of the hyperdisk and match it to the correct index/serial of the mounted disk on windows.

Which issue(s) this PR fixes:

Fixes # N/A

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ivanLeont (302d4d7, c6dc114, a9a2d90)

[k8s-ci-robot]

Welcome @ivanLeont!

It looks like this is your first PR to kubernetes-sigs/gcp-compute-persistent-disk-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gcp-compute-persistent-disk-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @ivanLeont. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivanLeont]

Adding a note here: This still works even if the GKE cluster has Workload Identity, as the Daemonset kube-system/gke-metadata-server that manages metadata access, seems to allow the SA pdcsi-node-sa (which is used by the pdcsi-node-windows Daemonset) to bypass all metadata restrictions, which compared to a user created SA has more restricted access to the metadata. I have confirmed this by creating a pod that uses the SA pdcsi-node-sa and was able to access the necessary disk data.

[mattcary]

Adding a note here: This still works even if the GKE cluster has Workload Identity, as the Daemonset kube-system/gke-metadata-server that manages metadata access, seems to allow the SA pdcsi-node-sa (which is used by the pdcsi-node-windows Daemonset) to bypass all metadata restrictions, which compared to a user created SA has more restricted access to the metadata. I have confirmed this by creating a pod that uses the SA pdcsi-node-sa and was able to access the necessary disk data.

Hmm, super interesting, this isn't expected. Can you give details of your setup? gcloud compute instances describe as well as kubectl get -o yaml for both the node and the pd csi daemonset deployment you use with this PR?

Thx.

[mattcary]

/ok-to-test

[ivanLeont]

Adding a note here: This still works even if the GKE cluster has Workload Identity, as the Daemonset kube-system/gke-metadata-server that manages metadata access, seems to allow the SA pdcsi-node-sa (which is used by the pdcsi-node-windows Daemonset) to bypass all metadata restrictions, which compared to a user created SA has more restricted access to the metadata. I have confirmed this by creating a pod that uses the SA pdcsi-node-sa and was able to access the necessary disk data.

Hmm, super interesting, this isn't expected. Can you give details of your setup? gcloud compute instances describe as well as kubectl get -o yaml for both the node and the pd csi daemonset deployment you use with this PR?

Thx.

Here are the details of the setup:

gke-metadata-server.txt
cd4-node.txt
cluster-info.txt
csi-gce-pd-node-win.yaml.txt
pdcsi-node.yaml.txt

When deploying a cluster with Workload Identity enabled, i noticed that after running ./deploy-drivers.sh, the windows node images deployed by the DS csi-gce-pd-node-win were crashing with the following error message

│ I0620 20:06:02.254719    5792 main.go:122] Operating compute environment set to: production and computeEndpoint is set to: <nil>                                                                                                                                        │
│ I0620 20:06:02.255761    5792 main.go:131] Sys info: NumCPU: 4 MAXPROC: 1                                                                                                                                                                                               │
│ I0620 20:06:02.255761    5792 main.go:136] Driver vendor version a9a2d90593f36050b197c7576b2970e8420b99e9                                                                                                                                                               │
│ I0620 20:06:02.257324    5792 safe-mounter_windows.go:66] using CSIProxyMounterV1, API Versions Disk: v1, Filesystem: v1, Volume: v1                                                                                                                                    │
│ F0620 20:06:02.267481    5792 main.go:271] Failed to set up metadata service: failed to get machine-type: metadata: GCE metadata "instance/machine-type" not defined  

https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/cmd/gce-pd-csi-driver/main.go#L271

I then enabled the default CSI and noticed that the windows image there was not crashing at that point, so I created two pods in both namespaces with different SA's

This pod, metadata-test, deployed in ns gce-pd-csi-driver uses the sa csi-gce-pd-node-sa-win which is is created using the ./deploy-driver.sh (I added the required bindings and annotations to work with Workload Identity)

PS C:\> Invoke-RestMethod -Uri 'http://metadata.google.internal/computeMetadata/v1/instance' -Headers @{'Metadata-Flavor'='Google'}
attributes/
hostname
id
name
service-accounts/
zone


PS C:\> Invoke-RestMethod -Uri 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts' -Headers @{'Metadata-Flavor'='Google'}
default/
sa_number@project_id.iam.gserviceaccount.com/


PS C:\> 

In this pod, metadata-test, deployed in ns kube-system uses the sa pdcsi-node-sa which comes in by default when enabling the default CSI

PS C:\> Invoke-RestMethod -Uri 'http://metadata.google.internal/computeMetadata/v1/instance/' -Headers @{'Metadata-Flavor'='Google'}
attributes/
cpu-platform
credentials/
description
disks/
gce-workload-certificates/
guest-attributes/
hostname
id
image
licenses/
machine-type
maintenance-event
name
network-interfaces/
partner-attributes/
preempted
remaining-cpu-time
scheduling/
service-accounts/
shutdown-details/
tags
virtual-clock/
zone

PS C:\> Invoke-RestMethod -Uri 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts' -Headers @{'Metadata-Flavor'='Google'}
240471850237-compute@developer.gserviceaccount.com/
default/

PS C:\> 

So when I used a webhook to swap the original image in the DS kube-system/pdcsi-node-windows my custom image was able to work fine. It was also observed that in the gke-metadata-server DS, there is a field --passthrough-ksa-list that has kube-system:pdcsi-node-sa as one of the SA's allowd passthrough

Let me know if you need any more details! 😃

[mattcary]

It was also observed that in the gke-metadata-server DS, there is a field --passthrough-ksa-list that has kube-system:pdcsi-node-sa as one of the SA's allowd passthrough

ahhh interesting we didn't know the security team did that :-P

Let us figure out if this is the right way to go, then

[julianKatz]

/retest

[julianKatz]

TLDR: Let's merge this, excluding the changes in resizefs_windows.go

1. GKE has pursued a fix for the resize issue separate of this PR. That has seen a multi-step investigation.

2. Let's remove the "error skipping" logic in resizefs_windows.go from this PR. It's more work to re-validate this PR than rely on existing work as far as the resize issue is concerned.

3. I've made a version of this PR without the changes to resizefs_windows.go and tested them on a C3 family Windows machine. That connects a disk via NVMe. The pod got to Running, signifying a successful mounting of the disk and the solving of the issue.

4. If the windows tests pass for this PR, that serves as validation that the change does not break Windows SCSI support. Currently, the windows tests (inherited from k/k) are run solely on a GCE machine type that uses SCSI to connect to disks.

5. Thus, our next step should be to remove those "error skip" clauses in resizefs_windows.go from this PR and merge it.

[msau42]

/approve

Thank you for your contribution and thorough investigation!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivanLeont, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@ivanLeont: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2022	302d4d7	link	false	/test pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2022

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[tonyzhc]

/cherry-pick release-1.19

[k8s-infra-cherrypick-robot]

@tonyzhc: new pull request created: #2122

In response to this:

/cherry-pick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.