Failed to mount Azure Disk as a PV when ADE is enabled #66443
Comments
k8s-ci-robot
added
needs-sig
|
/sig azure |
k8s-ci-robot
added
sig/azure
and removed
needs-sig
|
@tanaka-takayoshi thanks for the deep investigation. Then how is the PV disk mounted on VM? could you run following command to check? Thanks. /assign |
|
@tanaka-takayoshi I found the root cause, by encrpyting the vm disk, there would be a new disk |
|
There are two ways to fix this issue to support disk encryption since there would be a new disk
I would like to know your thoughts(@rootfs @karataliu) before fixing this issue, thanks. |
|
@andyzhangx Yes. That's what I want to explain. |
|
If you changed to always find a device bigger that 3 aren't you going to break situations where ADE is not involved? If the bek volume was labelled in a similar manor to root and resources disks, a udev rule could be created and it could be excluded in a similar way as is done in func listAzureDiskPath? |
no, the new data disk device would be always bigger than 3 no matter ADE is enabled or not. The data disk could always be found under |
|
To prove the original issue raised I tried the following to add BEK as a sym link in /dev/disk/azure as below. lrwxrwxrwx. 1 root root 9 Jul 17 15:32 root -> ../../sda When starting a pod with attached storage it now correctly uses /dev/sdd as the next device for the mounted storage. Having a udev environment var set as fabric_name=BEK could fix this with a simple addition of a udev rule which is exactly how the existing root and resource mounts are handled. |
|
@andyrd2405 Thanks, that would be a better solution without changing k8s code. I just verified that add one more udev rule(see below, may require reboot) in ATTRS{device_id}=="?00000000-0000-", ENV{fabric_name}="root", GOTO="azure_names" I will contact with azure team, try to make this rule as default in linux distro. |
|
@andyzhangx Thank you. Can I just confirm you tested this with just the additional UDEV rule? Does the device presented from Azure not need to have the fabric_name set first ? |
|
@andyrd2405 In my testing, insert following UDEV rule(may require reboot) would work: The |
|
@andyzhangx Yes I can confirm I added the UDEV rule to my VM's and stopped to deallocate and then start. Once the machine is up I can see the below under /dev/disk/azure lrwxrwxrwx. 1 root root 9 Jul 23 08:17 root -> ../../sda I will try a further mount / pod test but I think it should be good. |
|
@andyzhangx Just to confirm I have retested with POD's and mounted storage and all now works as expected after the UDEV rule change. Thank you for the help and support. |
|
@andyrd2405 that's great. Is it possible to file an issue to make that UDEV rule into RHEL image on Azure? @pdwyer I am not sure who is the right contact at this moment. |
|
@andyzhangx looks like rules are provided in WALinuxAgent So I assume that is where we need to get the issue filed. |
Recently we are trying to do azure disk encryption on Ubuntu & RedHat and found that a UDEV rule is missing, the new UDEV rule is like following in `/etc/udev/rules.d/66-azure-storage.rules`
```
ATTRS{device_id}=="?00000001-0001-*", ENV{fabric_name}="BEK", GOTO="azure_names"
```
After disk encryption:
- Without this UDEV rule
```
$ sudo tree /dev/disk/azure
âââ resource -> ../../sdb
âââ resource-part1 -> ../../sdb1
âââ root -> ../../sda
âââ root-part1 -> ../../sda1
âââ scsi1
âââ lun0 -> ../../../sdd
```
- With this UDEV rule
```
$ sudo tree /dev/disk/azure
âââ BEK -> ../../sdc
âââ resource -> ../../sdb
âââ resource-part1 -> ../../sdb1
âââ root -> ../../sda
âââ root-part1 -> ../../sda1
âââ scsi1
âââ lun0 -> ../../../sdd
```
All azure os, resource and bek disks should be under `/dev/disk/azure` directory, that's a rule.
The item `BEK -> ../../sdc` is very important in kubernetes, without this item, new data disk could not be recognized, details info could be found here: kubernetes/kubernetes#66443 (comment)
|
@pdwyer Thanks, I created a PR there: Azure/WALinuxAgent#1287 |
Recently we are trying to do azure disk encryption on Ubuntu & RedHat and found that a UDEV rule is missing, the new UDEV rule is like following in `/etc/udev/rules.d/66-azure-storage.rules`
```
ATTRS{device_id}=="?00000001-0001-*", ENV{fabric_name}="BEK", GOTO="azure_names"
```
After disk encryption:
- Without this UDEV rule
```
$ sudo tree /dev/disk/azure
âââ resource -> ../../sdb
âââ resource-part1 -> ../../sdb1
âââ root -> ../../sda
âââ root-part1 -> ../../sda1
âââ scsi1
âââ lun0 -> ../../../sdd
```
- With this UDEV rule
```
$ sudo tree /dev/disk/azure
âââ BEK -> ../../sdc
âââ resource -> ../../sdb
âââ resource-part1 -> ../../sdb1
âââ root -> ../../sda
âââ root-part1 -> ../../sda1
âââ scsi1
âââ lun0 -> ../../../sdd
```
All azure os, resource and bek disks should be under `/dev/disk/azure` directory, that's a rule.
The item `BEK -> ../../sdc` is very important in kubernetes, without this item, new data disk could not be recognized, details info could be found here: kubernetes/kubernetes#66443 (comment)
|
new UDEV rule has been added into wala agent rules, close this issue now fixed in WALinuxAgent-2.2.32 |
|
@andyzhangx: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@tanaka-takayoshi @andyzhangx |
|
The new data disk won't be encrypted even you specify as |
|
ok thanks @andyzhangx - What's your experience with that? When I run |
|
@gitphill if you use |
/kind bug
What happened:
kubernetes failed to mount directory in Azure Disk as a PV when Azure Disk Encryption is enabled in node host.
What you expected to happen:
Successfully to mount the disk.
How to reproduce it (as minimally and precisely as possible):
I confirmed the issue in AKS (kubernetes 1.9.9) and OpenShift Enterprise 3.9 on Azure. Here is how to reproduce with AKS.
Create a single node AKS.
Then enable ADE for created VM by AKS.
AKS creates the default storage class automatically.
Then create a PVC and pod.
Then I could see the failed event.
Anything else we need to know?:
I confirmed the issue only on AKS and OpenShift, but I wonder this could happen on kubernetes on Azure with Azure Cloud Provder enabled.
I checked the closed issue #57070 . I'm wondering the issue is caused because kubernetes doesn't consider
/dev/sdchas been already mounted as a BEK volume by ADE.kubenertes excludes disks used by azure as resource and OS root in /dev/disk/azure. However, BEK Volume doesn't show up under /dev/disk/azure. I expect this is the reason.
https://github.com/kubernetes/kubernetes/blob/v1.9.9/pkg/volume/azure_dd/azure_common_linux.go#L31-L47
Environment:
kubectl version):Cloud provider or hardware configuration:
Azure (AKS)
OS (e.g. from /etc/os-release):
uname -a):Install tools:
No
Others:
The text was updated successfully, but these errors were encountered: