New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS EBS] NVMe udev rename rules #2399

Closed
jalaziz opened this Issue Apr 9, 2018 · 14 comments

Comments

Projects
None yet
6 participants
@jalaziz

jalaziz commented Apr 9, 2018

Issue Report

Feature Request

Environment

AWS CoreOS 1688.5.3 HVM on m5.* or c5.* instances

Desired Feature

Add udev symlink rules to map NVMe devices to traditional xvd[a-z] device names.

Other Information

With the newer m5 and c5 instances, EBS volumes show up as NVMe devices. AWS Linux provides built-in udev rules that symlink the NVMe devices to their equivalent /dev/sd[a-z] naming. This keeps things consistent with the older naming rules and matches what is configured in EBS block device mappings provided when launching the instance.

It would be great if CoreOS could provide similar rules. This would allow systemd mounts to work across all EC2 instance types without special hacks or relying on fixed device names. AWS Linux handles this with the help of a python script named ebsnvme-id that read EBS information from the NVMe device. I realize python is not installed on CoreOS, but the script could be rewritten to provide the basic functionality needed for udev renaming.

More information can be found here:
kubernetes-incubator/kube-aws#1048
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html

An example udev rule without using the python script can be found here: https://github.com/oogali/ebs-automatic-nvme-mapping

I can't seem to find the python script publicly, but it's available on the Amazon Linux AMI and is licensed under the Apache 2.0 license. I've copied the current udev rules and scripts from the Amazon Linux AMI here: https://gist.github.com/jalaziz/c22c8464cb602bc2b8d0a339b013a9c4

One thing I've noticed (and has been mentioned elsewhere) is that the device name in the vendor-specific name is different depending on when and how the volume is mounted. For example, it seems that volumes mounted pre-boot do not have the /dev/ prefix in the vendor info, while volumes mounted after have the /dev/ prefix. Also, it appears that the device is not renamed to the xvd naming convention automatically.

@lucab

This comment has been minimized.

Member

lucab commented Apr 9, 2018

Thanks for the report. We are already collecting a few cloud-storage specific udev rules, so I think we should also add the AWS NVMe ones there too.

@jalaziz

This comment has been minimized.

jalaziz commented Apr 9, 2018

Until support is added to CoreOS, I've created a set of systemd services and a udev rule that works around the issue based on the resources I listed above: https://gist.github.com/jalaziz/bcfe2f71e3f7e8fe42a9c294c1e9279f

@venezia

This comment has been minimized.

venezia commented Apr 10, 2018

I'm also having this issue with coreos alpha and stable. @jalaziz scripts do identify the intended partition. Would be great to have this fixed within coreos. It seems reasonable to assume the OS would respect the wishes of the user to have /dev/sdk be the partition (for scripting purposes) rather than some haphazardly assigned nvme#n# partition.

core@whatever ~ $ cat /proc/partitions 
major minor  #blocks  name

 259        0    8388608 nvme1n1
 259        1    8388608 nvme2n1
 259        2    8388608 nvme0n1
core@whatever ~ $ sudo ./nvme.sh /dev/nvme1n1 
sdg xvdg
core@whatever ~ $ sudo ./nvme.sh /dev/nvme2n1 
sdk xvdk
@nielssorensen

This comment has been minimized.

nielssorensen commented Jun 4, 2018

You realize the repo you listed, https://github.com/oogali/ebs-automatic-nvme-mapping, has an MIT license, correct? I believe if anyone wishes to use it, the license must follow the code; like if you are merging it into your project. Or copying it as the case may be.

@r7vme r7vme referenced this issue Jun 7, 2018

Merged

Fix volume attachments while upgrade (v12 v13) #999

3 of 3 tasks complete
@jalaziz

This comment has been minimized.

jalaziz commented Jun 11, 2018

Thanks for the note @nielssorensen. I overlooked that. I've updated my gist to include the appropriate copyright notices and licenses.

@lucab

This comment has been minimized.

Member

lucab commented Jun 28, 2018

Udev rules and helper landed in coreos/init#268.
I added this to the GH board for the next alpha, bugfix is at coreos/coreos-overlay#3309.

@lucab

This comment has been minimized.

Member

lucab commented Jul 9, 2018

This has been released as part of CL 1828.0.0 (current alpha).

@zyclonite

This comment has been minimized.

zyclonite commented Sep 11, 2018

r5.* have the same issue

@zyclonite

This comment has been minimized.

zyclonite commented Sep 11, 2018

@lucab tested against 1883.0.0 with r5.* instances and it looks solved, although when i compare t2.* instances where the volumes get mapped to /dev/xvd* against r5.* where i get only /dev/sd*... which is again not consistent or am i missing something?

@lucab

This comment has been minimized.

Member

lucab commented Sep 11, 2018

@zyclonite you can look directly into nvme id-ctrl, but I think that difference is coming from AWS itself.

@zyclonite

This comment has been minimized.

zyclonite commented Sep 11, 2018

@lucab true, but on one instance type the ebs device is exposed as nvme and on the other as xvd or sd... wouldn't it make more sense to map ebs devices so that the same names can be used with the same single container linux config independent of the instance type?

@jalaziz

This comment has been minimized.

jalaziz commented Sep 11, 2018

The original workaround I proposed created symlinks for both variations /dev/xvd* and /dev/sd* because of the inconsistency from AWS. It's a bit unfortunate, but it allows us to configure everything based of a single naming convention without worrying what AWS is going to choose.

@lucab

This comment has been minimized.

Member

lucab commented Sep 18, 2018

I understand it is unfortunate and that the other script was blindly creating multiple device names, but that is not a correct thing to do in distribution vendor rules. Those device names don't really exist and in future may clash with other device naming choices by AWS.

AWS documentation explicitly mentions that device names can differ wildly according to multiple parameters: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

@frittentheke

This comment has been minimized.

frittentheke commented Sep 27, 2018

If you feel reusing device names that could exist regularly (though thats unlikely the case on the same instance), why not create new, artificial names like

/dev/clouddisk1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment