Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
CoreOS 1855.4.0 AWS EBS Mount Lockup #2511
Ignition crashes system if storage.filesystem is specified
Convert to userdata
Container Linux Version
At minimum format my block device
System does not boot
Worked before on older CoreOS-stable-1688.5.3-hvm (ami-a2b6a2de)
Manually booting without CT/Ignition allows manual format/mounting of /dev/sdb (mounting by label is also no problem)
This happens for me as well on the latest gen instances. Switching instances from t2 and t3 results into the system hanging on a systemd unit that's waiting for device "/dev/xvdg".
Perhaps this has something to do with switching to the NVME names that t3 instances do.
Fetched one of the logs from the machines, I see lots of these messages:
It eventually times out:
Creating a VM as a t2 instance and then changing the instance type to t3 works, I actually see the symlinks working:
Creating a fresh T3 instance results in the hanging behaviour
Okay looks like a mismatch between assigning EBS to /dev/sdb in the AWS console and /dev/xvdb appearing in linux.
Updating the config from sdb -> xvdb finishes the boot.
Is there already a ticket for sdb vs xvdb? I think on some systems (can't remember) /dev/sdb shows up instead.
As a follow on thought, it seems sometimes EBS volumes (add-on disks on AWS) show up as /dev/sdb and sometimes as /dev/xvdb. That makes ignition scripts fail if mismatched and makes it difficult to use the same script on various servers.
Is there any guidance on /dev/sdb vs /dev/xvdb going forward in coreos? Perhaps following such guidance would have prevented this ticket.
@lucab - Yes I too have seen where my EC2 launch spec and coreos device path mismatch. I think it's related to this item on the same page as the link you provided:
So it seems that the kernel configuration (and thus CoreOS) may also have some interaction. So it's not just "It's AWS", but "It's AWS and How CoreOS interact" which is why I'm bringing up this question. :)
Anyways, yes I read the other thread you linked to. I can share with you that on device mapping I've abandoned Ignition and resorted to a series of shell scripts to format and mount things properly (despite instance type changes). I'm not sure if that's the long term way to do it, but I'm pretty sure the discussion either way is lengthy and has plenty of ideology to go with it :)
When it comes to AWS totally changing device paths for NVMe, even I have trouble justifying automagic resolution with ignition.
It's definitely a usability discussion rather than a bug discussion.