Support for Vultr platform

[dghubble]

Opened per ignition#918

Vultr is a cloud (and bare-metal) hosting provider that may be reasonable for Fedora CoreOS to assemble images for. Vultr allows uploading ISOs by URL and iPXE so its been flexible to develop and hack with. They accept raw images similar to GCP. Vultr serves user-data for cloud-init, but its free-form and I've used it to serve Ignition just fine. Vultr provides CoreOS images themselves (cloud-init only), though it was not a CoreOS publish target. The platform's willingness for folks to use any raw image they can boot, to me, means its pretty unlikely there are required agents or any of that sort of thing.

I'd like to be able to build a raw image for Vultr that knows about their user-data (i.e. platform-id vultr). Such a raw image could be uploaded and used directly be end users. Possible steps:

• allocate the platform ID vultr (platforms: allocate ID for vultr fedora-coreos-docs#37)
• ignition support for vultr (providers/vultr: Add Vultr provider ignition#918)
• coreos-assembler produces raw image with platform vultr (src/cmd-buildextend-vultr: prepare a Vultr raw image coreos-assembler#1083)
• afterburn (platform: add support for vultr afterburn#343)
• build images via pipeline (Jenkinsfile: Add Vultr image build to pipeline fedora-coreos-pipeline#231)
• booting on vultr docs (provisioning: Add getting started on Vultr tutorial fedora-coreos-docs#71)
• fedora-coreos-stream-generator (Add Vultr disk artifacts to stream metadata fedora-coreos-stream-generator#10)
• coreos-meta-translator (coreos-meta-translator: Add Vultr to trans.py fedora-coreos-releng-automation#108)
• website (https://pagure.io/fedora-web/websites/c/a853006d8c5887ea493137b78b8ff3e5b461f9d2?branch=master)

Hacks today:

• Boot "installer" ISO - drops to emergency shell so you can install to disk with an ignition shim (ISO installer was deprecated)
• Boot "live" ISO - boots, but not ideal but you can't ssh in to add an ignition shim :(
• iPXE - you can use iPXE with their cloud VMs (though its not a great fit and better suited for bare-metal). Currently panics on boot (separate bug, discuss separately)

Background

I've been running Fedora CoreOS on Vultr for a few months as part of some personal experimental infrastructure. I used the installer ISO approach to prepare a snapshot (rather big, but works) after disk install and it operates alright, at least for Kubernetes node use cases.

Why

Though Vultr is not a major cloud, its quite flexible and developer-friendly. I'd consider Vultr (and DigitalOcean) in a separate category since support there is geared toward developer affection and interesting experiments, which spawns blog posts, tutorials, and just general good vibes. I think its an under-estimated value, and Fedora CoreOS could gain if partial or full support was developed.

Note: I don't have any affiliation with Vultr, just using it personally

[dghubble]

I've only had a bit of personal time, but was able to get coreos-assembler (quite different from CoreOS SDK) and build a vanilla image. Seems like I'd need the scripting to build a raw image similar to qemuvariant but specify the platform id to Ignition and bump Ignition somewhere, and then I could upload and test it out.

[bgilbert]

The live ISO should boot to a shell prompt on console, similar to the installer ISO failure case. I'm not sure I understand the problem you're having there; could you elaborate?

Adding the platform SGTM. We should certainly add Afterburn support in that case; it's not essential for boot but it's functionality we should provide consistently on all platforms.

[dghubble]

You're right, the current live ISO does automatic login. I must be recalling some prior build, I can't reproduce being stuck at a login prompt.

One complication with the live ISO approach is the RAM requirement. With the installer ISO, you can boot the $5 instance (1GB, 25GB disk), disk install with a tweaked ignition config to point to Vultr user-data, and snapshot for a 25GB snapshot (not great, but meh). The live ISO needs at least the larger instance (2GB, 50GB disk) and the snapshot ends up being 50GB. Arguably, Vultr could do disk snapshots differently or I could manipulate those raw images, but I know this is getting out of hand and isn't an ideal usage pattern.

That led to wanting to add Vultr awareness to Ignition and build the desired image directly.

[lucab]

I agree having directly usable cloud images is a better path forward.

It sounds like the platform provides network configuration via DHCP, right? In that case, Afterburn is not in the hot path. I've opened a ticket there to add support for SSH keys and attributes, which can be done at any later point.

I've updated the top-comment with more ticket references.

So far though I haven't found any docs regarding the disk image format. That is, do they maybe support some compressed format, or perhaps VM specific ones (qcow/vmdk/etc)?

[dghubble]

I don't find a docs page, but from the Vultr "snapshot" dashboard (separate from their ISO upload dashboard) where an end-user provides a URL for Vultr to go fetch a raw image:

- Stored snapshots are currently free - pricing subject to change.
- We recommend using DHCP for networking. By default, Vultr instances are configured to use DHCP.
- Snapshots can only be restored to equal or bigger disks. If there is a single partition, it will be automatically expanded.
- Snapshot must be in RAW format (no VMDK, QCOW2, etc)
- Maximum uploaded snapshot size is 150 GB
- Snapshot must support VirtIO Ethernet/Disk Controllers.

[jlebon]

This was discussed in the community meeting today. There was no major opposition since it seems rather straightforward to support. So 👍 from me!

[dghubble]

I was able to build a Vultr image (with recent ignition and ignition-dracut) and use it for instances on Vultr via custom snapshots. 🙌

Custom override testing seems quite nice once the pattern is known (mentioned below):

# ignition
make install DESTDIR=/home/user/fcos/overrides/rootfs
# ignition-dracut
make install DESTDIR=/home/user/fcos/overrides/rootfs
# fcos
cosa build
cosa buildextend-vultr

So after the next Ignition release / bohdi update, I believe cosa buildextend-vultr images will work for folks who choose to make em.

[lucab]

@dghubble great news!

I guess a parallel path to explore would be to check with Vultr folks whether they have any interest in directly offering Fedora CoreOS templates. They currently seem to have CoreOS Container Linux templates, which would be EOL anyway before the end of 2020.

/cc @ddymko @Oogy

[ddymko]

Hey all 👋

We are definitely interested!

I'll create an internal ticket to add in support for Fedora CoreOS snapshots and I'll update this ticket with the progress of that.

If there is anything else we can help with please us know.

[ddymko]

Hey @lucab @dghubble

I just spoke to the image team they are shooting for a mi march release for a ton of new snapshots which include Fedora CoreOS.

They let me know if you would like an early preview build of the snapshot we can add this to your Vultr accounts. This way you can do some sanity testing against the images we are going to be releasing.

Let me know if you guys are interested!

[dghubble]

I'm actively using my test build and am happy to try the Vultr previews (filed #DSB-47CQQ).

[ddymko]

Hey @dghubble

The preview snapshot should be ready today and it will get added to your account as a deploy-able snapshot.

[dghubble]

It may be useful to have releases for ignition and ignition-dracut that contain the merged patches, to eliminate the temporary go build and override step mentioned above. It works, but I suspect it may be more difficult for Vultr to create their image until then.

ignition-2.1.1-5.git40c0b57.fc31.x86_64 seems too old.

[dustymabe]

Agreed. I think there are some other patches in ignition (coming soon) that we'll want so we should be cutting a release soon.

[dghubble]

With the new ignition and ignition-dracut releases, I built a new vultr image and now use it in prod:

cosa build
cosa buildextend-vultr
# produced fedora-coreos-31.20200327.dev.0-vultr.x86_64.raw <- date at time of testing

I also tested one preview image from Vultr folks and they're working on getting their build process established I believe.

[summatix]

I was able to get Fedora CoreOS running on Vultr using coreos-installer. Seems to be working well so far.

[jlebon]

I added some more checkboxes to the original issue description for a few items more items that are needed before it's fully plumbed through.

[dghubble]

I can try to look into those missing plumbing items next week. Separately, I've done some testing with some snapshots with Vultr folks and that's looking promising. I'll let them chime in if they like or about remaining questions/hurdles.

[dghubble]

Missing plumbing based on @jlebon's links to prior examples, thanks:

• Add Vultr disk artifacts to stream metadata fedora-coreos-stream-generator#10
• coreos-meta-translator: Add Vultr to trans.py fedora-coreos-releng-automation#108

[lucab]

@dghubble @summatix @ddymko on a topic similar to #538, do machines on Vultr get their hostname from DHCP or do we need to wire it from the instance metadata?

[summatix]

I'm not sure how machines get their hostname. By default it seems machines have a hostname of vultr.guest. When I install FCOS the hostname is automatically set to [PUBLIC IP].vultr.com.

[dghubble]

Checking an instance, the hostname seems to be set early on in NetworkManager, I don't find a lease file like in #538 so I'm not sure.

NetworkManager[816]: <info>  [1592466207.7101] hostname: hostname: using hostnamed
NetworkManager[816]: <info>  [1592466207.7104] hostname: hostname changed from (none) to "ams"
...
NetworkManager[816]: <info>  [1592466207.7271] dhcp-init: Using DHCP client 'internal'

Vultr accepts instance hostnames at creation via the UI or API and I do see those names set.

The hostname is also available via the metadata service at http://169.254.169.254/metadata/meta-data/hostname.

[lucab]

Thanks. My guess here is that this sounds like DHCP is not offering the hostname and NM is setting it based on reverse DNS.

[dustymabe]

Checking an instance, the hostname seems to be set early on in NetworkManager, I don't find a lease file like in #538 so I'm not sure.

You don't see any files at /var/lib/NetworkManager/internal-*lease ?

[dghubble]

No lease files or domain in nmcli device show eth0

ls /var/lib/NetworkManager/
NetworkManager-intern.conf  secret_key  timestamps

There are PTR records, but they resolve to names that don't seem useful to me (given the address is known).

66.xx.yy.zz.in-addr.arpa. 3600 IN     PTR     zz.yy.xx.66.vultr.com.

Seems like systemd may be setting this early on accroding to dmesg.

[   14.140851] systemd[1]: Set hostname to <ams>.
[   14.144278] systemd[1]: Initializing machine ID from KVM UUID.
[   14.375048] systemd[1]: Configuration file /usr/lib/systemd/system/ignition-firstboot-complete.service is marked executable. Please remove executable permission bits. Proceeding anyway.

[dustymabe]

hey @ddymko - any chance we could get some accounts on vultr for Fedora so we can run some tests on some features we're adding? find me in #fedora-coreos on freenode if you want to chat higher bandwidth.

[ddymko]

@dustymabe sure thing! Create an account on vultr.com and create a support ticket. I'll get it squared away for you.

[dustymabe]

Thanks @ddymko - created an account. Will wait for more info from the partner team.

[summatix]

The new native Fedora CoreOS server type is working well for me. Makes it much easier to get started with FCOS on Vultr.

The only issue I'm facing is in regards to private networking. If I set the /etc/NetworkManager/system-connections/private-net.nmconnection file statically via ignition then everything works as expected. The trouble is, the contents of this file cannot be completely known until after the server is provisioned (since Vultr automatically assigns an available private IP). When I was using coreos-installer I was able to call upon Vultr's metadata API in order to dynamically generate this file for ignition. Now the ignition is set statically via userdata before provisioning.

I was hoping Fedora CoreOS would be able to pick up the private networking automatically (e.g. via DHCP). Instead, if I don't configure private-net.nmconnection statically via ignition while private networking is enabled, the NetworkManager service fails to start up:

$ journalctl -u NetworkManager-wait-online
Starting Network Manager Wait Online...
NetworkManager-wait-online.service: Main process exited, code=exit>
NetworkManager-wait-online.service: Failed with result 'exit-code'.
Failed to start Network Manager Wait Online.

[dghubble]

Ignition is a first-boot initramfs disk manipulation tool, for writing units, network configs, etc that are acted upon later, in userspace. I think your question is about the particular network config, static assignment vs DHCP assignment. For static assignment you'd need to reserve an IP with Vultr beforehand, for DHCP (default) that works for the public interface, but I recall Vultr was still fleshing out the details for private network.

Btw, the Vultr "native Fedora CoreOS" images you mention are built by Vultr. Maybe not quite be the same as the (new) build artifacts. I'm not sure what Vultr's final plan was (e.g. fetch FCOS published images or custom build).

[lucab]

Self-note:

• the current images on Vultr are a development build (32.20200618.dev.0) marked as stable but not following auto-updates for that stream
• I do see lease files at /var/lib/NetworkManager/*.lease, no hostname in there
• NM logs have a policy: set-hostname: set hostname to '136.244.81.212.vultr.com' (from address lookup)

[lucab]

Hey @ddymko, we are assembling the final bits of this and we realized we have a small impedance mismatch on how the snapshot create-url API works, see coreos/coreos-assembler#1595.

Would it be possible to augment the snapshot-by-URL API to optionally get these additional parameters from the client?

• a compression algorithm (in our case, xz), with your backend doing a decompression step
• a digest (in our case, SHA-256), with your backend doing an integrity check before decompressing/importing

[ghost]

The only issue I'm facing is in regards to private networking. If I set the /etc/NetworkManager/system-connections/private-net.nmconnection file statically via ignition then everything works as expected. The trouble is, the contents of this file cannot be completely known until after the server is provisioned (since Vultr automatically assigns an available private IP).

@summatix, based on your note, I updated the documentation for Vultr ignition files to make clear there isn't a chicken-and-egg problem. I added the following:

---

• When you enable private networking, you may use any RFC1918 private address for your ignition files: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
• You may choose any RFC1918 address, as long as there are no conflicts with your other instances at that location.
• Private networks can not communicate between locations, regardless of IP addressing. For example, server instances in Miami can not see private networks in Dallas.
• The private IP addresses shown in the customer portal are suggestions. You are not required to use these suggested private IP addresses.
• Private networks do not have DHCP, you must manually manage your IP address space or install your own DHCP server on your private network.
• For optimal performance, we suggest setting your private network adapters' MTU to 1450 when configuring the NIC at the OS level.

[ddymko]

Hey @ddymko, we are assembling the final bits of this and we realized we have a small impedance mismatch on how the snapshot create-url API works, see coreos/coreos-assembler#1595.

Would it be possible to augment the snapshot-by-URL API to optionally get these additional parameters from the client?

* a compression algorithm (in our case, xz), with your backend doing a decompression step

* a digest (in our case, SHA-256), with your backend doing an integrity check before decompressing/importing

@lucab unfortunately this isn't something that we could add to snapshot create-url

[lucab]

coreos/afterburn#451 added support for Vultr SSH keys, hostname, and metadata attributes to Afterburn upstream (not yet part of a tagged release).

[dustymabe]

Afterburn 4.5.0 was released and hit the testing stream in the last release. Anything left to do?

[dghubble]

I was going to check it out, to replace some manual hostname detection. Am I reading the package list right? https://getfedora.org/en/coreos?stream=testing says 32.20200809.2.1 is the latest at this time and has afterburn-4.4.2, same for next.

[dustymabe]

@dghubble - correct. We've got a round of releases in progress now and those should have afterburn 4.5.0.

Sorry I realize my earlier statement was incorrect. Afterburn 4.5.0 will be in the testing release that should go out tomorrow.

[dghubble]

Looks goood to me on f32.20200824.2.0 testing.

I think that completes all the items for Fedora CoreOS on vultr, awesome!