FR: PXE-booting "blank" VMs

[aspiers]

AFAICS it is not currently possible to write a Vagrantfile which persuades Vagrant to boot a "blank" VM, i.e. one with a fresh, uninitialized virtual disk suitable for PXE-booting, without involving any Vagrant box:

vagrant up controller1
Bringing machine 'controller1' up with 'libvirt' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:
* A box must be specified.

If that is true, please consider this a feature request, and if not, please consider it a documentation bug that it is not more obvious how to do it ;-)

The use case I need it for is a multi-machine Vagrantfile where the first machine boots from a box and sets up a DHCP/TFTP server which subsequent machines can PXE-boot from. In this way you could set up an entire environment of multiple machines from a single suitably-crafted box.

[mitchellh]

I believe you're correct that this isn't currently possible. Your paste uses libvirt but I believe that was in error since your text mentions VirtualBox.

I'm not interested in adding this myself since it is a very rarely requested feature, but would be happy to merge in a PR if one were to exist. Because this won't be done anytime in the short term, closing this. We generally don't keep feature requests open in issues.

[aspiers]

Thanks for the reply.

Your paste uses libvirt but I believe that was in error since your text mentions VirtualBox.

I didn't mention VirtualBox so I'm not sure where you're getting that from, but we actually need the feature for both libvirt and VirtualBox.

but would be happy to merge in a PR if one were to exist.

If you give some hints on how to best go about it, we can probably write and submit this.

Because this won't be done anytime in the short term, closing this.

That somewhat contradicts your previous sentence ;-)

We generally don't keep feature requests open in issues.

So where do you keep them? And out of curiosity, why not?

[mitchellh]

@aspiers We used to at one point and we had hundreds of open issues that were simply never going to get closed because no one was working on them. We want to avoid that scenario so that issues remain things that actually have some sort of intention of being fixed.

At this stage, I can't say I have any intention of adding this feature. It is in the "closed" history, though, so if that ever does come up, I know its there, I know who requested it, I know about it, etc.

Sorry, I misread "Vagrant box" above as "VirtualBox." Well, for libvirt, that is an external plugin so you should report that to whereever that is. I believe Vagrant gives them the framework they need to implement this without issue.

For VirtualBox (which is core), have a look at plugins/providers/virtualbox as a good starting point.

[kikitux]

Hello Adam, just for curiosity.

What is the OS you plan to deploy?

and how starting from PXE is different from a base box with a minimal setup?

Just interested to understand the business case / test case here

Thanks,
Alvaro.

On Wed, Sep 10, 2014 at 10:36 PM, Adam Spiers notifications@github.com
wrote:

AFAICS it is not currently possible to write a Vagrantfile which
persuades Vagrant to boot a "blank" VM, i.e. one with a fresh,
uninitialized virtual disk suitable for PXE-booting, without involving any
Vagrant box:

vagrant up controller1
Bringing machine 'controller1' up with 'libvirt' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:

• A box must be specified.

If that is true, please consider this a feature request, and if not,
please consider it a documentation bug that it is not more obvious how to
do it ;-)

The use case I need it for is a multi-machine Vagrantfile where the first
machine boots from a box and sets up a DHCP/TFTP server which subsequent
machines can PXE-boot from. In this way you could set up an entire
environment of multiple machines from a single suitably-crafted box.

—
Reply to this email directly or view it on GitHub
#4487.

[aspiers]

Well, for libvirt, that is an external plugin so you should report that to whereever that is. I believe Vagrant gives them the framework they need to implement this without issue.

For VirtualBox (which is core), have a look at plugins/providers/virtualbox as a good starting point.

Thanks, but I was hoping for slightly more specific design hints ;-)

For example, what should be the convention in Vagrantfile for indicating that you want a "blank" VM? It could be simply to omit specifying the box name, but that may confuse newbies who want a normal box-based appliance and simply forgot the node.vm.box = ... line. Another approach might be to allow reference to a symbol:

node.vm.box = :no_box

and add code to deal with this case.

A third approach might be to vagrant box add some kind of stub box called blank or pxe which is simply constructed from a blank .qcow2 or .vmdk. Maybe these blank boxes could be offered out of the err... box - I mean immediately available after installation? Or support something like vagrant box add --blank?

These design considerations are mostly independent of the provider being used, so they are valid questions to ask here, not just to the vagrant-libvirt guys. But it makes sense to standardize the solution so all providers have the option to support it.

Like I said, we can probably implement this - I'm part of a team which is being paid to build (amongst very many other things) a working Vagrant environment for our product. And we need it soon. So this is a real opportunity to help improve Vagrant. But I could really use some guidance on the design, otherwise we risk spending a fair amount of time building a PR, only to have it rejected due to non-conformance with some other design considerations of which we are not yet aware.

Thanks in advance!

[aspiers]

What is the OS you plan to deploy?

SLES11 and SLES12.

how starting from PXE is different from a base box with a minimal setup?

Our PXE environment is configured to install the whole OS from scratch in a very specific way. Starting with a base box is another option we are already doing, but we need both in order to demo both capabilities of the product.

[aspiers]

Hmm... do we actually even need any code changes here? Can't I just create a box with a blank disk image and call it something like pxe-16GB?

[mitchellh]

@aspiers It is possible. With VirtualBox, it'll force some things like a NAT interface on adapter 1. But otherwise it should work.

In response to further above, node.vm.box = :no_box. I added in Vagrant 1.5.0 the ability for provider plugins to say that they can operate with no box. This makes it so the box parameter is not required. I imagine if PXE environments were fully supported, this might be the mode of operation a provider goes toward. For an example of this, see the Docker provider.

If you ask more specific questions about design, I can answer them. I'm not intending on writing a full guide to Vagrant internals here. There are tons of examples (lots of providers that aren't very big) and the entire core plugins directory dog-foods the plugin API. It should be quite easy to get going.

[aspiers]

@aspiers It is possible. With VirtualBox, it'll force some things like a NAT interface on adapter 1. But otherwise it should work.

I've tried it with the libvirt plugin and it works except that the VM doesn't get PXE-booting enabled on the NICs automatically, and the boot order is also wrong. Not sure if the same problem exists with VirtualBox.

In response to further above, node.vm.box = :no_box. I added in Vagrant 1.5.0 the ability for provider plugins to say that they can operate with no box. This makes it so the box parameter is not required. I imagine if PXE environments were fully supported, this might be the mode of operation a provider goes toward. For an example of this, see the Docker provider.

Ah that's cool! I'll look into that if I get a chance.

If you ask more specific questions about design, I can answer them.

Great, thanks.

I'm not intending on writing a full guide to Vagrant internals here.

Right - and just in case it wasn't clear, I wasn't asking for one ;-)

There are tons of examples (lots of providers that aren't very big) and the entire core plugins directory dog-foods the plugin API. It should be quite easy to get going.

Sure. I was just asking for guidelines on the best way to add support for this. By the way it seems to me like it would be worth having provider-agnostic support for adding extra virtual disks, just like it is already possible to add extra virtual NICs. The most common use case would be to support data disks, but it would also allow a nice way to support PXE-booting by omitting to reference any box, combined with requesting an extra blank disk which would end up as the root disk. That way, the size of the root disk could be specified in Vagrantfile rather than be dictated by the size of the blank PXE box.

[aspiers]

In response to further above, node.vm.box = :no_box. I added in Vagrant 1.5.0 the ability for provider plugins to say that they can operate with no box. This makes it so the box parameter is not required. I imagine if PXE environments were fully supported, this might be the mode of operation a provider goes toward. For an example of this, see the Docker provider.

I guess by this you mean f2bd698 and 61ffa53. Seems like it should be trivial to add this to the libvirt plugin.

[mitchellh]

Yep, exactly.

[infernix]

@aspiers I've gotten proper PXE boot order merged in vagrant-libvirt/vagrant-libvirt#425 and I'm looking at the docker example, but don't see a trivial way to add the remaining :no_box functionality to vagrant-libvirt just yet.

I think there are at least three use cases here:

• PXE booting VMs with no disk whatsoever (yours)
• PXE booting VMs with no box but with an extra disk on which the OS gets installed over the network (mine)
• PXE booting VMs with a blank box and no extra disk on which the OS gets installed over the network (mine)

In all cases I think the behaviour should be such that Vagrant builds the VM (and optionally attaches a blank box) but then does nothing more than start it; it shouldn't even care to see what DHCP IP it was given. Not sure if the libvirt plugin can really handle that case well.

[aspiers]

Thanks! Actually my use case is the second not the first :) But there's probably someone out there who might appreciate the first (e.g. testing root over NFS environments). I agree with your assessment of the desired behaviour.

[infernix]

@aspiers if able, please test my PR @ vagrant-libvirt/vagrant-libvirt#442. Thanks!

[aspiers]

I will try to find time but unfortunately it's very unlikely to happen in the next month or so, sorry!

[darksheer]

I would like to see this capability as well. We're using the product mentioned, but have a number of use cases for allowing the provisioning of systems via vagrant straight into pxe.

[infernix]

@darksheer this is merged already in https://github.com/pradels/vagrant-libvirt, just not released yet as a new version.

[darksheer]

Great, thank you for the update. Will the same change be allowed for other providers than libvirt? VMware, VirtualBox, fusion, etc?

[infernix]

It is possible from Vagrants point of view but each provider plugin needs to implement it individually. I personally only need libvirt so you'd need to check with the plugin authors.

[darksheer]

Thanks, I'll pass along your PR as reference to the other provider plugin groups.

@mitchellh Is there any consideration in allowing a similar change for the core plugin set (virtualbox, vmware)?

[marcindulak]

I have two questions about the status of pxe booting of boxes.

At https://github.com/pradels/vagrant-libvirt#no-box-and-pxe-boot I'm reading:
"In short, VMs without a box can be created, halted and destroyed but all other functionality cannot be used.". Specifically:
...

• No provisioning scripts are ran
• No network configuration is being applied to the VM
• No SSH connection can be made
  ...
  What is the source of these limitations: vagrant, libvirt, the vagrant-libvirt plugin itself?

Second question, related to the first one:

I'm taking a different approach to pxe boot in Vagrant with Virtualbox provider: I take a standard box (e.g. puppetlabs/centos-6.6-64-nocm), perform a pxe installation ontop of it (using the old systemimager software), and use kexec to boot into the installed kernel. The installed system is functional, I can ssh to it, network settings and provisioning are applied, etc., but such VM hangs after rebooting (VBox's GUI shows black screen with a "dead" cursor). The Vagrantfile is available at https://github.com/marcindulak/vagrant-systemimager-tutorial-centos6/ . Is this hang related to some limitations of this approach?

[infernix]

@marcindulak SSH connections cannot be made because Vagrant does not do any provisioning. When a PXE box is started with no disks for example, Vagrant does not know if there is even a vagrant user or what the IP of the PXE booted box would be. So any vagrant actions that rely on ssh (provisioning) are not supported because we can't guarantee that they will work in such cases. Once the PXE node is booted and an OS is running on it, ssh outside of vagrant will work fine of course.

Can't say much about your VBox problem unfortunately.

[jamesalbert]

Late to the party here. @infernix does that mean when vagrant is "Waiting for machine to boot", that it's normal for it to fail (because it can't connect)? If so, is there any way of confirming that it actually pxe booted?

[eoli3n]

Need it too with virtualbox provider.
If it can help : https://github.com/eoli3n/vagrant-pxe

[benh57]

I got this partially working with Virtualbox. I'll turn it into a PR at some point.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.