Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for linked clones for VirtualBox. #4484

Merged
merged 8 commits into from Oct 6, 2015

Conversation

mpoeter
Copy link
Contributor

@mpoeter mpoeter commented Sep 10, 2014

This allow creation of linked clones using the VirtualBox 4.3 provider:

config.vm.provider "virtualbox" do |vb|
    vb.use_linked_clone = true
end

Instead of creating a fresh machine via importing the ovf file a master VM is created once (via ovf import) and the clones are created based on this master VM. The id of the master VM is stored in a file "master_id" in the directory of the according box.

There are a few bits missing like preventing concurrent creation of master VMs for the same box, as well as maintaining the lifetime of the master VM (when shall it be destroyed and who is responsible for it?).

Also the code might need a few adaptions since this is the first time I wrote some Ruby.

However, I thought this might be a good starting point for a discussion about how this feature could be added to Vagrant.

@npf npf mentioned this pull request Oct 6, 2014
@roidelapluie
Copy link

👍

@Jcunning
Copy link

This works great for me except after every clone it tries to re-import the master.

I believe this may be resolved by returning after determining that the master has already been imported:

env[:master_id] = master_id_file.read.chomp if master_id_file.file?          
if env[:master_id] && env[:machine].provider.driver.vm_exists?(env[:master_id])
  # Master VM already exists -> nothing to do - continue.
  return @app.call(env) 
end          

env[:master_id] = master_id_file.read.chomp if master_id_file.file?
if env[:master_id] && env[:machine].provider.driver.vm_exists?(env[:master_id])
# Master VM already exists -> nothing to do - continue.
@app.call(env)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appears to need a return here to avoid falling through.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return @app.call(env)

@sciurus
Copy link
Contributor

sciurus commented Nov 1, 2014

This is a great idea, we do something similar by default in vagrant-libvirt. Regarding

maintaining the lifetime of the master VM (when shall it be destroyed and who is responsible for it?)

my suggestion is that for each box you destroy the master VM, if it exists, when vagrant box remove is run. I would expect that to fail if any VMs still existed that were linked clones of the master VM.

@mpoeter
Copy link
Contributor Author

mpoeter commented Nov 10, 2014

Thanks for the feedback and corrections!

my suggestion is that for each box you destroy the master VM, if it exists, when vagrant box remove is run. I would expect that to fail if any VMs still existed that were linked clones of the master VM.

Thats exactly what I thought, but that means I would have to track which VMs are linked to the box/master VM - something that is not done by vagrant at the moment (at least to my knowledge).

@sciurus
Copy link
Contributor

sciurus commented Nov 10, 2014

at means I would have to track which VMs are linked to the box/master VM

That depends on how Virtualbox handles a request to delete a VM that has others VMs linked to it. If Virtualbox throws an error and prevents you from doing that, vagrant shouldn't have to track the links itself.

@mpoeter
Copy link
Contributor Author

mpoeter commented Nov 21, 2014

That would be a possibility... I'll try to implement this when I have the time. Unfortunately I'm quite busy with other stuff right now.


env[:ui].info I18n.t("vagrant.actions.vm.clone.importing", name: env[:machine].box.name)

#TODO - prevent concurrent creation of master vms for the same box.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will want to handle this before we can merge. The easiest solution (although not the most performant) is to lock everything up to line 16 in the global lock. That way a master is created one at a time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another alternative is to print a warning message and we can fix it up in another PR

@sethvargo
Copy link
Contributor

Hi @mpoeter

Thank you for the Pull Request. Overall I think this is a step in the right direction, especially for an initial pass. I made a few comments, but I would be in favor of merging this after those comments are fixed up and then we can address individual issues as subsequent PRs. I know you said your time is a bit limited, so I think that's a good compromise as well for you 😄.

If you wouldn't mind taking a look at those comments and let me know what you think.

@mpoeter
Copy link
Contributor Author

mpoeter commented Jun 1, 2015

Hi, thanks for the feedback!
I completely forgot about this because I'm still pretty slammed with other work, but I'll try to fix these issues within the next few weeks so you can merge it!

Deleting the lock file can fail when another process is currently trying to acquire it (-> race condition).
It is safe to ignore this error since the other process will eventually acquire the lock and again try to delete the lock file.
@mpoeter
Copy link
Contributor Author

mpoeter commented Jun 3, 2015

Hi,

I (hopefully) fixed all the open issues.

What's still missing is removing the master VM when the box is deleted, but we can address this in a subsequent PR.

Cheers,
Manuel

@sethvargo
Copy link
Contributor

@mpoeter awesome, thank you!

Since this is a pretty exciting new feature, would you like to write the documentation for it? All the docs live in the website/ folder of this repository 😄. If not, I can write them, just let me know, since I know time is limited.

@mpoeter
Copy link
Contributor Author

mpoeter commented Jun 5, 2015

Sure, I think I can spare a few minutes in the next days...

@mpoeter
Copy link
Contributor Author

mpoeter commented Jun 8, 2015

I have just pushed my changes - maybe you can take a look at them.

<div class="alert alert-info">
<strong>Note:</strong> the generated master VMs are currently not removed
automatically by Vagrant. This has to be done manually. However, a master
VM can only be remove when there are no linked clones connected to it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be "can only be removed", typo?

@lvillani
Copy link

Works admirably well with VirtualBox 5 and a couple of Windows boxes. The first time I ran vagrant up though I got this error:

Bringing machine 'default' up with 'virtualbox' provider...
/Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `initialize': No such file or directory - /Users/lvillani/.vagrant.d/data/lock.lvillani/win2008r2.lock (Errno::ENOENT)
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `open'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `block in lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:516:in `lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:491:in `lock'
[...]

I worked around it by creating the directory /Users/lvillani/.vagrant.d/data/lock.lvillani manually. If you want to give an additional look to this problem I can post the full stack trace and do some additional testing.

@lvillani
Copy link

I'd like to add that I did not get the aforementioned error if vb.use_linked_clone was false or unset.

@mpoeter
Copy link
Contributor Author

mpoeter commented Jul 27, 2015

I might have an idea what's causing this. Could you provide a full stack trace?
What's the name of your box?

@lvillani
Copy link

The (private) box is seen by vagrant box list as lvillani/win2008r2.

And here is the full log:

➤ vagrant up windows
You appear to be running Vagrant outside of the official installers.
Note that the installers are what ensure that Vagrant has all required
dependencies, and Vagrant assumes that these dependencies exist. By
running outside of the installer environment, Vagrant may not function
properly. To remove this warning, install Vagrant using one of the
official packages from vagrantup.com.

Bringing machine 'windows' up with 'virtualbox' provider...
/Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `initialize': No such file or directory - /Users/lvillani/.vagrant.d/data/lock.lvillani/win2008r2.lock (Errno::ENOENT)
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `open'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `block in lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:516:in `lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:491:in `lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/plugins/providers/virtualbox/action/import_master.rb:16:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/plugins/providers/virtualbox/action/customize.rb:40:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/plugins/providers/virtualbox/action/check_accessible.rb:18:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builder.rb:116:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `block in run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/util/busy.rb:19:in `busy'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builtin/call.rb:53:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builder.rb:116:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `block in run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/util/busy.rb:19:in `busy'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builtin/call.rb:53:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/warden.rb:34:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/builder.rb:116:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `block in run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/util/busy.rb:19:in `busy'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/action/runner.rb:66:in `run'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/machine.rb:214:in `action_raw'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/machine.rb:191:in `block in action'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:516:in `lock'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/machine.rb:178:in `call'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/machine.rb:178:in `action'
    from /Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'
    from /Users/lvillani/.gem/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
    from /Users/lvillani/.gem/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

@lvillani
Copy link

Ah, got another one with boxcutter/ubuntu1404:

Bringing machine 'worker-1' up with 'virtualbox' provider...
/Users/lvillani/.gem/gems/vagrant-1.7.4.dev/lib/vagrant/environment.rb:492:in `initialize': No such file or directory - /Users/lvillani/.vagrant.d/data/lock.boxcutter/ubuntu1404.lock (Errno::ENOENT)

I see the pattern here but I know nothing about Vagrant's internals to be of any help.

@mpoeter
Copy link
Contributor Author

mpoeter commented Jul 28, 2015

The problem is the slash in the machine name. I use the machine name as part of the filename for the lock file. With the machine name containing a slash this results in a path with an additional directory that does not exist and therefore creation of the lock file fails.
Thanks for reporting this, I will improve this so that it works with machine names with arbitrary characters.

@lvillani
Copy link

Dang, for some reason I don't get an email when this PR is updated. I'll test it again as soon as I come back home.

@StefanScherer
Copy link
Contributor

@mpoeter Thanks for this PR! This really blows me away. Now Windows VM's make much more fun as they are much heavier in disk size than Linux VM's.
Just tested it with VirtualBox 5.0.0 on OSX 10.10.4 spinning up a Win10 guest.

How does this PR work with vagrant box remove and vagrant box update?

@lvillani
Copy link

@mpoeter Looks like the issue with the lock file is gone 👍

@mitchellh mitchellh merged commit 2a2f0a4 into hashicorp:master Oct 6, 2015
@mitchellh
Copy link
Contributor

Merged :)

@mpoeter
Copy link
Contributor Author

mpoeter commented Oct 10, 2015

Awsome! :-)

@hashicorp hashicorp locked and limited conversation to collaborators Apr 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants