New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lxc-create fails mysteriously if /var/lxc is BTRFS and ~/.vagrant/boxes is not #81

Closed
sport-clan opened this Issue May 30, 2013 · 7 comments

Comments

Projects
None yet
3 participants
@sport-clan

sport-clan commented May 30, 2013

Hi,

I'm following the README and failing at the vagrant up stage. I'm running Ubuntu 13.04. Here's some sample output:

james@simba:~/projects/hq/hq-mongodb-installer$ vagrant up --provider=lxc
Bringing machine 'default' up with 'lxc' provider...
[default] Importing base box 'quantal64'...
There was an error executing ["sudo", "lxc-create", "--template", "vagrant-tmp-hq-mongodb-installer-1369893590", "--name", "hq-mongodb-installer-1369893590", "--", "--arch", "amd64", "--release", "quantal", "--tarball", "/home/james/.vagrant.d/boxes/quantal64/lxc/rootfs.tar.gz", "--auth-key", "/opt/vagrant/embedded/gems/gems/vagrant-1.2.2/keys/vagrant.pub"]

For more information on the failure, enable detailed logging by setting
the environment variable VAGRANT_LOG to DEBUG.

If I set the debug mode I can see the following output from lxc-create:

DEBUG subprocess: stdout: 
DEBUG subprocess: stdout: lxc-create: No config file specified, using the default config /etc/lxc/default.conf
DEBUG subprocess: stdout: Extracting /home/james/.vagrant.d/boxes/quantal64/lxc/rootfs.tar.gz ...
DEBUG subprocess: stderr: rm: cannot remove '/var/lib/lxc/hq-mongodb-installer-1369893761/rootfs': Operation not permitted
DEBUG subprocess: stderr: lxc-create: failed to execute template 'vagrant-tmp-hq-mongodb-installer-1369893761'
DEBUG subprocess: stdout: Delete subvolume '/two/lxc/hq-mongodb-installer-1369893761/rootfs'
DEBUG subprocess: stderr: lxc-create: aborted
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 1

This turns out to be a problem with the way my filesystem is set up... I hade made /var/lxc a symlink to /two/lxc, where /two is a BTRFS filesystem. I guess lxc-create is trying to use some kind of BTRFS magic to clone the filesystem from my vagrant boxes dir ~/.vagrant/boxes to the /var/lxc location.

I managed to fix this by creating a new subvolume and making ~/.vagrant/boxes to that. Follow the instructions again and the box comes up fine. I wonder if you should make this part of the documentation, or even better detect it and warn the user beforehand.

I imagine that using BTRFS to drive /var/lxc will be a common requirement, and if you already have a root partition with another filesystem then this is an easy way to achieve it.

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm May 31, 2013

Owner

@sport-clan tks for reporting this, I haven't used BTRFS before but after some debugging looks like the issue is this line on creation scripts. could you please try commenting out that rm -rf from your ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and recreating the box to see if it works?
TBH I'm not really sure why I added that, I'm gonna have to look at the git history to try to find out :-P

Owner

fgrehm commented May 31, 2013

@sport-clan tks for reporting this, I haven't used BTRFS before but after some debugging looks like the issue is this line on creation scripts. could you please try commenting out that rm -rf from your ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and recreating the box to see if it works?
TBH I'm not really sure why I added that, I'm gonna have to look at the git history to try to find out :-P

@jamespharaoh

This comment has been minimized.

Show comment
Hide comment
@jamespharaoh

jamespharaoh May 31, 2013

Sorry, I reported under the wrong user id, this is my personal account.

As for the bug, I can't reproduce it now :-(

I've set my boxes dir back to a plain old directory (on my ext3 root partition as opposed to a symlink to the same btrfs partition where /var/lib/lxc points). But it seems to work fine now!

I wonder if I've installed an update that has fixed the underlying behaviour since yesterday...

jamespharaoh commented May 31, 2013

Sorry, I reported under the wrong user id, this is my personal account.

As for the bug, I can't reproduce it now :-(

I've set my boxes dir back to a plain old directory (on my ext3 root partition as opposed to a symlink to the same btrfs partition where /var/lib/lxc points). But it seems to work fine now!

I wonder if I've installed an update that has fixed the underlying behaviour since yesterday...

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm May 31, 2013

Owner

@jamespharaoh I was able to reproduce it and looks like removing the rm -rf did the trick. Since I've never used BTRFS before, I'll put up a Vagrant + VBox example setting things up when I have a chance for you to try out and check if I'm missing something else

Owner

fgrehm commented May 31, 2013

@jamespharaoh I was able to reproduce it and looks like removing the rm -rf did the trick. Since I've never used BTRFS before, I'll put up a Vagrant + VBox example setting things up when I have a chance for you to try out and check if I'm missing something else

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm Jun 2, 2013

Owner

@jamespharaoh would you be able to help me understand if I got it right? I've used an Ubuntu Raring VBox machine that you can find on https://gist.github.com/fgrehm/b07c6370a710be622807 and I followed instructions from http://www.funtoo.org/BTRFS_Fun to set things up and reproduce the problem.

Once you vagrant up that VBox machine, you can ssh into it and run:

dd if=/dev/zero of=/tmp/btrfs-vol0.img bs=1G count=1
sudo losetup /dev/loop0 /tmp/btrfs-vol0.img
sudo mkfs.btrfs -m single /dev/loop0
sudo mkdir -p /var/lib/lxc
sudo mount /dev/loop0 /var/lib/lxc

That will make /var/lib/lxc a BTRFS filesystem, once that is set, you can run:

mkdir ~/btrfs-test
cd ~/btrfs-test
vagrant init quantal64 http://dl.dropbox.com/u/13510779/lxc-quantal-amd64-2013-05-08.box
VAGRANT_LOG=debug vagrant up

And you'll get the same error again. If you remove the line 169 from ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and try bringing up that machine again it will just work.

Does that represent the setup you had when you've hit this?

Owner

fgrehm commented Jun 2, 2013

@jamespharaoh would you be able to help me understand if I got it right? I've used an Ubuntu Raring VBox machine that you can find on https://gist.github.com/fgrehm/b07c6370a710be622807 and I followed instructions from http://www.funtoo.org/BTRFS_Fun to set things up and reproduce the problem.

Once you vagrant up that VBox machine, you can ssh into it and run:

dd if=/dev/zero of=/tmp/btrfs-vol0.img bs=1G count=1
sudo losetup /dev/loop0 /tmp/btrfs-vol0.img
sudo mkfs.btrfs -m single /dev/loop0
sudo mkdir -p /var/lib/lxc
sudo mount /dev/loop0 /var/lib/lxc

That will make /var/lib/lxc a BTRFS filesystem, once that is set, you can run:

mkdir ~/btrfs-test
cd ~/btrfs-test
vagrant init quantal64 http://dl.dropbox.com/u/13510779/lxc-quantal-amd64-2013-05-08.box
VAGRANT_LOG=debug vagrant up

And you'll get the same error again. If you remove the line 169 from ~/.vagrant.d/boxes/quantal64/lxc/lxc-template and try bringing up that machine again it will just work.

Does that represent the setup you had when you've hit this?

@jamespharaoh

This comment has been minimized.

Show comment
Hide comment
@jamespharaoh

jamespharaoh Jun 3, 2013

@fgrehm Yes i think that is basically the same as what I had when I saw the error.

jamespharaoh commented Jun 3, 2013

@fgrehm Yes i think that is basically the same as what I had when I saw the error.

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm Jun 3, 2013

Owner

@jamespharaoh cool, I'll release new boxes with the updated lxc template as soon as I have the time. tks for reporting :-)

Owner

fgrehm commented Jun 3, 2013

@jamespharaoh cool, I'll release new boxes with the updated lxc template as soon as I have the time. tks for reporting :-)

@fgrehm fgrehm closed this in 5eb15d8 Jun 6, 2013

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm Jul 19, 2013

Owner

@jamespharaoh this has been released ;)

Owner

fgrehm commented Jul 19, 2013

@jamespharaoh this has been released ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment