Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lxc-clone on top of LVM thin provisioning + XFS doesn't quite work #807

Closed
lxnt opened this issue Feb 7, 2016 · 6 comments
Closed

lxc-clone on top of LVM thin provisioning + XFS doesn't quite work #807

lxnt opened this issue Feb 7, 2016 · 6 comments

Comments

@lxnt
Copy link

lxnt commented Feb 7, 2016

debian8(jessie) lxc 1.0.6-6+deb8u2

box:~#lxc-clone -s d8template cloname  fails:
  Logical volume "cloname" created
lxc_container: Error destroying rootfs for cloname
clone failed

In fact the cloned container is fine. However, to start it I need to:

  • lvchange -kn /dev/data/cloname -- because thin provisioned LVs are skip-autostart by default
  • lvchange -ay /dev/data/cloname -- because the LV doesn't get activated due to skip-autostart

and then

  • xfs_admin -U generate /dev/data/d8template

this changes filesystem UUID (doesn't matter which one - template or clone) and is needed because othewise xfs.ko refuses to mount it (duplicate UUIDs) thus lxc-start fails.
@vbaskoff
Copy link

Hi,
I see this in lxc 2.0.0rc13 + OracleLinux7.2; LVM thin provision without XFS

STR:

  • Create base container like:
    sudo lxc-create --name='template' --bdev='lvm' --vgname='lxdata' --thinpool='lxpool' --fssize='7G' --template='./lxc-customtemplate' -- --target='55.0' --repo-domain='packages..com'
  • Try to clone it like:
    sudo lxc-copy --name='template' --newname='config+log' --snapshot -B lvm --fssize='5G' --logpriority='DEBUG' --logfile='./copy.log'

Actual results:

$ cat ./copy.log
lxc-copy_ 20160331141405.882 DEBUG bdev - bdev/bdev.c:find_fstype_cb:885 - trying to mount '/dev/lxdata/template'->'/var/lib/lxc/template/rootfs' with fstype 'xfs'
lxc-copy 20160331141405.890 DEBUG bdev - bdev/bdev.c:find_fstype_cb:893 - mount failed with error: Invalid argument
lxc-copy 20160331141405.890 DEBUG bdev - bdev/bdev.c:find_fstype_cb:885 - trying to mount '/dev/lxdata/template'->'/var/lib/lxc/template/rootfs' with fstype 'ext4'
lxc-copy 20160331141405.912 INFO bdev - bdev/bdev.c:find_fstype_cb:901 - mounted '/dev/lxdata/template' on '/var/lib/lxc/template/rootfs', with fstype 'ext4'
lxc-copy 20160331141405.924 INFO bdev - bdev/bdev.c:detect_fs:659 - detected fstype ext4 for /dev/lxdata/template
lxc-copy 20160331141406.067 INFO lxc_container - lxccontainer.c:copy_file:2637 - Error stat'ing /var/lib/lxc/template/lxc_rdepends
lxc-copy 20160331141406.067 INFO lxc_container - lxccontainer.c:copy_rdepends:2796 - Error copying reverse dependencies
lxc-copy 20160331141406.094 ERROR lxc_container - lxccontainer.c:container_destroy:2391 - Error destroying rootfs for config+log

$ sudo lvs lxdata
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
config+log lxdata Vwi---tz-k 7.00g lxpool template
lxpool lxdata twi-aotz-- 66.36g 9.34 4.47
template lxdata Vwi-a-tz-- 7.00g lxpool 88.58

sudo lvchange -kn -ay lxdata/config+log allows me to boot the clonned container and seems it is ok.

@vbaskoff
Copy link

vbaskoff commented Apr 7, 2016

@stgraber

Sorry for bother you, but could some one check this issue? It is quite inconvenient to suppress 'lxc-clone' exit code and add direct lvm manipulations in build scripts.

@vbaskoff
Copy link

Situation getting worth in lxc 2.0.0, now 'lxc-copy' fails to determine fs on cloned LV and removes it, so 'lvchange' can't help anymore.

STR:
0. sudo lxc-create --name=template --bdev=lvm --vgname=lxdata --thinpool=lxpool --fssize=15G --template=./template/lxc-lxsys.sh -- --target-mr=55.0 --storage-url=ftp://md.intra/

  1. sudo lxc-copy --name='template' --newname='config+log' --snapshot -B lvm --logpriority='DEBUG' --logfile='./copy.log'

cat ./copy.log
lxc-copy 20160413141629.413 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/template'->'/var/lib/lxc/template/rootfs' with fstype 'xfs'
lxc-copy 20160413141629.423 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: Invalid argument
lxc-copy 20160413141629.423 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/template'->'/var/lib/lxc/template/rootfs' with fstype 'ext4'
lxc-copy 20160413141629.432 INFO bdev - bdev/bdev.c:find_fstype_cb:918 - mounted '/dev/lxdata/template' on '/var/lib/lxc/template/rootfs', with fstype 'ext4'
lxc-copy 20160413141629.446 INFO bdev - bdev/bdev.c:detect_fs:659 - detected fstype ext4 for /dev/lxdata/template
lxc-copy 20160413141629.603 INFO lxc_container - lxccontainer.c:copy_file:2622 - Error stat'ing /var/lib/lxc/template/lxc_rdepends
lxc-copy 20160413141629.603 INFO lxc_container - lxccontainer.c:copy_rdepends:2781 - Error copying reverse dependencies
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'xfs'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext4'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext3'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext2'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'iso9660'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: Permission denied
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'vfat'
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.617 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'hfs'
lxc-copy 20160413141629.620 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such device
lxc-copy 20160413141629.620 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'hfsplus'
lxc-copy 20160413141629.623 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such device
lxc-copy 20160413141629.623 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype '*'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such device
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'xfs'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext3'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext2'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'ext4'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'iso9660'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: Permission denied
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:902 - trying to mount '/dev/lxdata/config+log'->'/usr/lib64/lxc/rootfs' with fstype 'vfat'
lxc-copy 20160413141629.626 DEBUG bdev - bdev/bdev.c:find_fstype_cb:910 - mount failed with error: No such file or directory
lxc-copy 20160413141629.626 ERROR bdev - bdev/bdev.c:mount_unknown_fs:780 - failed to determine fs type for '/dev/lxdata/config+log'
lxc-copy 20160413141629.732 INFO lxc_container - lxccontainer.c:container_destroy:2379 - Destroyed rootfs for config+log
lxc-copy 20160413141629.732 INFO lxc_container - lxccontainer.c:container_destroy:2395 - Destroyed directory for config+log

@vbaskoff vbaskoff mentioned this issue Apr 13, 2016
Open
@lxnt
Copy link
Author

lxnt commented Apr 13, 2016

Obviously, bdev/bdev.c::do_lvm_create() should be patched to do the -ay -kn dance right after successful creation in the thinpool case.

https://github.com/lxc/lxc/blame/master/src/lxc/bdev/lxclvm.c#L111

Should be trivial.

This will not help with the duplicate UUID problem though. It may not prevent operation in the ext4 case, but most certainly will in the xfs one. It is also quite untidy to have multiple differing filesystems with the same UUID.

So a call to xfs_admin -U generate / tune2fs -U random / whatever it is for other filesystems should be inserted somewhere between snapshot/clone and attempts to mount. I have no idea where this point would be in the codebase. Will also mean dependencies on e2fsprogs/xfsprogs/whatever.

Ah, and yes, one can get rid of UUIDs completely ( xfs_admin -U nil / tune2fs -U clear). This should be mentioned in the documentation though.

Some advocacy in the mailing lists might help attacting someone wiser to this issue.

@lxnt
Copy link
Author

lxnt commented Apr 14, 2016

A workaround for the activation issue: #968 (comment)

set 'auto_set_activation_skip = 0' in '/etc/lvm/lvm.conf'

(thanks to @vbaskoff )

So the complete workaround is:

  • set auto_set_activation_skip = 0
  • clear template/source lv's uuid

I still think this should be fixed in lxc code.

@brauner
Copy link
Member

brauner commented Feb 8, 2018

The storage drivers have been reworked substantially if this is still reproducible on master, please re-open.

@brauner brauner closed this as completed Feb 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lxnt @vbaskoff @brauner