Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for striped LVM storage volumes #5211

Open
danielrobbins opened this issue Oct 27, 2018 · 14 comments
Assignees
Milestone

Comments

@danielrobbins
Copy link

@danielrobbins danielrobbins commented Oct 27, 2018

As a sysadmin, sometimes I would like to be able to create a striped lvm logical volume to use as a backing store for a container to take advantage of its performance characteristics. Currently, it appears that lxd does not provide a mechanism to do this. Special options would need to be passed to lvcreate by lxd to accomplish this. Without these options, the underlying vg will be used as a concatenated (linear) volume rather than a stripe, since lvcreate will not be "told" to set up the logical volume as a stripe.

Proposed solution: at container creation time, provide a mechansim for a sysadmin to specify parameters to be used for creating the thinpool to allow optimal use of the LVM volume group. Also allow a sysadmin to set defaults as desired. I am not sure where the best place for this is, but I could see the option varying per container. But it does need to be set at container creation time.

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Oct 27, 2018

A LVM specific option on the storage pool, similar to that we have for thinpool would likely be acceptable, though I don't expect us to have much of an interest in doing this work, so we'd need someone who cares about this to send a pull request.

@stgraber stgraber added the Maybe label Oct 27, 2018
@stgraber stgraber added this to the later milestone Oct 27, 2018
@stgraber stgraber added the Feature label Oct 27, 2018
@danielrobbins

This comment has been minimized.

Copy link
Author

@danielrobbins danielrobbins commented Oct 27, 2018

I could potentially send you a pull request -- can you give me an idea on the optimal way to specify this option -- either command-line -- or where it would be stored -- in which part of the config? And it's preferred name?

@danielrobbins

This comment has been minimized.

Copy link
Author

@danielrobbins danielrobbins commented Oct 28, 2018

Also wondering if maybe there should be some user-friendly option for creation like lxc launch --lvm_mode=striped, which would reference some specific some specific configuration defined somewhere else? Is there any way that these default settings could be placed inside profiles?

@danielrobbins

This comment has been minimized.

Copy link
Author

@danielrobbins danielrobbins commented Oct 28, 2018

I guess my dilemma is this... As a sysadmin, when I do create a container with "special options" for the underlying storage pool, I would like a way -- after the fact -- to be able to "see" that I did this, so I can know that I did it without doing a deep dive into the storage technology to re-discover it.

So I am wondering what the best way would be to store this information in lxd. Maybe a container-specific configuration variable that can only be set on creation time, but is immutable after the container is created? If you can give me some hints about this part and the best way to approach it from an lxd perspective, then I should be able to come up with a patch that is more suitable to be merged upstream, rather than a "hack" :)

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Oct 29, 2018

I think we'd want a volume.lvm.striped boolean option at the storage pool level which would then affect any LV created after the value is changed. That value is then copied as lvm.striped on the storage volume as a read-only property.

For custom storage volumes, that'd let you do something like lxc storage create default blah lvm.striped=true to override it just for that one custom volume.

For containers, there is no easy way to do so right now, but you'd be able to temporarily flip volume.lvm.striped on your storage pool, switching it to true, create the container and switch back to false.

@stgraber stgraber changed the title lxd does not allow the creation of striped lvm volumes Allow for stripped LVM storage volumes Oct 31, 2018
@stgraber stgraber added API Documentation and removed Maybe labels Sep 23, 2019
@stgraber stgraber added the Easy label Oct 3, 2019
@mparfan

This comment has been minimized.

Copy link

@mparfan mparfan commented Oct 29, 2019

Hi, I'm a student taking a virtualization class at UT Austin, and I'm currently collaborating with two others. @stgraber Is it possible that we can get more information about this issue if it's still available?

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Oct 29, 2019

So my understanding of that LVM feature is pretty limited so we'll need @danielrobbins to confirm that this would work for him. My understanding is that for striping, you'll want to configure both the number of stripes and the stripe size.

For LXD, we'll want this applied on a per-volume basis with support for a default volume as would typically be used for creating a new container from an image.

So I think we'd be looking at adding two pool config keys:

  • volume.lvm.stripes (must be >= 1, defaults to 1, no striping options passed if it's <= 1)
  • volume.lvm.stripes.size (amount of bytes (with our usual suffix support), default is empty and causes lvcreate to use the default)

And the matching keys for the volumes:

  • lvm.stripes
  • lvm.stripes.size

If lvm.stripes is set to > 1, then the -i option will be passed to lvcreate, if lvm.stripes.size is set as well, then both -i and -I will be passed.

@rcash

This comment has been minimized.

Copy link

@rcash rcash commented Oct 30, 2019

Thank you for the additional info! I'm one of @mparfan 's collaborators, we're interested in this one as well, just pending instructor approval.

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Oct 30, 2019

Cool, let me know when you get the go ahead and I'll assign it to the two of you.

@rcash

This comment has been minimized.

Copy link

@rcash rcash commented Nov 1, 2019

Hi @stgraber , we're approved! Could you assign @Lay-ton @mparfan and I?

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Nov 1, 2019

Hi @rcash I've assigned it to you and @mparfan, I can't assign to @Lay-ton as Github won't let me assign to someone who hasn't commented in the issue.

@Lay-ton

This comment has been minimized.

Copy link

@Lay-ton Lay-ton commented Nov 1, 2019

Hello, could you assign me as well?

@stgraber stgraber changed the title Allow for stripped LVM storage volumes Allow for striped LVM storage volumes Nov 5, 2019
@rcash

This comment has been minimized.

Copy link

@rcash rcash commented Nov 19, 2019

Hi @stgraber, I just wanted to give a status update since we've been silent for a few days and need some help/clarification on setting the volume keys.

We've spent awhile reading through files relevant to the issue and have added the storage pool config keys (volume.lvm.stripes and volume.lvm.stripes.size) and currently set the default number of stripes (volume.lvm.stripes) in StoragePoolCreate(). Where should the corresponding volume keys (changeableStoragePoolVolumeProperties for these be set? When the storage pool is created, when the volume is created, or another place? Our current assumption is the user will do something like lxc storage create poolA lvm volume.lvm.stripes=2 volume.lvm.stripes.size=64kB and the volumes that follow suit from this command will be striped like that as a result.

Edit: We are currently setting the default for number of stripes (volume.lvm.stripes) in StoragePoolInit() as well but have removed that since we know it is incorrect, that is not reflected in the repo linked above.

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Nov 20, 2019

lvm.stripes and lvm.stripes.size should go in changeableStoragePoolVolumeProperties, StorageVolumeConfigKeys and VolumeValidateConfig.

One quick way to see pretty much everywhere changes are needed would be to look for zfs.use_refquota which is a similar key in the zfs backend.

stgraber@castiana:~/data/code/lxc/lxd (lxc/master)$ grep -r zfs\.use_refquota . | grep -v storage\.zfs
./lxd/patches.go:			if !shared.IsTrue(pool.Config["volume.zfs.use_refquota"]) {
./lxd/patches.go:				pool.Config["volume.zfs.use_refquota"] = ""
./lxd/patches.go:				if !shared.IsTrue(volume.Config["zfs.use_refquota"]) {
./lxd/patches.go:					volume.Config["zfs.use_refquota"] = ""
./lxd/storage_pools_config.go:		"volume.zfs.use_refquota",
./lxd/storage_pools_config.go:	"volume.zfs.use_refquota":     shared.IsBool,
./lxd/storage_volumes_config.go:		"zfs.use_refquota",
./lxd/storage/utils.go:	"zfs.use_refquota": func(value string) ([]string, error) {
./lxd/storage/utils.go:			if config["zfs.use_refquota"] != "" {
./lxd/storage/utils.go:				return fmt.Errorf("the key volume.zfs.use_refquota cannot be used with non zfs storage volumes")
./test/suites/storage.sh:      lxc storage create "lxdtest-$(basename "${LXD_DIR}")-valid-zfs-pool-config" zfs volume.zfs.use_refquota=true
./test/suites/storage.sh:      ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-btrfs-pool-config" btrfs volume.zfs.use_refquota=true || false
./test/suites/storage.sh:    ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-dir-pool-config" dir volume.zfs.use_refquota=true || false
./test/suites/storage.sh:      ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-lvm-pool-config" lvm volume.zfs_use_refquota=true || false
./test/suites/storage.sh:      lxc storage volume set "lxdtest-$(basename "${LXD_DIR}")-pool1" c1pool1 zfs.use_refquota true
./scripts/bash/lxd-client:      volume.zfs.use_refquota zfs.clone_copy zfs.pool_name"
./scripts/bash/lxd-client:      security.unmapped security.shifted zfs.remove_snapshots zfs.use_refquota"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.