Add configure option to use different vault location

[NickZ]

Running on Ubuntu 18.04; my root file system does not have a lot of space, so I had previously symlinked the vault directory in /var/snap/multipass/common to a directory on another drive. However, the recent 0.9 beta seems to have added an AppArmor policy that prevents me from doing that.

We should give users the ability to configure a different location for the vault since the symlink is no longer an available workaround for this.

[Saviq]

Hi @NickZ, it sounds like what you really should do is have /var mounted from wherever you have the space. Even if for Multipass we'll allow this, whatever the location you choose, will have to be made accessible via one of the snap interfaces. And then some other snap will come around (or just a lot of them) that will fill your /var/snap storage.

Something to propose to Snapd itself maybe, would be to have it grow the capability for the user to be able to choose where the writable area of a given snap goes.

[NickZ]

but Multipass is in classic confinement? I thought that meant it didn't require interfaces to access the rest of the file system?

[Saviq]

@NickZ we're moving to strict confinement soon (#1074). That's why the AppArmor profile was introduced. And even if we were not, the "other snaps will fill up your /var" argument stands.

In the mean time, you could bind-mount rather than symlink, that should make AppArmor happy.

[Saviq]

@callegar replying here to keep under one issue:

Currently, on Linux, multipass images go in /var/snap/multipass/ (at least as long as multipass is installed as a snap). This is probably a less than ideal location. For instance, many systems have their root on smallish SSDs to make them faster and \var as a subdir of root. In this case filling up the disk with multipass becomes way to easy, and may bring severe consequences (e.g., breaking system updates or making the system configuration inconsistent because a part of a system setup change cannot make it to the root disk).

As an example, trying to build complex snap packages with snapcraft (which uses multipass) can easily lead to many tens of GB of disk being taken.

While I understand this is a problem, I'm not sure a setting in Multipass is the right thing to do, see my replies above.

For maximum safety, it would be ideal to set up ad hoc spaces for virtual machines to make sure that virtual machine creation cannot end up with filling root. Indeed, this would be automatically guaranteed by placing \var on its own partition, but this is not how most system get setup as of today. While the appearance of multipass and snapcraft may now make one reconsider the advantage of placing \var in its own partition, changing that for systems that had already been setup is not easy.

Not sure what you mean by "ad hoc spaces"? We could include a disk space check in some operations (certainly on launch, maybe others as well) and warn the user about it running out. We can't be hard on it, though, as the disk images are dynamic (they will only grow up to the --disk limit if the space is used), so being able to create "too much" of disk space for the instances is a feature, not a bug.

Setting up multipass to store its images in other places would thus be ideal. However, without a configuration option it is not easy to do so. Trying to move away \var\snap\multipass and use a symlink to make it still appear in \var\snap does not work. Apparently, the only thing that works is a bind mount of some dedicated space to \var\snap\multipass.

Please, let multipass have a configuration file (e.g., in \etc\multipass.conf) and let it contain an entry for the location of the images.

See above, I really do think snapd needs to at least relate to this as a bigger whole.

[callegar]

@Saviq,
read the discussion above... thanks, confused about a few points, though.

1. With respect to:

Even if for Multipass we'll allow this, whatever the location you choose, will have to be made accessible via one of the snap interfaces

and

I really do think snapd needs to at least relate to this as a bigger whole.

I think that conditioning the way in which multipass works based on the (current) limitations of snapd is a weird choice and surely a flip of paradigm wrt how software has so far been developed. Originally, software devel was orthogonal of the packaging. Software was developed based on expectations on what it should do and what capabilities the OSs on which it was meant to be ported could offer. Packaging was following up, working hard to assure that the packaged software could work in the same way as software compiled and installed from source. We are now moving to a situation where software development is self imposing artificial limitations based on the way it is expected to be packaged, which is ... weird.

I think that if this happens, it will considerably slow the development of packaging solutions such as snapd too: if the software is programmed to stay in the limits set by some specific packaging anyway (and do so in a conservative way), why should packaging solutions improve to allow a better management of their limits?

Furhermore, I believe that multipass is also developed to run on windows and macos where different packaging formats exist.

I really think that multipass should expect to be able to read some configuration file and to try to do things according to it. Then if the packaging related confinement does not allow to do it, it should provide feedback, so that one can either change the configuration or change the confinement.

2. With respect to:

And then some other snap will come around (or just a lot of them) that will fill your /var/snap storage

I do not really care about other snaps that could fill /var. That is not, strictly speaking, a multipass problem. IMHO, multipass should focus on multipass. Otherwise, I could also be worried that other apps use so much RAM that multipass cannot run because of OOM, and based on this make totally weird requirements about the amount of RAM that a system must have to run multipass.

IMHO, the multipass problem is that it is inflexible about where it places its virtual machines disk images, which is a problem that has also been risen by users on windows or macos. This is because virtual images are not the usual piece of code or data. They are potentially huge things that may exist in a potentially large number. This is why virtualization solutions typically let you decide where the images are stored (and often on an individual basis).

Another reason for this flexibility is to make it easier to preserve the virtual images in case the system is updated, reinstalled, modified in its setup, etc. Just to mention an issue: currently, if one by mistake purges the snapd package or the multipass snap from the system, then all the images are gone.

3. With respect to:

what you really should do is have /var mounted from wherever you have the space

New systems may get installed based on different assumptions with regards to /var or /var/snap than in the past, but I see no way in which admins of systems that are already installed and working fine go through full reinstall or a migration of /var. The /var wisdom from a few years ago is that /var can often go (ore even should go) on the root partition and that when it is on its own partition the latter does not usually need to be that large. For instance, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/s2-diskpartrecommend-x86 and https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_a_standard_rhel_installation/graphical-installation_graphical-installation#manual-partitioning_graphical-installation

4. With respect to:

Not sure what you mean by "ad hoc spaces"?

I mean having a dedicated space or partition different from /root, /var or /var/snap for virtual images.

I do not want to break my system because virtual images fill / (if /var is in the root partition) while the system is trying to do something critical. I do not want my system to stop logging (e.g., network attacks) because virtual images fill /var. I do not want to stop being able to installing snaps because virtual images fill /var/snap (in case this is a dedicated partition).

Pesonally, I'd like to be able to tell multipass to use e.g., /mnt/multipass and to ask the snapd developers if they can let me provide a way to authorize the multipass snap to access /mnt/multipass after the snap has been installed in a sort of strict confinement that is configurable post-install.

Currently, bind mounts may help, but I do not like the idea that bind mounts are needed in fstab just to to work around software limitations that could best be removed in the software. Furthermore, the fact that installing and uninstalling the multipass snap seems to require it not to find /var/snap/multipass (for the install) or being able to completely to destroy it (for the purge) does not help the management of bind mounts, that work best if one can prepare the mount point beforehand.

For the reasons above, I really thinks that the possibility for multipass to provide configuration options can be considered.

[Saviq]

Hey @callegar thanks for the extensive post :)

I agree with some of your points, and lean towards making this an option in Multipass simply because even if I think it's snapd / bindmounts that are the better solution (because the problem needs only be fixed once then, rather than in all software), it's utopic to think that such a solution will come across platforms when it hasn't come to this date. IMHO it's a shame that every software has to solve this problem for itself rather than rely on the system, which has much better means to do so.

[swissguest]

+1 urgent need for customized location of vm images here, including ability to move existing images to different location

Unfortunately, I already had a number of vm images prepared earlier last year (on multipass version 0.7), and /var/snap/multipass/common/data/multipassd/vault/instances was linked to a separate disk due to space requirements. Today I noticed that multipass was updated to 1.0.2, and I could neither start existing images nor create new ones - when I had to quickly respond to a client request. So this update broke existing functionality.

[Saviq]

Here's a related post on the snapcraft forums.

[joe7575]

+1 I had the same problem, trying to build a simple Python based application. This was my first test example. It cost me some time to figure out, why my Ubuntu system did not boot any more (out of disc space).

[juanrubio]

+100

@Saviq

My development machine has a 30GB root partition, and around 8-9GB of those are always free...except for when I try to build the Snap of my application (https://tizonia.org/)

I fall into this trap every single time. I don't build Snaps every day or use Multipass for a living, so when I need to refresh the Snap of my application every 3-6 months, Snapcraft/Multipass get my root partition filled without any warnings, and when I realize, then it's too late, and it's a pain. It really is infuriating.

Even if there is no way to configure the location of the VMs for now, shouldn't be somewhere some logic to prevent the filling of the root partition, e.g. by warning the user that the root partition might get filled before the VM is created?

[tkedwards]

I also fell into this trap, built a snap using multipass and then the next time I rebooted my laptop it didn't come up. When I figured out it was because root was full I booted into rescue mode, and sure enough just one multipass VM took 6.4GiB.

Using bind mounts isn't a good solution. They're brittle and breakable (I've had lots of trouble with them with the LXD snap trying to do the exact same thing), and there's no good way to set them up - you have to manually edit /etc/fstab.

[Saviq]

Hi all, I've prepared a temporary workaround for this, by setting the (system-wide) MULTIPASS_STORAGE environment variable you can tell Multipass where to put all of its storage.

I've described in #1789 (comment) how to set it up for each of the platforms, and #1789 (comment) has the prerelease packages.

Unfortunately there's no easy way to move existing instances (which is why this is a workaround rather than a solution), so if you have data in the instances that you need, make sure to get it out first. Please let us know if this works for you.

[panlinux]

Hi all, I've prepared a temporary workaround for this, by setting the (system-wide) MULTIPASS_STORAGE environment variable you can tell Multipass where to put all of its storage.

Just to be clear, you can only set a path within /home or one of the removable media mount points, right?

[ricab]

Just to be clear, you can only set a path within /home or one of the removable media mount points, right?

Hi @panlinux. Yes, that's indeed the case on Linux with snap confinement (assuming home and removable-media plugs are connected).

[daniel-lenz]

Please, let multipass have a configuration file (e.g., in \etc\multipass.conf) and let it contain an entry for the location of the images.

[erlangparasu]

@Saviq so where is MULTIPASS_STORAGE default value? so i can move my existing data

[ricab]

Hi @erlangparasu, there is no default value for MULTIPASS_STORAGE. When that variable is not defined, Multipass uses different more or less standard locations for data and cache, depending on the platform.

We don't support moving existing data to the new location – that is one of the reasons why MULTIPASS_STORAGE is only a temporary measure – but if you tell me what platform you are on, I can tell you what the data path is and you can try copying it under $MULTIPASS_STORAGE/data. Make sure the daemon is not running when you do that. Cache I imagine you can tolerate leaving behind.

There is no guarantee this will work. Your best bet is still to extract your data from your VMs and start fresh in the new location.