cloud-init cannot create swap files on paths that it mounts

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1869114

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = None
date_created = 2020-03-26T01:03:00.287171+00:00
date_fix_committed = None
date_fix_released = None
id = 1869114
importance = wishlist
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1869114
milestone = None
owner = johnsonshi
owner_name = Johnson Shi
private = False
status = triaged
submitter = johnsonshi
submitter_name = Johnson Shi
tags = []
duplicates = []

Launchpad user Johnson Shi(johnsonshi) wrote on 2020-03-26T01:03:00.287171+00:00

1. cloud-init's swap module runs before the mount module, which causes swapfile creation on runtime mounted disks to fail.
2. cloud-init's swap create does not ensure it has a path to the file (if specified with a directory that does not exist yet).

Bug exists in all clouds, but the bug was discovered through deploying an Azure VM, mounting the Azure resource disk, and then creating a swapfile within the mounted resource disk.

Relevant logs: https://paste.ubuntu.com/p/QdGybC2wyc/

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2020-03-31T14:53:37.371652+00:00

Hi Johnson, thanks for the bug report!

Swap file creation and mounting are both handled by the same module, cc_mounts. This has two phases: /etc/fstab generation and /etc/fstab application. In order for the use of swap files to be persisted between boots, they have to be included in /etc/fstab, so the swap file is created during the generation phase. This, of course, means that the other generated mounts haven't yet been applied, which leads to the behaviour you've observed here.

As other mounts (in general) presumably aren't dependent on the swap file's creation, it may be possible to change the order of these operations to behave as you would like without affecting other use cases, but we would probably need the expected new behaviour to be specified well to satisfy ourselves of this.

Thanks!

Dan

[ubuntu-server-builder]

Launchpad user Johnson Shi(johnsonshi) wrote on 2020-04-06T22:34:14.150895+00:00

Hi Dan, thanks for getting back to me.

I discussed this with Ryan Harper through IRC and he confirmed that this is a bug.

Relevant links:

cloud-init/cloudinit/config/cc_mounts.py

Lines 478 to 480 in e30599c

	swapret = handle_swapcfg(cfg.get('swap', {}))
	if swapret:
	actlist.append([swapret, "none", "swap", "sw", "0", "0"])

Swapfile setup (L478 where cloud-init attempts to create the swapfile) and writing of swap entry to /etc/fstab happens much earlier than "mount -a" (happens later below the code block).

cloud-init/cloudinit/config/cc_mounts.py

Lines 523 to 535 in e30599c

	if needswap:
	activate_cmds.append(["swapon", "-a"])
	if len(sops) == 0:
	log.debug("No changes to /etc/fstab made.")
	else:
	log.debug("Changes to fstab: %s", sops)
	need_mount_all = True
	if need_mount_all:
	activate_cmds.append(["mount", "-a"])
	if uses_systemd:
	activate_cmds.append(["systemctl", "daemon-reload"])

Also, "swapon -a" is issued before "mount -a".

Suggested approach:

1. Write mount entries to /etc/fstab first. Then issue "mount -a" cmd.
2. After step 1 happens, cloud-init would then run handle_swapcfg where it attempts to create the swapfile

   cloud-init/cloudinit/config/cc_mounts.py

   Line 478 in e30599c

   swapret = handle_swapcfg(cfg.get('swap', {}))

3. After that, cloud-init would then write the swap entry to /etc/fstab.
4. Cloud-init then issues "swapon -a" cmd.

This suggested change will enable cloud-init users to create swapfiles (not just swap partitions) on mounted disks on various clouds (not just Azure but also AWS and GCE).

Indeed, I think mounts in general won't presumably depend on swap files being created beforehand, while on the other hand, having swapfile creation happen after mounting would make more sense and unlock more user potentials, so our suggested changes won't disrupt users too much.

Thanks!
Johnson

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2020-04-07T21:12:38.569427+00:00

Hi Johnson, thanks for the update! I agree that this is something we can address. I see you've assigned yourself, are you interested in (or already) working on a fix?

[ubuntu-server-builder]

Launchpad user Marco Wester(mwester) wrote on 2021-02-02T20:28:05.415282+00:00

Hi everybody,

today I ran into the same problem.
Johnsons approach should work.
Is still somebody working on the fix?

Thanks!
Marco