Skip to content

spread: add LXD VM backend#185

Open
bboozzoo wants to merge 12 commits into
canonical:masterfrom
bboozzoo:bbboozzoo/lxd-vm-backend
Open

spread: add LXD VM backend#185
bboozzoo wants to merge 12 commits into
canonical:masterfrom
bboozzoo:bbboozzoo/lxd-vm-backend

Conversation

@bboozzoo

@bboozzoo bboozzoo commented Mar 27, 2024

Copy link
Copy Markdown
Contributor

Like LXD but with a twist, as instead of containers, it's using VMs. This is an improvement over the qemu backend, as LXD --vm generally runs with a virtio focused configuration ensuring that we're using hardware efficiently.

Another upside is that one does not need to build a special VM image anymore, LXD handles everything through cloud-init or the agent.

Comment thread spread/lxd.go
Comment on lines +481 to 491
for ifacename, ifaceconf := range sjson.State.Network {
if ifacename == "lo" {
// ignore loopback interface
continue
}
for _, addr := range ifaceconf.Addresses {
if addr.Family == "inet" && addr.Address != "" {
return addr.Address, nil
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one overlaps with #178, right? If this gets merged first, no need to merge #178.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, I think I mentioned somewhere that I had exact same change :) I will rebase as needed

Comment thread spread/lxd.go
Comment thread spread/lxd.go
Comment thread spread/lxd.go
Comment thread spread/runner.go
Add a new backend named lxd-vm, which is essentially VMs orchestrated through
LXD.

Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
LXD VMs, especially when running in GCP take quite long to have IP addresses
assigned to them. Use a longer timeout in this case.

Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
… and lxd-vm backends

Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
@bboozzoo
bboozzoo force-pushed the bbboozzoo/lxd-vm-backend branch from 11aef7b to 77fbb21 Compare March 28, 2024 13:46
Comment thread tests/lxd-vm/spread.yaml Outdated
backends:
lxd-vm:
systems:
- ubuntu-20.04

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably also test 16.04, 18.04 and 22.04 as per #182 (assuming there are VM images for all old releases as well)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

16.04 and 18.04 don't work in a VM, lxd-agent does not come start, so I've limited this to 20.04 and 22.04 for now. Another issue I ran into, is that tests need to be run one by one, otherwise the host runs out of disk space.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, and 22.04 will only work when #184 or other similar fix lands.

Prepare the test to run on multiple systems and mention why Ubuntu 16.04, 18.04,
and 22.04 are not currently working.

Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
@cmatsuoka

Copy link
Copy Markdown
Collaborator

I'm seeing this message on execution using the lxd-vm backend, it can also be adjusted to mention virtual machine instead of container:

2024-08-27 18:07:46 Waiting for lxd container spread-8-ubuntu-22-04 to have an address...

cmatsuoka and others added 6 commits December 10, 2024 13:07
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>

@fnordahl fnordahl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this,I discovered this PR after authoring #221

How can we work together to bring this to life?

Comment thread spread/lxd.go
}
}

func LXDVM(p *Project, b *Backend, o *Options) Provider {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In light of Spread already having support for declaring multiple named backends, choosing the provider with the type keyword, what benefit does adding a new provider type give?

Comment thread spread/lxd.go
args = append(args, "-c", fmt.Sprintf("limits.memory=%dMiB", mem))
}
if system.Storage != Size(0) {
args = append(args, "-d", fmt.Sprintf("root,size=%d", system.Storage))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Containers could also benefit from support for the storage keyword.

Comment thread spread/lxd.go
if p.vm {
// VM may take considerably longer to start
// TODO: should this be configurable?
maxTimeout = 180 * time.Second

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not see this in my testing but it is probably true, are you using emulated VMs?

After the VM has an address, subsequent calls to lxc exec will fail until the lxd-agent inside the VM has started, ab8232d could be useful for this.

Comment thread spread/lxd.go
}
if system.Storage != Size(0) {
args = append(args, "-d", fmt.Sprintf("root,size=%d", system.Storage))
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no way to specify the number of vCPUs for the VM. This is an important requirement as the default of 1 vCPU is most likely not sufficient.

In 804d1ee generic support for the LXD instance type concept is introduced driven by the already existing plan keyword.

LXD applies these constraints both to containers and VMs.

@yanksyoon

Copy link
Copy Markdown
Member

Hello! Out of curiosity, any updates on this/how we might progress on it?

@alanmcanonical

Copy link
Copy Markdown

+1. Any updates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants