Application Container in a Linux Container in a deeper hierarchy

[tazmo]

Hi NetBox Gurus-

I was playing around in the demo.netbox.dev instance and can't figure something out... how would you represent a (application) docker container running in a (userspace) lxc container in a LXD cluster on multiple Ubuntu VMs across separate nodes in a XCP-NG cluster?

I see the answer to 9655 but this seem to be a more complicated hierarchy than that... (please note this is not for me!)

Regardless, any ideas would be appreciated. Thx.
Bob

[candlerb]

Welcome to inception :-) I would suggest the following:

1. The Ubuntu VMs are Netbox "VirtualMachines" linked to a cluster which contains the physical device nodes.
2. The LXD containers are Netbox "VirtualMachines" linked to a cluster (representing the LXD cluster) with no devices.
3. The docker containers are Netbox "VirtualMachines" linked to a cluster (representing an LXD instance or a group of LXD instances) with no devices.

Certainly (1) and (2) work fine. The fact that you cannot build a cluster out of "virtual machines" means that you cannot directly associate your LXD cluster to its constituent VMs; you might be able to cobble something together with custom fields though, e.g. an Object field on VirtualMachine which points to Cluster.

(Aside: Netbox 3.3 has gained the ability to "pin" a virtual machine to the specific device in the cluster that it's running on. That's neat, but doesn't help the case where a virtualization cluster itself is built from virtual machines, not devices)

I've never attempted to model docker containers or a kubernetes cluster in Netbox to the level of the individual containers - I consider that to be part of the application, not data centre infrastructure. I don't see why it shouldn't be done - after all, in Kubernetes at least, each pod does have its own IP address and virtual interface. Again, in Netbox, these docker containers are associated with a "cluster" but that cluster is not directly associated with its parent lxd container instance(s), without extra customization.

However, it really depends what you want to use this data for. If you're going to use it for provisioning docker containers - perhaps using Netbox as a source-of-truth to push out Kubernetes resources or whatever - then you're going to need a lot more detail about how to provision a container. At that point, it may make more sense to write a plugin with a custom object type for a Docker container, which in turn can point to either a VM or a Device or a Cluster as its parent.

[tazmo]

Thank you @candlerb. I will try the custom field idea at that particular level of the hierarchy.
I appreciate the response....

[erik78se]

Is there a way to have netbox be "updated" dynamically with the virtual machines (VirtualMachines) that are fed from a "Cluster" which is a LXD host for example?

The task of keeping the netbox inventory updated continously is what I try to achieve by either pushing or pulling data into netbox.

Is there a mechanism for this? E.g. keeping a netbox inventory of many LXD hosts and the VirtualMachines that are added/removed continously. Is there a way otherwise to implement such a thing?

[candlerb]

Is there a way to have netbox be "updated" dynamically with the virtual machines (VirtualMachines) that are fed from a "Cluster" which is a LXD host for example?

If you're using Netbox that way, then you're Doing It Wrong™ according to the designers. Netbox is not meant to act as a portal onto data collected by other systems. It's meant to be a curated source of truth, which can be used to configure other systems.

If you want a live view of the current state of what VMs are running, then your VM cluster's own management interface is the best place to look. In the case of lxd, it has an API, and in more recent versions a web interface you can enable:

snap set lxd ui.enable=true && snap restart lxd

Suppose you want to coordinate allocation of IP addresses between VMs and other devices; that could be done by integrating into Netbox at the point of creating a VM, to pick the IP address to pass to the instance. In that case Netbox is the source of truth for available IP addresses, and is being used to push the IP address to the VM you're creating.

Of course, Netbox is open source software, and you're free to use it however you like. I'm just trying to explain why you won't find built-in support for what you're asking, or indeed any other sort of network auto-discovery.

[erik78se]

"Doing it the wrong way" makes absolutely no sense in this context.

I mean, why would Netbox even have then support for "VirtualMachines" in the first place then, which by all means are ephemeral by nature and where Netbox for all that matters wouldn't make up any useful "source of truth" to begin with?

However, I poked around in the API and wrote an application which is capable of populating the Netbox database via the API which achieves what I needed.

I'll post it here for reference after I've battled-tested it some more.

[candlerb]

I mean, why would Netbox even have then support for "VirtualMachines" in the first place then, which by all means are ephemeral by nature

Certainly you can use Netbox as the source of truth for VMs.

My workflow for LXD containers is as follows. I first create the container in Netbox - script here. A webhook triggers updates to the forward and reverse DNS.

I then run a script on the LXD host to create the container, passing its name. It calls out to Netbox's API to find what IP address to use, and inserts this in the cloud-init data. (I haven't implemented a webhook here, to avoid having to model all possible lxd settings that I might want to supply when provisioning)

This flow ensures IP address uniqueness. If you're allocating VM IP addresses via DHCP, then that's less of a concern - in effect, the DHCP server is the source of truth for how its pool is being utilized.