Container getting different IP Address on LXD host reboot.

[ak2766]

Required information

 * Distribution:           Ubuntu
 * Distribution version:   18.04.3 LTS
 * Kernel version:         4.15.0-72-generic
 * LXC Client version:     3.18
 * LXC Server version:     3.18
 * LXD version:            3.18
 * Storage backend in use: dir

Issue description

Whenever the LXD host is rebooted, some containers intermittently come up with different IP addresses yet MAC addresses have not changed. Is this a known issue?

I would have thought that DHCP leases would be stored on the host file system and that this file would be referenced when a container requests for an IP on reboot.

Steps to reproduce

1. Step one
   Save the output of `lxc list | egrep RUNNING`
2. Step two
   Reboot the host system
3. Step three
   Compare the output of `lxc list | egrep RUNNING` with that
   saved previously

Note: You may notice that some containers will come back up with a different IP - 
      not always the case.  But I'd wager you have about a 50/50 chance of a 
      container coming up with a different IP on reboot.

Information to attach

N/A

[stgraber]

Most Linux systems issue a DHCP RELEASE on shutdown marking their lease as unused and available for reuse.

It's then up to the individual DHCP server to decide what to do. Most will try to give back the same address based on past leases but there is no guarantee.

dnsmasq is using some kind of hash on the MAC to find a suitable lease, given that container MACs are very similar (same first 24 bits) and the IP space is limited (8 bits), the hash mechanism will often collide and so cause containers to switch addresses when no active lease is present AND the address is unused.

The normal expiry time may also be at play (1h). If that's the case, bumping it to a higher value on the bridge will help.

If you don't want addresses to potentially change, it's best to set a static lease on the nic device (ipv4.address).

[ak2766]

Yes, I had resorted to using static IP's; had to learn netplan.

Does dnsmasq check to see if an IP is already in use before it hands it out to a client? If not, is the only solution to split the address space to cater for dhcp and static?

[tomponline]

@ak2766 you shouldn't need to use netplan to configure static assignments, as LXD supports specifying static IP assignments via dnsmasq's DHCP.

e.g.

lxc config device set c1 eth0 ipv4.address=192.168.1.x

This will instruct dnsmasq to always give out the static IP to the container's MAC address.

This way dnsmasq is aware of the static assignments and will not allocate them to dynamic containers.

If you do want to use netplan to statically assign instead, then LXD disable's the dnsmasq feature that performs a ping to check whether an address is active before allocating it as this slows down container boot. There is also a scenario one could consider where a manually configured container is stopped, and a dynamic container boots up and is assigned the IP of the manually configured container. So even with the ping option in dnsmasq, it is not guaranteed safe, and in that situation it would be best to split the dynamic address range (which can be configured by LXD's network options https://linuxcontainers.org/lxd/docs/master/networks#bridges).

[ak2766]

@tomponline Thank you so much for going into detail. I've used lxc config before when attaching mount points but it escaped me on this issue. I'd better get rid of the netplan static entry in the container to avoid IP duplication.

[ak2766]

For completeness sake:

Quoting one of @stgraber's posts and slightly modified:

lxc stop container
lxc network attach lxdbr0 container eth0 eth0
lxc config device set container eth0 ipv4.address 10.99.10.42
lxc start container

[erik78se]

This didn't work for me. My container still has the old IP:

lxc stop juju-c2111d-0
lxc network attach lxdbr0 juju-c2111d-0 eth0 eth0
lxc config device set juju-c2111d-0 eth0 ipv4.address 10.51.45.220
lxc start juju-c2111d-0
lxc list
+---------------+---------+---------------------+------+-----------+-----------+
|     NAME      |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+---------------+---------+---------------------+------+-----------+-----------+
| juju-c2111d-0 | RUNNING | 10.51.45.221 (eth0) |      | CONTAINER | 0         |
+---------------+---------+---------------------+------+-----------+-----------+

Worth mentioning, is that another container (stopped) was assigned the same IP which might be present in the dnsmasq? If that is the case, how would I deal with this situation as I would expect the static assignment to have some precedence over dynamic leases ... right?

[update]
When I set the address also on the conflicting container to a "free" IP, the container juju-c2111d-0 now gets a completely different ipv4 address than I've set:

lxc list
+---------------+---------+---------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------+---------+---------------------+------+-----------+-----------+
| juju-c2111d-0 | RUNNING | 10.51.45.224 (eth0) | | CONTAINER | 0 |
+---------------+---------+---------------------+------+-----------+-----------+

This makes absolutely no sense at all to me.

lxc version
Client version: 5.15
Server version: 5.15

[update]
So, if I set to some other IP address (possibly not previously assigned or present in some lease) it seems to work as expected. So, my question would then be how I would go ahead and

A) Find whatever container seems to be holding the ip-address I want and...
B) Swap the ip, such as the correct container juju-c2111d-0 will have it and the conflicting one get a new.

[update]
So, if I assign a new ip address on the conflicting container (which I accidentally found) and start it. Then, I can also start my juju-c2111d-0 container, which then gets the correct IP-address.

This situation is either right, or wrong, since it would require me to actually START the conflicting container, before I can resolve the situation. While I now know how to sort of get out of my current issue, it still leaves the three questions:

A) Find whatever container seems to be holding the ip-address I want and...
B) Swap the ip, such as the correct container juju-c2111d-0 will have it and the conflicting one get a new.
C) Should it really be necessary to start a conflicting container, to let a second be able to have the assigned address? It feels awkward and wrong.