Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
VIPs with keepalived with systemd-networkd and ipv6 #1170
I came across several issues when using virtual floating IPs (VIPs) on Linux with systemd. Most of the tutorials online describing VIP setups on Linux do not work properly when
I'm here for advice on what are the best practises to run systemd with Linux distributions like Ubuntu 18.04 LTS (and alike), especially on ipv6 setups.
Ubuntu's default network configuration for servers relies on netplan.io, which itself defaults to
I've tried using a dummy interface for the VIP instead of the primary interface. This works quite nicely on ipv4, details in this blog post.
But: This doesn't work for ipv6, as keepalived defaults to send the unsolicited neighbour adverts via the same dummy interface in vrrp_ndisc.c:
I believe that a setup like this is quite common, and I was wondering if there are stable best practises given the situation with
A possible solution might be (also I'm not sure if this would even work), to have a configuration option that makes it possible to send out the neighbour adverts via a different interface then the one specified with
I'm aware of #836, but I believe this issue is different. I don't want keepalived to failover when systemd-networkd is restarted. This seems to be a feature of systemd - but before I open an issue there, I'd like to get some feedback on whether I'm missing on a fundamental best practise here.
Thanks in advance!
I am not familiar with using systemd-networkd, but if it cannot be configured to allow other processes to manage IP addresses, then it is a problem with systemd-networkd. It is absolutely not right that systemd-networkd should interfere with other processes and the addresses they manage. I would suggest raising an issue against systemd, but with nearly 1000 open issues, it isn't likely to get anywhere any time soon; nevertheless I think an issue should be raised there. You could also raise an issue against systemd-networkd on Ubuntu's launchpad.
You state in your blog post:
There are a couple of points regarding the above. First Ubuntu ships a snap of keepalived, and the version in the stable channel is currently v2.0.10.
The second point is that if a VIP is removed, the vrrp instance owning the VIP will transition to backup state and the highest priority vrrp instance will become the master within 1 advert interval. That should be the vrrp instance whose VIP was removed and that has just transitioned to backup (since it was previously the master due to being the highest priority), and so it should not be another machine that takes over as master. Behaviour could be different if
In your blog you also refer to using a dummy interface, but that it's state is always down. It is possible to up a dummy interface (
You refer above to IPv6 not working with a dummy interface, since it fails to send the NA packets. Looking at the code for IPv4 (function send_arp() in vrrp_arp.c), it also appears to use the ifindex that the address is configured on; it just seems not to report an error. Are you able to confirm whether or not gratuitous ARPs are actually sent for IPv4? I can't see how the kernel could work out an interface to send them on since keepalived specifies the dummy interface.
Does configuring keepalived to use vmacs (macvlans) make a difference? If systemd-networkd only removes IP addresses from the interfaces that it has configured, that might work. Or is systemd-networkd so intrusive that it removes the macvlans?
Does setting CriticalConnection=true in the systemd-networkd configuration on the interfaces keepalived is using make a difference?
It would be possible to add a configuration option against a vip such as
I have just tested the latest version of keepalived using dummy interfaces, and it now detects that the dummy interfaces do not support multicast, and hence since the GARP/GNA messages cannot be sent, the vrrp instances go to fault state:
(VI_4 is an IPv4 instance, VI_6 is an IPv6 instance).
referenced this issue
Mar 20, 2019
Thank you for your fast and detailed answer!
I've created an issue on systemd systemd/systemd#12050
Thanks for the hint! I've just updated to
I've noticed that the snap doesn't support reloading the daemon?
With keepalived-1, I've used
Thanks for the explanation. I can confirm that this is the case - but it nevertheless causes a small downtime when restarting
I can confirm that
I have not looked into this deeper. The default renderer on Ubuntu doesn't implement macvlans yet, but it's on their feature list
This prevents deletion of DHCP addresses on renewals, but doesn't prevent pruning of the VIP.
I should've clarified that I'm using unicast. But you're right anyway: The dummy interface can't properly send out GARP/NA. This basically renders my workaround with a dummy interface unusable.
My workaround so far:
If this is within scope of this repository, adding a
Otherwise only the decision whether it would make sense to add the option for a
There's no feedback so far from systemd...
If this is out of scope and you've decided against the additional options, feel free to close.
There does appear to be a response to systemd/systemd#12051, suggesting "dropping in a .network file matching the dummy device".
Regarding systemd/systemd#12050 the issue that other packages can legitimately manage IP addresses appears to have been completely overlooked (?ignored) - I'm really not clear where the idea of leaving old configuration on an interface comes from.
It might be worth asking the specific question against issue 12050: how can a system with systemd-networkd can be configured so that keepalived can work without having its addresses removed.
It is not within the scope of this repository to modify the snap file, although I might be able to do that in relation to some other work I am doing.
It would be really helpful to know if using macvlans can work. If you add
I think I am not in favour of suggesting the use of dummy interfaces, since it requires certain specific interface configuration (is it RP_FILTER?), and if a system doesn't use the relevant setting, it won't work.
I'm happy to consider garp-dev/gna-dev configuration options, but I would like to know first if configuring keepalived to use maclans get around the problem of systemd-networkd removing keepalived's IP addresses.
Yes. This helps - but then, systemd takes over management of the interface and prunes the address upon reload. Therefore, it doesn't help as a workaround for the underlying problem.
I thought I did that with systemd/systemd#12050 (comment) - maybe I should've been more detailed?
I'd apprechiate that! If it belongs to an open source repository, I might also be able to contribute on this. As a workaround, I'm running
That's a good hint! I wasn't actually aware of
In my tests,
With the current vmac setup, I'm having issues with ARPs though. The base interface can't announce the VIP. (I set the
Even when manually sending
This might not be directly related to
Also, documentation for using
Other than specifying
The document you found is VERY old, and rather out of date now (we know we need to update it, but writing code is mode interesting :).
You don't need to set any
I have never found that I need to use
I presume that the interface you are specifying in the vrrp_instance is ens3 (from your blog post). Assuming that is the case, it would be helpful if you could post the content if all the 'files' in the following directories (keepalived will need to be running when you do this):
It would also be helpful to see your keepalived configuration, the keepalived log entries from the time it starts, and the output of
I've just started with two fresh instances, trying
As soon as I enable
NOTE: I've allowed all ipv6 VRRP traffic and ICMP on their firewalls, not set any
When changing one instance to
After changing the second instance to
Here's the configuration I'm using (unicast peer and src addresses are flipped on the other node):
Here's the files in
The logs are this:
Many thanks for the info. I'm sorry I wasn't clear about what I wanted from the proc files. Could you please run the following and post the output:
I've now realised that
Oh, that explains things! I must have overread this in the documentation.
Unfortunately, our Hoster doesn't allow for multicast at the moment, so we're stuck with unicast :(
To summarize: Am I right that the best solution would be to use
If systend-networkd doesn't interfere with IP addresses on macvlan interfaces that aren't created by it, then there might be one possibility of making this work.
If you manually create macvlan interfaces on your ens3 interfaces, and assign IP addresses to them, prior to keepalived running, then you should be able to configure keepalived to use the macvlan interfaces with the IP addresses you have assigned to the macvlan interfaces. In this case, you would not configure
Mmh, manually using a macvlan (in bridge mode) works, but the VIP is not available. I suppose most likely because my carrier only allows one MAC address on the interface (?).
When using ipvlan instead, it seems to work fine though, including the unsolicited neighbour adverts (ipv6). Is there a reason not to use ipvlan on production? Seems like the general recommendation is using macvlan.
The ipvlan configuration:
ip link add link ens3 keepalived0 type ipvlan mode l2 ip link set keepalived0 up
I can see no reason why using an ipvlan should not work. The reason I suggested macvlan is that I have used them, but I haven't used ipvlans and didn't think of doing so.
If you have this working now, can the issue be closed?
That would simplify the setup quite a bit for people that can't use VMACs, as a lot of carriers do not allow multiple MAC addresses per interface.
@chr4 I had thought about that as an option myself, so yes, I think it is a good idea. Providing a patch via a pull request would be very helpful; I would be quite happy to provide feedback on any patch if necessary.
A couple of differences to consider for ipvlans vs macvlans:
Thanks for the input. Some further research suggests, that IPVLAN only works properly when attaching interfaces to a specific network namespace.
In my tests it works quite well when using ipv6, but communication isn't possible with ipv4 (pings received by parent interface, but never replied to). When attaching the interface to a namespace and use keepalived's
Documentation seems to be really scarce? The kernel documentation only provides examples including namespaces.
I have manually created an ipvlan interface in each of two systems, not using network namespaces, and am successfully running keepalived over the ipvlan interfaces.
The commands I used to create the interfaces were:
I then specify
I've been using the exact same commands, even copy and pasted yours, but I can't get it to work. The interface is created successfully, also the VIP is assigned correctly. But incoming ICMPs (let alone other packets) are just not replied to. I've doublechecked the
I'm currently testing in an OpenStack environment. I thought the IPVLAN approach is transparent to the underlying network hardware, but there might be something I've missed? What's weird is, that I can see the incoming packets with
Just for information sake for upstream and other interested parties
The issue with networkd pruning floating virtual IPs for high-availability environments is being address in 
Note downstreams will need to backport this fix once it has landed.
I have tested this on kernels from 4.16 through to 5.0 with both IPv4 and IPv6 and it works successfully for me. @chr4 if you try using this and have problems making it work, please log them in this issue, and if necessary we will reopen the issue.
Note that adding an [DHCP] section and setting CriticalConnection=yes in that [DHCP] section (even if you do not use DHCP) should work as a workaround. ( I have not managed to find the time to test this myself thou ) so someone here might be able to confirm/deny if that works as a workaround for them.