Supervisor isolated physical network access/macvlan support

[agners]

Problem statement

Add-ons (in Supervisor based installations, so Home Assistant OS or Supervised) run by default in isolated network environment in a common, internal network named hassio. Each add-on gets its own internal IP address and applications can reach the user's local network and the internet via NAT.

Optionally, add-on developers can request host network access via the host_network add-on configuration option.

The isolation provided by these two options are in rather stark contrast: On one hand the default networking isolates the add-on to a large degree from the user's local network, on the other hand using host_network option has no isolation at all and gives add-on full access to the hosts network interfaces, along with management interfaces like the hassio network itself, which is often not of interest for the add-ons and adds to confusion and issues at times.

Over time, various advanced networking needs have arisen by different type of add-ons:

• Many add-ons would like to make use of mDNS hostname resolution. This has largely been resolved: Add-on can use the regular DNS resolver through a locally run DNS (DNS plug-in) which resolves .local addresses as well.
• Many add-ons would like to resolve services through mDNS/DNS-SD. The Multicast plug-in's mdns-repeater service forwards the link-local mDNS multicasts on the host network to the hassio network, which allows mDNS clients on the hassio network to learn about services on the outside network.
• Custom multicast protocol. E.g. the RaspberryMatic add-on would like to listen to custom multicast protocol (see HmIPW-DRAP/HmIP-HAP cannot identify RaspberryMatic as a HA addon jens-maus/RaspberryMatic#1373). Previous attempts to use multicast routing via pimd have not worked out.
• IPv6 support in general. Currently the isolated network supports IPv4 only. Technically IPv6 could be supported in a similar fashion to how IPv4 is supported (using NAT), this is not possible for all IPv6 use cases (and also not how IPv6 is run, ideally).
• IPv6 router support. The OpenThread Border Router add-on creates a separate network interface using the Linux TUN driver and acts as Thread border router routing IPv6 packets. The OTBR uses the IPv6 Neighbor Discovery Protocol to announce itself on the local network and find other OTBRs. The OTBR is a rather specific use case, but it could potentially also benefit from such a feature.

Almost all use-cases can (and typically are) solved by simply using host network, however, with the downside of having no isolation and higher chance of add-ons interfering with each other.

Another networking need comes from multi-homed hosts: Often users would like to offer services only for a particular network (interface). E.g. the DHCP add-on allows to configure the interface it hands out leases to. The Samba add-on listens on all interfaces by default, but that might not be ideal for all use cases. The OpenThread Border Router add-on uses the primary interface as deemed by the NetworkManager as backbone/infrastructure link.

While most services allow to define the network interface they listen on, some listen on any interface by default. E.g. the Matter Server listens to mDNS/DNS-SD announcements on any interface without a built-in option to limit the interfaces used. Since this is mostly a passive listen-only, this is not that problematic. However, currently this behavior creates some log noise since the host network has also virtual interfaces from all containers, which do not have IPv6 addressing enabled:

2024-02-02 15:47:51 core-matter-server chip.DL[224] ERROR MDNS failed to join multicast group on veth7b5ad0c for address type IPv4: src/inet/UDPEndPointImplSockets.cpp:777: Inet Error 0x00000110: Address not found
2024-02-02 15:47:51 core-matter-server chip.DL[224] ERROR MDNS failed to join multicast group on veth912a60c for address type IPv4: src/inet/UDPEndPointImplSockets.cpp:777: Inet Error 0x00000110: Address not found

This architecture discussion aims to discuss these needs and find a better solution which fulfills many of the above mentioned requirements. The goal is to strike a good balance between capability, usability and usefulness of such a feature.

Possible solution

Add macvlan support to Supervisor

One possible solution could be to support macvlan support to Supervisor. Docker offers the macvlan network driver which allows to add a virtual network interface with a separate MAC address which is bridged to the physical network interface. Since this is a separate L2 virtual network interface, it needs its own IP configuration. By default static IP configuration is available from Docker.

Implementing macvlan support could be done in various ways in HA: In a simple version we could just add a "virtual network" configuration page in the add-on, and allow the user to manually configure a macvlan interface. A more advanced version of this would allow multiple macvlan interfaces per add-on.

Even more advanced would be a common place to create a named "virtual network" which would allow to add multiple macvlan interfaces along with the network configurations for each one of those. These then could be attached to add-ons.

Downsides:

• Additional IP address for the isolated add-ons
• Added complexity
• Promiscuous mode support required (is this available and safe to be used on all hardware 🤔 ?)

Upsides:

• Strong isolation possible
• Flexible approach
• Full multicast support
• Transparent for applications (can be applied generically)
• Can also provide generic interface selection

[agners]

Continuing discussions from home-assistant/addons#3416 (comment)

Creating macvlan networks could go on the System -> Network page ... But where would the associations to add-ons be configured? (Maybe in each add-on's Info or Configuration tab?) ... And how would we determine/configure which add-ons should use this vs which add-ons should use the host network (I imagine some add-ons will need to be able to access multiple physical networks, so macvlan wouldn't work for those) - Should the add-on specify whether host network or macvlan should be used, or should that be configurable by the user for all add-ons, or ...?

@PaulSD these suggestions pretty much match with what I had in mind. One thing we need to keep in mind is how do we deal with existing add-ons and how can we minimize the user impact for people which do not need/want this feature.

At first I thought this could be an add-on configuration: An add-on opt-in to macvlan support. If enabled, the user must configure a macvlan for it.

But thinking a bit more about it, I start to feel that this should be an alternative option for host_network. E.g. by default the add-on with host_network just runs in host network. But the user optionally can assign such an add-on a macvlan. So existing add-ons would just continue to work transparently. And only users who want a defined set of interface exposed to a add-on would need to go through the work of setting up a macvlan 🤔 Furthermore, this would also have the least chance of negative impact, e.g. in case promiscuous mode doesn't work on some hardware or other issues with the feature.

[agners]

Continuing discussion from home-assistant/addons#3416 (comment)

I'm not sure that using macvlan is the correct solution here, since it's not that a multi-homed HA needs multiple MACs on each interface, but rather that add-ons should not make assumptions about what interfaces to use where it's necessary or desirable to pick one. Definitely some add-ons might benefit from their own L2 MAC and associated IP, but I don't think that's the main use case, and it potentially requires assigning many more external IPs to HA than necessary. I can see the benefits to assigning add-ons distinct internal IPs to permit inter-add-on communication, but not so much for external use.

Far point, one could argue macvlan is not the right tool/is over-engineered for that problem at hand. I guess my point here is if we build feature, presumably mainly for other reasons than simply network interface selection, we could use it as a generalized way of providing interface selection to add-ons. 🤔 I guess distinct IPs for external use is more a necessary side effect of the isolation, than really something I am looking for here. But it might also come in handy when debugging, e.g. it would make it easy to monitor the traffic of a add-on with host network access by simply filtering by IP.

I think something inspired by swconfig or DSA to assign interfaces to add-ons makes more sense than just being able to give add-ons their own macvlan(s). (It would still be possible to create an external macvlan interface and assign it to a single add-on.) As an example use case, consider a dnsmasq add-on that serves DHCP clients on some interfaces, and uses a separate upstream interface for DNS. This could also solve the issue of Matter knowing which interface to used when passed a link-local IPv6 address; it could also be leveraged to reduce broadcast traffic to interfaces that do not need it (e.g.: mDNS requests for IoT devices being sent on a non-IoT VLAN).

I am not very familiar with swconfig or DSA, but isn't this for actual switch hardware? I don't think this can be used on arbitrary hardware.

[codyc1515]

I would not support this, unless it was touchless like today. Adding IPv6 for internal traffic is a great idea but not at the expense of the user experience.

[agners]

What does "this" refer to here?

But yeah I agree, the typical use-case must not require any additional user interference. Either it is not active in that case, or things happen automatically.

[Jc2k]

I think Macvlan is a very useful tool to have in the bag. I run my HA this way, and I much prefer it to host network mode.

One of the reasons we need to carry a patch in HAOS is because running something like OTBR and matter server together in the same network namespace is sort of undefined behaviour under RFC 4191. According to the kernel maintainers, routers are not supposed to have the neighbour validity checking the matter server needs to not have 30 minute outages all the time. If we could run OTBR under Macvlan, I think we might not hit that edge case in RFC 4191 and we wouldn't need that patch?

I also wonder whether the TREL route loop that needed an OTBR patch would have behaved differently under that kind of setup?

[jens-maus]

Seeing that since Feb there haven't been any more activity here. So what is the current state of affairs here on the macvlan integration in HA?

To make my own point a bit more clear, since I am one of the HA add-on developers which would highly benefit from being able to utilize macvlan functionality, I want to point out again that from a technical point of view a per-addon macvlan already works in HA. Just have a look at the support patch/script I have created for my users (cf. https://github.com/jens-maus/RaspberryMatic/wiki/Installation-HomeAssistant#hmip-haphmipw-drap-support-patch and https://github.com/jens-maus/RaspberryMatic/blob/master/scripts/patch-ha-addon-macvlan.sh). Executed on a HA host, this simple bash script generates the necessary docker networks (https://github.com/jens-maus/RaspberryMatic/blob/master/scripts/patch-ha-addon-macvlan.sh#L90-L94) and then simply assigns a static IP and connects the add-on docker container to this new macvlan based docker network.

So from my point of view, implementing add-on based macvlan support could/should be implemented in a rather simple and add-on centric way:

• add an additional config.yaml option so that add-on developers can specify that their add-on requires this macvlan network functionality.
• if specified allow users to enable macvlan (keep default to non macvlan) and if enabled allow to specify the static network settings (IP, subnet, etc.) in a new configuration tab in the add-on settings UI.
• if the add-on is started with enabled macvlan HA should then create the necessary docker network similar to what my current workaround support patch script is performing and connect the add-on docker container to this new network.

Therefore, I think that a general or more broad/global macvlan functionality is not really required in HA itself. IMHO add-on developers like me should know best if their add-on requires this special functionality. Thus, a simple config.yaml addon config option should be enough for add-on developers to allow to specify that this addon might require macvlan functionality.

So looking forward to seeing something similar being integrated in one of the next HA/supervisor versions so that my users (who require this functionality) can finally get rid of this workaround support patch which they have to re-execute on each add-on restart or HA restart. :)