Multiply OTBR do not allow scaling of devices number limit

[michal-markiewicz-ci]

We are trying to build an OpenThread mesh network using multiple OTBRs.

With a single OTBR, we can connect around 25 devices, and they are able to reach external resources (outside the mesh), such as IoT Hub.

After adding a second OTBR and another 25 devices (all devices are in one large room and can see each other), the new OTBR becomes a secondary router. The new devices do not connect to IoT Hub. In fact, some of the first 25 devices also lose connectivity to IoT Hub. It looks like there is no proper scaling mechanism and the mesh becomes overloaded. All traffic leaving the mesh goes through a single border router, creating a bottleneck.

We are using NAT64. The secondary OTBR sees the primary one over both radio and Ethernet. We tried modifying the radio selection algorithm to always prefer TREL.

As a result, the two OTBRs started communicating via TREL instead of radio. However, all devices still use the primary OTBR to send their data outside the mesh, so the bottleneck remains.

Is there any way to force some devices to route their traffic through the second OTBR?

Should both OTBRs use different prefixes? Is there a way to create sub-networks or partitions within a single network? Or maybe NAT64 is the problem because two OTBRs cannot have it enabled at the same time. Or maybe we should use separate partitions and both OTBRs should provide a synchronization mechanism for communication between both partitions?

[abtink]

If you are using NAT64 on the BR itself, then you would be limited by the fact that only one of the BRs would act as the NAT64 translator. The fact that all traffic will go through that BR is expected and normal behavior in such a case.

[jwhui]

NAT64 is stateful. As a result, all traffic for a given NAT64 prefix must flow through a single NAT64 translator. This is not specific to Thread.

Can you describe what is causing the bottleneck performance issue? You mentioned enabling TREL, so the 802.15.4 channel utilization should not be the bottleneck.

[michal-markiewicz-ci]

We suspect that in Case 2 the issue is that all traffic leaving the Thread mesh is routed through a single border router (OTBR1).

Case 1 (case1.png)
We have two floors with no radio connectivity between them, only Ethernet. Each floor has one OTBR and about 25 Thread devices.
The OTBRs connect to each other using TREL. OTBR1 becomes the Primary Border Router and enables NAT64 (as defined by the OpenThread TREL mechanism).
As a result:

1. OTBR1 routes external traffic for the 25 devices on its own floor over radio.

2. OTBR1 also receives traffic from OTBR2 for the 25 devices on the second floor.

3. OTBR2 disables NAT64 after establishing the TREL connection.

4. The outcome is that all 50 devices can reach external networks, even though only one OTBR performs NAT64.

Case 2 (case2.png)
We have a single large room. All 50 devices and both OTBRs have radio visibility to each other.
We force a TREL connection between the OTBRs by modifying the radio-selection mechanism.
However, in this configuration:

1. Almost all devices choose OTBR1 as the next hop for external traffic.

2. OTBR2 is rarely selected, even though it is available.

3. TREL does not help distribute or balance the traffic load.

4. As a result, OTBR1 becomes a bottleneck due to radio throughput limitations.

5. We suspect this behavior is defined by Thread routing rules, which prefer a single Primary Border Router for off-mesh traffic.

Our question
We would like to understand how to scale the number of devices by adding additional border routers.
Specifically, we want to know whether it is possible to distribute off-mesh traffic across multiple OTBRs using NAT64 or how to do that without this feature?

[jwhui]

In Case 1, is NAT64 the performance bottleneck? As previously mentioned, NAT64 is a stateful service and all traffic for a given NAT64 prefix must flow through a single NAT64 translator. That NAT64 translator does not have to be hosted by a Thread BR and can be hosted anywhere in your site, but all traffic for the given NAT64 prefix must flow through it.

In Case 2, if all devices are within radio range, then they must all share the same radio spectrum. Given that the Thread BRs are within one hop, it will not matter which Thread BR they are routing traffic through.

One way to improve scale is by segmenting your site across multiple Thread networks. For example, each Thread network could be configured with a different NAT64 service and Thread channel.

[michal-markiewicz-ci]

In Case 1, NAT64 is not the bottleneck. A larger number of OTBRs might eventually cause CPU or RAM pressure on the primary border router — for example, five OTBRs serving more than 100 devices could become problematic. But with only two OTBRs, everything works fine.
However, are you suggesting that in this scenario NAT64 could be handled by the LAN router instead of the OTBR?

Regarding “multiple Thread networks”:
Do you mean completely separate networks operating on different channels? If so, how would devices from these separate networks communicate — would this require border routers and communication mechanisms outside of Thread (on application layer)? For example, how would a device in one Thread network send a message to a device in another?
Or do you mean networks that share the same dataset except for the channel, using some kind of subnetwork mechanism?

[jwhui]

1. NAT64 on the LAN Router:

   • Yes, that is exactly what I was suggesting. If your infrastructure router (or another server on the LAN) provides NAT64/DNS64 services, the OTBRs can be configured to simply route traffic for the NAT64 prefix toward the LAN. This moves the stateful NAT64 translation and the associated CPU/RAM pressure off the OTBRs and onto a more capable device. Multiple OTBRs can then concurrently route traffic for the same NAT64 prefix to the backbone, though the NAT64 translator itself remains a single point of state for those flows.

2. Multiple Thread Networks and Communication:

   • Separate Networks: This means using different PAN IDs, Channels, and Network Keys. Each network operates independently on its own radio channel, which is the most effective way to scale the total number of devices in a single physical area.
   • Cross-Network Communication: Communication between devices on different Thread networks happens at the IPv6 layer. If the OTBRs for both networks are connected to the same Ethernet/Wi-Fi backbone, they act as standard IPv6 routers. A device in Network A can reach a device in Network B using its off-mesh-routable prefix
   • Discovery: To allow devices to "find" each other across networks, you would typically use DNS-SD. The OTBRs can proxy discovery requests between the Thread networks and the LAN.
   • By segmenting into multiple networks on different channels, you not only distribute the radio load but also ensure that a bottleneck on one OTBR or channel doesn't degrade the entire system.

[michal-markiewicz-ci]

We have made many tests. Our conclusion is that our border router / channel has support for 30-40kbps for outside network traffic for one otbr.

Additionally, we tested a setup with two OTBRs and increased the number of devices in the one network, but it did not improve performance. tcpdump logs suggest that most traffic still goes through the primary border router, even when IPv6 is used on the LAN (we do not use NAT64). In our experience, separate Thread meshes perform better.

I have some questions about our test:

a) How can I distinguish between a border router throughput bottleneck and actual RF channel congestion in an OpenThread network? Are MAC counters such as TxErrCca and TxErrBusyChannel reliable indicators of channel occupancy, and what patterns typically suggest RF congestion rather than BR saturation?

b) In a dual‑border‑router setup (primary + secondary OTBR), can the secondary OTBR be configured to offload traffic from the primary one to achieve at least partial scalability? IPv6 on the LAN does not seem to help, as the primary OTBR still attracts most traffic even without NAT64. Is there any recommended method for distributing traffic between multiple OTBRs?

c) Is it possible to attach specific multicast groups or CoAP channels to selected border routers so that they act as controlled “bridges” between chosen independent Thread networks? If so, what mechanisms or configurations support selective multicast/CoAP forwarding between networks? Right know we would like to create for example 10 networks with 100 devices each and keep communication between devices from different networks. Is it combination of srp and dns-sd mechanism?

[jwhui]

Here are some insights that might help clarify your observations:

a) Distinguishing BR Bottleneck vs. RF Congestion

Yes, the MAC counters you mentioned are reliable indicators of RF channel conditions:

• TxErrCca: Increments when the device fails to transmit because the Clear Channel Assessment (CCA) detected that the channel was busy.
• TxErrBusyChannel: Increments when a frame is dropped due to repeated channel access failures.

Patterns indicating RF Congestion:

• High rates of TxErrCca and TxErrBusyChannel relative to successful transmissions (TxSuccess).
• High MAC retry counts.
• This suggests the physical medium is saturated or experiencing significant interference.

Patterns indicating BR Saturation:

• If the MAC counters show low error rates but throughput is capped, the bottleneck is likely in the processing capacity of the Border Router or the network interface queues.

b) Traffic Distribution in Dual-Border-Router Setup

In a single Thread network with multiple OTBRs (Border Routers), traffic distribution is determined by Thread routing and Infrastructure link routing (RA/ND).

• Thread Routing: OTBRs advertise themselves as default routers and provide external routes via On-Mesh Prefixes (OMR). If both advertise the same prefix with the same priority, Thread devices generally do not perform dynamic load balancing per flow. A device will often prefer the path with the lower cost (e.g., fewer hops or better link quality). If both paths are equal, implementation-dependent behavior might result in sticking to one.
• Infrastructure Side: When traffic comes from the LAN to the Thread network, hosts on the LAN will use standard IPv6 routing. If two Thread Border Routers are of equal cost, it will pick whichever was first.
• Offloading: To distribute traffic, you could consider using different OMR prefixes for different BRs or configuring different priorities.

c) Connecting Independent Networks

• Service Discovery (SRP & DNS-SD): The combination of SRP (Service Registration Protocol) and DNS-SD (DNS Service Discovery) is the standard way to enable service discovery across independent networks.
  • Devices in Mesh A register their services with the local BR via SRP.
  • The BR makes these services available on the infrastructure network via mDNS or a DNS-SD Discovery Proxy.
  • Devices in Mesh B can then discover these services by querying the Discovery Proxy or via mDNS on the LAN.
  • Once they have the IPv6 address of the target device, standard IP routing across the infrastructure network allows them to communicate directly (Mesh A Device <-> Mesh A BR <-> LAN <-> Mesh B BR <-> Mesh B Device).

Creating multiple independent networks (e.g., 10 networks with 100 devices each) and relying on Border Routing and DNS-SD for cross-network communication is a recommended approach for large-scale deployments, as it limits the size of the collision domain on the RF channel and improves overall system scalability.