Spanning Tree Protocol (STP) doesn't work on Realtek RTL838x

[joe2xyz]

Describe the bug

I wanted to connect more than 2 switches (everything running OpenWrt) in a loop to make it HA in case of a cable / switch failure but I noticed when I finish the loop, the switches stop working properly since Spanning Tree Protocol (STP) that I enabled in Bridge configuration on all switches doesn't work. I tested that both on the most current stable version and the current snapshot and with ZyXEL GS1900-24 with RTL838x switches and ZyXEL GS1900-48 with RTL839x switches.

Thank you all Developers and Maintainers for building and supporting OpenWrt. OpenWrt is the best! Thank you!

OpenWrt version

r22000-6bc675c0be

OpenWrt target/subtarget

realtek/rtl838x

Device

ZyXEL GS1900-24 v1

Image kind

Official downloaded image

Steps to reproduce

• Enable STP: network.switch.stp='1'
• Connect switches in a loop
• Later I tried to change the following values but that had no effect:

network.switch.priority=<different for different switches>
network.switch.hello_time='4'
network.switch.forward_delay='15'
network.switch.max_age='20'

Actual behaviour

• STP seems to be doing something since I can see the following messages in serial terminal:

[40254.045245] switch: port 3(lan3) entered blocking state
[40254.051121] switch: port 3(lan3) entered listening state
[40270.584219] switch: port 3(lan3) entered learning state
[40287.224253] switch: port 3(lan3) entered forwarding state

• But when I run: brctl showstp switch I get:

switch
 bridge id              <same for all ports>
 designated root        <same for all ports>
 root port                 0                    path cost                  0
 max age                  10.00                 bridge max age            10.00
 hello time                1.00                 bridge hello time          1.00
 forward delay             8.00                 bridge forward delay       8.00
 ageing time             300.00
 hello timer               0.16                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                  91.99
 flags


lan23 (23)
 port id                8017                    state                  disabled
 designated root        <same for all ports>    path cost                100
 designated bridge      <same for all ports>    message age timer          0.00
 designated port        8017                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.00
 flags

 ...

• That tells me that all switches are Root bridge which means that the switches don't know about each other.

• I can verify that STP doesn't work also because when I finish the loop connection, the switching stop working completely or partially and I get the following messages in serial terminal:

[   83.135326] switch: received packet on lan3 with own address as source address (addr:<switch MAC>, vlan:0)
[   83.147165] switch: received packet on lan9 with own address as source address (addr:<switch MAC>, vlan:0)

• That means that broadcast storm is happening.

Expected behaviour

brctl showstp switch should should different IDs for all switches except one.

Additional info

I made sure I'm testing the most simple configuration as possible so I disabled VLANs, firewall accepting everything, etc.. This is my configuration that is same on all switches:

configuration.txt

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[brada4]

Different hardware must have distinct bridge id
It is derived from MAC on first port up (or at least was long ago)

[joe2xyz]

Thank you for commenting @brada4 .

Absolutely, each switch has different Bridge ID. The problem is that brctl showstp switch should not show on each switch that the bridge is also root. In other words each switch has bridge that is also Root bridge and with functional STP that would happen only when each switch was completely disconnected from other switches.

[joe2xyz]

This is how it should like:

But this is the reality:

[brada4]

Does it disconnect one port when you loop a single switch two ports?

[joe2xyz]

I've just tested that and the both links stay up:

• A cable connected to port 15 and 16:

[  474.920158] rtl83xx-switch switch@1b000000 lan16: Link is Up - 1Gbps/Full - flow control rx/tx
[  474.930217] switch: port 16(lan16) entered blocking state
[  474.936440] switch: port 16(lan16) entered listening state
[  475.715472] rtl83xx-switch switch@1b000000 lan15: Link is Up - 1Gbps/Full - flow control rx/tx
[  475.726063] switch: port 15(lan15) entered blocking state
[  475.732137] switch: port 15(lan15) entered listening state
[  483.065255] switch: port 16(lan16) entered learning state
[  484.345219] switch: port 15(lan15) entered learning state
[  491.385220] switch: port 16(lan16) entered forwarding state
[  491.391495] switch: topology change detected, propagating
[  492.665253] switch: port 15(lan15) entered forwarding state
[  492.671525] switch: topology change detected, propagating

ip link

...
17: lan15@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master switch state UP mode DEFAULT group default qlen 1000
    link/ether ee:43:f6:fe:07:fa brd ff:ff:ff:ff:ff:ff permaddr ec:43:f6:fe:07:ec
18: lan16@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master switch state UP mode DEFAULT group default qlen 1000
    link/ether ee:43:f6:fe:07:fb brd ff:ff:ff:ff:ff:ff permaddr ec:43:f6:fe:07:ec
...

brctl showstp switch

switch
 bridge id              7fff.ec43f6fe07ec
 designated root        7fff.ec43f6fe07ec
 root port                 0                    path cost                  0
 max age                  10.00                 bridge max age            10.00
 hello time                1.00                 bridge hello time          1.00
 forward delay             8.00                 bridge forward delay       8.00
 ageing time             300.00
 hello timer               0.09                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                 283.68
 flags

...
lan15 (15)
 port id                800f                    state                forwarding
 designated root        7fff.ec43f6fe07ec       path cost                  4
 designated bridge      7fff.ec43f6fe07ec       message age timer          0.00
 designated port        800f                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.52
 flags
...
lan16 (16)
 port id                8010                    state                forwarding
 designated root        7fff.ec43f6fe07ec       path cost                  4
 designated bridge      7fff.ec43f6fe07ec       message age timer          0.00
 designated port        8010                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.01
 flags
...

[brada4]

... so wire protocol is not sent and/or received correctly...
Since you can test on a single switch - disable 3 surplus links for now.
Connect wireshark to one port, your pc needs to get stp pdu every 1..5..30 secs or.. not. - ie is there transmission or not.

Make linux bridge with STP on physical port, confirm with tcpdump/wireshark it sends STP PDU. It shall stop forwarding or ifdown switch port - ie if the PDU is recognized by switch.

[joe2xyz]

Connect wireshark to one port, your pc needs to get stp pdu every 1..5..30 secs

Yes, switch is sending STP packets.

I'll test Linux bridge with STP now...

[joe2xyz]

• on Linux PC:

sudo -i
ip link add name mybridge type bridge stp_state 1
ip link set mybridge up
ip link set myif1 up
ip link set myif1 master mybridge
bridge link

4: myif1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master mybridge state disabled priority 32 cost 100

ip a add 10.0.0.251/24 dev mybridge

• connect ethernet cable: myif1 -> OpenWrt port
• a minute later on Linux PC:

brctl showstp mybridge

mybridge
 bridge id              8000.aa7fb2d2dfda
 designated root        7fff.4c9eff8cdd13
 root port                 1                    path cost                  5
 max age                  10.00                 bridge max age            10.00
 hello time                1.00                 bridge hello time          1.00
 forward delay             8.00                 bridge forward delay       8.00
 ageing time             300.00
 hello timer               0.00                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                 150.70
 flags


myif1 (1)
 port id                8001                    state                forwarding
 designated root        7fff.4c9eff8cdd13       path cost                  5
 designated bridge      7fff.4c9eff8cdd13       message age timer          9.73
 designated port        8006                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.00
 flags

findings

• OpenWrt correctly sends STP packets
• Linux PC correctly interprets STP packets and sets OpenWrt as Root bridge (Linux PC bridge ID is 8000.aa7fb2d2dfda, designated root is 7fff.4c9eff8cdd13 which is OpenWrt bridge ID)
• OpenWrt ignores STP packets sent by other devices OR OpenWrt receives them but doesn't process them correctly

[brada4]

receives them but doesn't process

Probably tag developer who helped you lately with vlans?

[joe2xyz]

Hi @svanheule and @luizluca , tagging you on brada4's request.

[KanjiMonster]

OpenWrt ignores STP packets sent by other devices OR OpenWrt receives them but doesn't process them correctly

Have you checked if OpenWrt actually receives any STP packets? They should be visible in tcpdump. Maybe the switch chip filters them out by default instead of redirecting them to the CPU port.

[joe2xyz]

Good question @KanjiMonster , thank you. I can try it but I assume CPU should not process STP anyway. That should be handled by the switch ASIC.

[KanjiMonster]

Good question @KanjiMonster , thank you. I can try it but I assume CPU should not process STP anyway. That should be handled by the switch ASIC.

In my experience STP is never handled by the ASIC, and needs to be implemented in the CPU. The ASIC just supports filtering STP packets / redirecting to CPU and setting port (M)STP config, but does not implement the protocol itself. At least I haven't seen any ASICs that implement STP themselves.

So AFAIK for STP to work, the CPU needs to see the STP packets.

[joe2xyz]

OK, that makes sense if the CPU is getting only STP packets and not for example TCP then it can handle STP on all ports. I'll try tcpdump on eth0 which I'm assuming is the CPU link.