Same MAC address on all SONiC devices

[marian-pritsak]

SONiC switches when connected one to another cannot resolve each other because their ports have the same MAC. We need to provide MAC address distribution based on base MAC from device EEPROM.

Here's what is observed when I try to ping one SONiC switch from another:

root@arc-switch1025:/home/admin# ping 20.0.0.5
PING 20.0.0.5 (20.0.0.5) 56(84) bytes of data.
09:04:57.386041 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=1 Destination Host Unreachable
09:04:58.555442 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:04:59.554031 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:00.554003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=2 Destination Host Unreachable
From 20.0.0.4 icmp_seq=3 Destination Host Unreachable
From 20.0.0.4 icmp_seq=4 Destination Host Unreachable
09:05:01.571378 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:02.570035 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:03.570071 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=5 Destination Host Unreachable
From 20.0.0.4 icmp_seq=6 Destination Host Unreachable
From 20.0.0.4 icmp_seq=7 Destination Host Unreachable
^C
--- 20.0.0.5 ping statistics ---
8 packets transmitted, 0 received, +7 errors, 100% packet loss, time 7031ms
pipe 3
root@arc-switch1025:/home/admin# 09:05:04.587432 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:05.586050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:06.586058 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28

Logs from second switch:

09:04:53.329642 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:54.329629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387772 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387809 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:56.385577 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:56.385602 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:57.385587 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:57.385630 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:58.554960 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:58.554989 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:59.553578 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:59.553621 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:00.553522 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:00.553554 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:01.570977 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:01.571022 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:02.569651 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:02.569698 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:03.569617 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:03.569665 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:04.587050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:04.587094 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:05.585667 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:05.585712 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:06.585616 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:06.585666 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28

Second switch sends ARP reply with same MAC as on first, which cannot be handled correctly.

[lguohan]

the orchagent has -m option to specify the switch mac address. There are some issues on mellanox platform to support SWITCH_ATTR_SRC_MAC_ADDRESS. On mellanox platform, a k/v is proposed to set up the switch mac, but this has not been integrated yet into orchagent.

[xinliu-seattle]

Reopen. Add to backlog. Will close when the fix is integrated to Orch-agent.