Test Seeed Studio's Dual Gigabit CM4 Carrier Board

[geerlingguy]

See the Dual Gigabit Ethernet Carrier Board for Raspberry Pi Compute Module 4.

It has two USB 3.0 ports, plus an internal USB 3.0-to-Gigabit Ethernet bridge chip so it can have two Gigabit Ethernet ports.

I'm going to test it alongside the DFRobot Routerboard.

[ruobinl]

Is the seeed studio's dual gigabit cm4 carrier board a commercial board? I was a little confused about on your web . and seeed seems to launch a new carrier board for cm4 named reterminal. Maybe you can add it to your web, and, also, Hope to see your test! Here is the link of that boardhttps://wiki.seeedstudio.com/reTerminal/

[geerlingguy]

@ruobinl - Can you open a new separate issue for the reTerminal? I would love to get it added.

Seeed's board seems to be available now, so if it's still under 'prototype' it should be updated on the site.

[geerlingguy]

This board uses a USB-to-Gigabit network adapter, the Microchip LAN7800. It's been in the mainline Linux kernel since 4.x I think, but on first attempt with OpenWRT it didn't seem to identify the controller.

Might have to dig in a bit more or build a custom kernel :/

[geerlingguy]

From https://downloads.openwrt.org/snapshots/targets/bcm27xx/bcm2711/ I downloaded the rpi-4-ext4-factory.img.gz image and expanded it.

I then wrote it to the card:

sudo dd if=/Users/jgeerling/Downloads/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img of=/dev/rdisk2 bs=1m

Then transferred the card to the Seeed board and booted. The Pi starts booting, and the lights on the ETH0 port light up, and the kernel gets to this point but seems to stop booting:

IPv6: ADDRCONF(NETDEV_CHANGE): lan: link becomes ready

[geerlingguy]

Hmm... there's actually an rc build of 21.02 from April 20! See: https://downloads.openwrt.org/releases/21.02.0-rc1/targets/bcm27xx/bcm2711/

I'll give that a go.

Edit: Note, exact same issue. I also noticed the board gets HOT. Maybe packing in the USB controller, Ethernet controller, and Pi right on top (plus looks like some power bits on the bottom?) is not the best solution for thermals. I'll test it under IR in a bit.

[geerlingguy]

I just flashed normal Pi OS to the microSD card and am trying to boot from that.

[geerlingguy]

Pi OS boots, and I get:

pi@raspberrypi:~ $ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 3: Dev 2, If 0, Class=Vendor Specific Class, Driver=lan78xx, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

And it looks like the interface is there and it's up, but the LEDs on the port don't actually light up... that's kind of a bummer.

pi@raspberrypi:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:e8:6a:ba brd ff:ff:ff:ff:ff:ff
    inet 10.0.100.12/24 brd 10.0.100.255 scope global dynamic noprefixroute eth0
       valid_lft 86309sec preferred_lft 75509sec
    inet6 fe80::a6a8:e8ff:ecc1:7e92/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether da:3c:45:93:55:7e brd ff:ff:ff:ff:ff:ff
    inet 169.254.166.79/16 brd 169.254.255.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::6b80:8576:92b1:dc4e/64 scope link 
       valid_lft forever preferred_lft forever

[geerlingguy]

Going to try to control the LEDs:

sudo apt-get install -y libusb-1.0-0-dev
git clone https://mockmoon-cybernetics.ch/cgi/cgit/linux/lan7800-led-ctl.git
cd lan7800-led-ctl
make
sudo make install

Using it:

pi@raspberrypi:~/lan7800-led-ctl $ sudo lan7800-led-ctl -h
usage: lan7800-led-ctl [--led0=x][--led1=x]
	where x is one of:
	0 - turn LED off
	1 - turn LED on
	s - LED shows status
pi@raspberrypi:~/lan7800-led-ctl $ sudo lan7800-led-ctl --led0=1
setting LED0 to status 1
turning LED on not supported yet
pi@raspberrypi:~/lan7800-led-ctl $ sudo lan7800-led-ctl --led0=0
setting LED0 to status 0
pi@raspberrypi:~/lan7800-led-ctl $ sudo lan7800-led-ctl --led0=s
setting LED0 to status 2
pi@raspberrypi:~/lan7800-led-ctl $ sudo lan7800-led-ctl --led1=s
setting LED1 to status 2

Nothing made a difference, I have to wonder if maybe the LEDs aren't wired in at all somehow?

[geerlingguy]

Going to set up the Seeed board as a very simple router:

1. Edit /etc/dhcpcd.conf and add the following static IP config at the bottom:

   interface eth1
   static ip_address=192.168.1.1/24
   static routers=192.168.1.1 
   static domain_name_servers=10.0.100.52 
   

2. Reload dhcpcd: sudo systemctl daemon-reload && sudo systemctl restart dhcpcd

3. I ran the script in Ubuntu's guide for Enable IP forwarding and masquerading.

4. On my Mac, adjust the interface connected to the Pi's eth1 port and manually set it to 192.168.1.2

At this point I disconnected all my other Mac interfaces and was getting all my connectivity through the Pi.

[geerlingguy]

(Reminder: all benchmarks are run from my Mac, connected solely through the Pi router (WiFi disabled, 10G NIC is unplugged) to the rest of my network.)

First benchmark is Speedtest.net, which gives 466 Mbps down, and 39 Mbps up.

Second is iperf3, between my Mac and my ASUS WiFi 6 router: 420 Mbps (and varies a bit). I noticed with atop that the irq interrupts were maxing out a core:

Going in reverse (iperf3 -c [host] -R) I was able to get 500-520 Mbps. Bidirectional:

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  23.5 MBytes  19.8 Mbits/sec                  sender
[  5][TX-C]   0.00-10.00  sec  23.5 MBytes  19.7 Mbits/sec                  receiver
[  8][RX-C]   0.00-10.00  sec   585 MBytes   491 Mbits/sec    0             sender
[  8][RX-C]   0.00-10.00  sec   582 MBytes   488 Mbits/sec                  receiver

[geerlingguy]

Next up, calculating packets per second (pps), I got:

$ sudo time hping 10.0.100.10 -q -i u30 --icmp|tail -n10
^C
--- 10.0.100.10 hping statistic ---
165731 packets tramitted, 165716 packets received, 1% packet loss
round-trip min/avg/max = 0.3/0.6/18.7 ms
Command terminated abnormally.
        9.03 real         1.20 user         2.70 sys

(Note that to decrease the massive amount of packet loss, I doubled the interval from 15 to 30.)

Result: 36,703 pps, which is just less than half of the result I got on DFRobot's board.

Using LAN-over-USB is really killer—or maybe just the driver/featureset of the Microchip LAN7800 :/

[geerlingguy]

I'm re-flashing the OS and will test just the USB-to-Gigabit side (LAN1 port) by itself, using iperf3 between my Mac and the Pi, just to see if not having to route the traffic internally leads to any better speeds. (Mostly to see if the interface itself is slow, or it is just slow as an overall system on Pi OS.)

[geerlingguy]

So even on the Pi direct (I have it's LAN1 port plugged into my regular home network, fresh Pi OS install with just iperf3 installed, no routing), I tested throughput between it and three different known-good devices (my Mac, my ASUS router, and another Pi that consistently gets 930+ Mbps), and the USB-to-Gigabit port only gives 650-700 Mbps with no routing whatsoever.

In reverse (to the Pi), I can get 940 Mbps from my router (but interestingly, not from my Mac connected directly through my office switch). But traffic flowing from the Pi over that interface maxes out at 700 Mbps.

On the built-in interface, I'm seeing 930-940 Mbps consistently.

[6by9]

LAN7800 is the same USB to gigabit ethernet chip used on Pi3B+, so it is in the bcm2709, bcm2711, and aarch64/bcmrpi3 and aarch64/bcm2711.
Note that it is a USB2 device, so limited to the absolute max of 480Mbit/s (more reasonably 300Mbit/s), and therefore generally wants flow control enabled on the network. Edit: it's the Pi3B+'s interface that is limited to USB2, not the chip.

LED support depends on the EEPROM or OTP setting in the chip. See raspberrypi/linux@c0e68a6 for a patch added to handle things in the absence of an EEPROM (eg on Pi3B+)
We also set the default LED modes via device tree (even though it's a USB device) - https://github.com/raspberrypi/linux/blob/rpi-5.10.y/arch/arm/boot/dts/bcm283x-rpi-lan7515.dtsi

[SeeedRobin]

Please try to download OpenWRT from:https://openwrt.org/toh/hwdata/raspberry_pi_foundation/raspberry_pi_foundation_raspberry_pi_4_b "Firmware OpenWRT snapshot install"
I guess, since the board gets hot, the speed can't reach the fastest. Maybe you can try cooling it.

[geerlingguy]

@SeeedRobin - Good point, the board does get quite hot (and not just the CM4 — it seems to be staying under throttling temperatures). I'll try putting a decent amount of airflow through the gap between the board and CM4 and see if that helps.

And @6by9 I'll see what I can do—maybe I can figure out a workaround using the device tree or something from that patch. Didn't realize it's the same chip used on the 3B+!

[geerlingguy]

@SeeedRobin - I tried downloading the openwrt-bcm27xx-bcm2711-rpi-4-squashfs-factory.img.gz image, expanding it, flashing it to the microSD card with dd, and booting the Pi, but it got stuck in a reboot loop with the message:

No filesystem could mount root, tried:
 squashfs
 ext4

Kernel panic - not syncing: VFS: Unable to mount root fs...

I'm going to try the openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz image instead and see if that boots.

[geerlingguy]

With openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz loaded, the Pi just sits and hangs the same way it did last time I tried the ext4 snapshot image yesterday (same as here: #137 (comment)).

IPv6: ADDRCONF(NETDEV_CHANGE): lan: link becomes ready

...oooooh... Just found this comment:

The snapshots don't have a GUI (web interface is called LuCI) compiled in.

You can install it however by running

# opkg update
# opkg install luci
# reboot

Then try http://192.168.1.1 10 . See also the Quick start guide 44 from the homepage, which contains a detailed explanation.

To use the snapshot I'll have to get a keyboard plugged in, which means I'll have to enable the CM4's USB port in config.txt.

Actually, since I can use the USB 3.0 ports, I don't have to enable USB 2.0 in /boot/config.txt. Yay!

[geerlingguy]

All right, I have the Pi booted and tried running opkg update, but it seems like the WAN side is not connecting. I tried plugging in on LAN0 and LAN1 but neither one is picking up DHCP. ip a only shows eth0 and lan, and lan has a static IP of 192.168.1.1.

[geerlingguy]

Trying the factory 21.02 rc release now (https://downloads.openwrt.org/releases/21.02.0-rc1/targets/bcm27xx/bcm2711/). Same result. Going to see if I can manually add a WAN port (https://www.onetransistor.eu/2017/04/wan-port-openwrt-lede-vlan.html).

Via console, ran:

uci set network.lan.ipaddr=10.0.100.221
uci set network.lan.gateway=10.0.100.1
uci set network.lan.dns=10.0.100.52
uci commit network
reboot

And that did it! I can get opkg update to work and see the Internet from OpenWRT.

[geerlingguy]

Weird thing, though—for some reason even though the USB 3.0 ports are working (I'm typing on a keyboard plugged into one of them), the 2nd network interface seems to not be recognized. In LuCI it's not showing me an option of another interface for a WAN port...

Weird, because it 'just works' in Pi OS, but it's not seen at all using OpenWRT. Going to install USB utils and PCI tools and see what the bus says.

[geerlingguy]

Installed usbutils using the Web UI and on the console I ran lsusb -tv:

root@OpenWrt:~# lsusb -tv
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    ID 1d6b:0003  
    |__ Port 3: Dev 2, If 0, Class=, Driver=, 5000M
        ID 0424:7800  
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    ID 1d6b:0002  
    |__ Port 1: Dev 2, If 0, Class=, Driver=hub/4p, 480M
        ID 2109:3431  
        |__ Port 2: Dev 3, If 0, Class=, Driver=hub/4p, 480M
            ID 05e3:0610  
            |__ Port 1: Dev 4, If 0, Class=, Driver=usbhid, 1.5M
                ID 04d9:0006  
            |__ Port 1: Dev 4, If 1, Class=, Driver=usbhid, 1.5M
                ID 04d9:0006 

0424:7800 is, I believe, the 7800 chip. So not sure why it's not loading.

root@OpenWrt:~# dmesg | grep 2-3
[    0.936510] usb 2-3: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    0.960712] usb 2-3: New USB device found, idVendor=0424, idProduct=7800, bcdDevice= 3.00
[    0.967337] usb 2-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0

[geerlingguy]

I wonder if the BCM2711 config target used for the Pi 4-compatible OpenWRT image doesn't include the CONFIG_USB_LAN78XX flag, therefore doesn't include the driver... lsmod doesn't show it.

[6by9]

    Port 3: Dev 2, If 0, Class=, Driver=, 5000M
        ID 0424:7800

Looks like they've removed the lan78xx driver from their kernel then. Can you do the normal sudo modprobe configs zcat /proc/config.gz | grep LAN78 to see whether they have built the module?

[geerlingguy]

@6by9 - Yeah, looks like that's the case (you and I came to the same conclusion within 23 seconds ;)

root@OpenWrt:~# zcat /proc/config.gz | grep LAN78
# CONFIG_USB_LAN78XX is not set

Definitely not in their build for the 2711: https://github.com/openwrt/openwrt/blob/master/target/linux/bcm27xx/bcm2711/config-5.4

For now, I'm thinking I'll stick with not using OpenWRT just because I've already put in a bit more time than I should be on this particular board. I'll re-benchmark on Pi OS with better cooling and see what the results are there. And I'll have to mention in my eventual video that right now, at least, OpenWRT's images don't have support for the 2nd port on the Seeed Studio board (AFAICT).

Seeed could build a custom image like DFRobot is currently to add in that support, or maybe see if OpenWRT would be willing to merge this PR (openwrt/openwrt#4191) to add support upstream.

(cc @SeeedRobin)

[bobafetthotmail]

I can recompile an OpenWrt with that change and provide an image so you can do your tests.

It's pretty quick to do, I have a build environment for OpenWrt set up already.

[bobafetthotmail]

Here a zip with images compiled. There is also luci web interface included https://drive.google.com/file/d/1ARejcyKqpbqC6mpM_amGMPDCSyloddFA/view?usp=sharing

[geerlingguy]

@bobafetthotmail - Thanks! I'll test it out tonight or tomorrow morning!

[geerlingguy]

@bobafetthotmail - I have it installed, had to set up a custom IP on the eth0 interface to access it and get opkg to work (following these directions: #137 (comment)). Checking with dmesg:

root@OpenWrt:~# dmesg | grep 78xx
[    0.324560] usbcore: registered new interface driver lan78xx
[    2.199548] lan78xx 2-3:1.0 (unnamed net_device) (uninitialized): No External EEPROM. Setting MAC Speed
[    2.216663] libphy: lan78xx-mdiobus: probed
[    2.222143] lan78xx 2-3:1.0 (unnamed net_device) (uninitialized): int urb period 64

And it shows up as a usable interface:

root@OpenWrt:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000
    link/ether dc:a6:32:e8:6a:ba brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:9a:be:6e:96:e0 brd ff:ff:ff:ff:ff:ff
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether dc:a6:32:e8:6a:ba brd ff:ff:ff:ff:ff:ff
    inet 10.0.100.221/24 brd 10.0.100.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fdb4:284:1ecd::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::dea6:32ff:fee8:6aba/64 scope link 
       valid_lft forever preferred_lft forever

So looks like it works! I'll reformat my PR in a bit.

[bobafetthotmail]

good.
As for the PR there is no rush, review and merge process in OpenWrt is slow, it can take months.

In devices with more than one port, OpenWrt usually sets up the second one as WAN and dhcp client (so it becomes a router). But obviously this is not the case for raspberry because it's defined as single-port device and so it is configured just with a LAN interface.

You can add a new interface to use that new port by going to Network -> Interfaces and then click on the button Add a new interface on the bottom left.

Then you give a name like WAN, and choose "Static interface" or "DHCP client".
Then you go in the Physical tab and at the option at the bottom you can select eth1 so this new port is assigned to interface WAN, then you can go in the Firewall tab and assign it a firewall zone like wan.

then save the change.

Otherwise if you want to use commandline, add this to the /etc/config/network

config interface 'wan'
	option proto 'dhcp'
	option type 'bridge'
	option stp '1'
	option igmp_snooping '1'
	option ifname 'eth1'

to create a WAN interface set as dhcp client

and add this (or check if it's there already, I don't know) to /etc/config/firewall
to create wan firewall zone and put WAN interface in it

config forwarding
	option src 'lan'
	option dest 'wan'

config zone
	option name 'wan'
	option output 'ACCEPT'
	option forward 'REJECT'
	option mtu_fix '1'
	option masq '1'
	option input 'REJECT'
	option network 'wan'

Now it is set up like a router, with a LAN and a WAN interfaces.

then run

service firewall restart
service network restart

so the system reloads the configs you just changed.

[geerlingguy]

@bobafetthotmail - I can confirm that all I needed to do was make the following addition to /etc/config/network and then run service firewall restart && service network restart:

config interface 'wan'
	option proto 'dhcp'
	option type 'bridge'
	option stp '1'
	option igmp_snooping '1'
	option ifname 'eth1'

The firewall rules were already present. Now it's working the same as the DFRobot board, so I can do 1:1 comparisons, yay!

[geerlingguy]

I thought I'd try getting WiFi working on the Seeed board with it's OpenWRT install—and I finally ran into this forum topic, which led me to the iw reg get command:

root@OpenWrt:~# iw reg get
global
country 00: DFS-UNSET
	(2402 - 2472 @ 40), (N/A, 20), (N/A)
	(2457 - 2482 @ 20), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
	(2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, PASSIVE-SCAN
	(5170 - 5250 @ 80), (N/A, 20), (N/A), AUTO-BW
	(5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, AUTO-BW, PASSIVE-SCAN
	(5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, PASSIVE-SCAN
	(5735 - 5835 @ 80), (N/A, 20), (N/A), PASSIVE-SCAN
	(57240 - 63720 @ 2160), (N/A, 0), (N/A)

Looks like I need to set the country code and maybe then WiFi can work? Well, I tried setting it but still nothing:

root@OpenWrt:~# iw reg set US
root@OpenWrt:~# iw reg get
global
country US: DFS-FCC
	(2400 - 2472 @ 40), (N/A, 30), (N/A)
	(5150 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
	(5250 - 5350 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
	(5470 - 5730 @ 160), (N/A, 23), (0 ms), DFS
	(5730 - 5850 @ 80), (N/A, 30), (N/A)
	(57240 - 71000 @ 2160), (N/A, 40), (N/A)

root@OpenWrt:~# iw dev
root@OpenWrt:~# wifi status
{
	
}

Kernel module is loaded, too:

root@OpenWrt:~# lsmod | grep brcmfmac
brcmfmac              270336  0 
brcmutil               16384  1 brcmfmac
cfg80211              348160  1 brcmfmac
compat                 16384  2 brcmfmac,cfg80211

Hmm... dmesg:

root@OpenWrt:~# dmesg | grep brcmfmac
[    6.865501] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    6.889369] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.txt failed with error -2
[    6.901659] brcmfmac mmc1:0001:1: Falling back to sysfs fallback for: brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.txt
[    6.917922] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.txt failed with error -2
[    6.927674] brcmfmac mmc1:0001:1: Falling back to sysfs fallback for: brcm/brcmfmac43455-sdio.txt
[   11.913882] usbcore: registered new interface driver brcmfmac
[   12.933667] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50

Ah... found openwrt forum: Raspberry Pi CM4 wifi not working — I'm not alone.

[geerlingguy]

Trying the following based on this slackware forum topic:

cd /lib/firmware/brcm
cp brcmfmac43455-sdio.raspberrypi,4-model-b.txt brcmfmac43455-sdio.raspberrypi,4-compute-module.txt
reboot

...and voila! It works! I posted this to the OpenWRT forum as well.

root@OpenWrt:~# uci show wireless
wireless.radio0=wifi-device
wireless.radio0.type='mac80211'
wireless.radio0.channel='36'
wireless.radio0.hwmode='11a'
wireless.radio0.path='platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1'
wireless.radio0.htmode='VHT80'
wireless.radio0.country='US'
wireless.radio0.cell_density='0'
wireless.default_radio0=wifi-iface
wireless.default_radio0.device='radio0'
wireless.default_radio0.network='lan'
wireless.default_radio0.mode='ap'
wireless.default_radio0.ssid='OpenWrt'
wireless.default_radio0.encryption='none'
wireless.wifinet1=wifi-iface
wireless.wifinet1.device='radio0'
wireless.wifinet1.mode='ap'
wireless.wifinet1.ssid='OpenWrt'
wireless.wifinet1.encryption='psk-mixed'
wireless.wifinet1.key='mineralocorticoids'

[geerlingguy]

Except it doesn't; can't see SSID after setting it in AP mode, and getting this in dmesg:

[   14.485010] ieee80211 phy0: brcmf_cfg80211_start_ap: Set Channel failed: chspec=57386, -52
[   14.496398] ieee80211 phy0: brcmf_cfg80211_add_iface: iface validation failed: err=-16
[  234.339902] device wlan0 left promiscuous mode
[  234.344441] br-lan: port 2(wlan0) entered disabled state
[  236.273290] br-lan: port 2(wlan0) entered blocking state
[  236.278609] br-lan: port 2(wlan0) entered disabled state
[  236.284038] device wlan0 entered promiscuous mode
[  236.291274] ieee80211 phy0: brcmf_run_escan: error (-52)
[  236.296598] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
[  236.819939] br-lan: port 2(wlan0) entered disabled state
[  236.828504] device wlan0 left promiscuous mode
[  236.833024] br-lan: port 2(wlan0) entered disabled state
[  236.887841] ieee80211 phy0: brcmf_run_escan: error (-52)
[  236.893192] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)

root@OpenWrt:~# iw dev
phy#0
	Interface wlan0
		ifindex 4
		wdev 0x1
		addr dc:a6:32:fe:89:91
		type AP
		channel 36 (5180 MHz), width: 20 MHz, center1: 5180 MHz

[geerlingguy]

I set the channel to 100 (5500 MHz) manually and it seemed like the scan errors went away (see this forum topic), but I'm still not seeing the SSID on any of my other devices... it seems like everything is configured correctly in OpenWRT though.

[geerlingguy]

OH HEY! Tada:

I'm connected via WiFi now too. Going to unplug wired and do some speed tests there...

[geerlingguy]

Ouch; download is 58.94 Mbps, upload is 31.18 Mbps. To be expected—even without AP mode, onboard WiFi never gets past about 80-90 Mbps download speeds in my experience (even with an external antenna on the CM4).

iperf3 gave 62.2 Mbps.

But... this is making me more excited about the possibilities with my dual-2.5 GbE 802.11ax 6E build I'm going to attempt!

Plus... if you're someone with < 50 Mbps of Internet, or you just stream < 4K video to one or two devices—60 Mbps is plenty adequate. The built-in chip kinda falls apart with more than maybe 10 devices connected, though.

[bobafetthotmail]

Yeah that's because the wifi module is wired on the SDIO interface. It's decent enough for what it is, but it's not AP-grade hardware by any stretch, anything on PCIe lanes is going to be so much better.

[geerlingguy]

@bobafetthotmail - Definitely; but a lot of people might not know that and would expect that "The Pi has 802.11ac, so it should be as good as [insert router here] with 802.11ac!" :)

Have to get the raw numbers and be able to demonstrate how much different it is. I wish I had a wireless testing lab like giant megacorps have, I'd probably spend a few months on just one device :D

[geerlingguy]

I have no idea what just happened, but my results comment just completely vanished, so I have to re-test some things:

• Speedtest.net: 678.59 Mbps down / 39.92 Mbps up
• iperf3 to ASUS router through board: 679 Mbps
• hping: 36,480 pps
• Idle current: 0.78A @ 5.08V (3.96W)
• Full usage current: 0.98A @ 5.05V (4.95W)
• Max temp of board component: 54°C (IC on bottom — I can't measure the NIC and USB chips on top since they're sandwiched between the CM4 and the board itself)

[mingzhangqun]

@geerlingguy , hi!
About the speed, you can try this:
https://wiki.seeedstudio.com/Dual-Gigabit-Ethernet-Carrier-Board-for-Raspberry-Pi-CM4/#ethernet-ports-configuration

[geerlingguy]

@mingzhangqun - Thanks; looks like the script in question is here: https://github.com/Seeed-Studio/seeed-linux-dtoverlays/blob/mipi_dsi/scripts/cm4_lan7800.sh

I'll give this a try and re-run speed tests to see what the difference is. Did you run it and see what the overall difference was? And is there any chance the optimizations you have in that repository might be contributed back to the Linux kernel so it can be improved for other users of the LAN7800 chip?

[6by9]

https://wiki.seeedstudio.com/Dual-Gigabit-Ethernet-Carrier-Board-for-Raspberry-Pi-CM4/#ethernet-ports-configuration

Ethernet Ports Configuration

Once you flash the Raspberry Pi OS onto the CM4 module, you will get the full Gigabit speed (1Gbps) only on the Ethernet port connected to the Gigabit Ethernet PHY of the CM4 module which is based on Broadcom BCM54210PE (right-side port). The port which is connected to the Microchip's LAN7800 USB 3.0 to GbE (Gigabit Ethernet Bridge) (left-side port), will not provide the full speed of 1Gbps, but rather a much reduced speed. This is because the lan78xx driver in the raspberry core is not up to date.

Odd statement.
The only mainline patches added between 5.10 and 5.12 are:
c23d544e995f net: usb: lan78xx: use new tasklet API
06cd7c46b3ab net: usb: lan78xx: Remove lots of set but unused 'ret' variables

Neither of which have any functional difference.

There are a couple of downstream patches, and potentially raspberrypi/linux@d5a91b7 will alter the total throughput because it changes the urb period (can be changed through a module parameter), and disabling Transmit Segment Offload pushes more processing onto the CPU (again should be possible to disable through a module parameter).

Minor concern that your version of the files don't match up with mainline, then again it appears at first glance to only be a module rename from lan78xx to lan7800.

[geerlingguy]

@6by9 - My guess is it won't make a substantial difference, but I'll test it to be thorough. Most of the USB-to-gigabit adapters I've tested on the Pi seem to top out under 800 Mbps anyways, so it's not a big surprise.

[geerlingguy]

Interesting result—iperf3 direct to the board through the LAN7800 chip now gives 940 Mbps:

pi@raspberrypi:~ $ iperf3 -c 10.0.100.66
Connecting to host 10.0.100.66, port 5201
[  5] local 10.0.100.130 port 42622 connected to 10.0.100.66 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   110 MBytes   922 Mbits/sec    0    246 KBytes       
[  5]   1.00-2.00   sec   113 MBytes   944 Mbits/sec    0    246 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    246 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   939 Mbits/sec    0    246 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    246 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    246 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    246 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec    0    246 KBytes       
[  5]   8.00-9.00   sec   113 MBytes   944 Mbits/sec    0    304 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    304 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.09 GBytes   938 Mbits/sec                  receiver

I'll set it up as a router and see if it can pump more packets through too...

Just noting that it's a lot harder to try to get these changes into an OpenWRT build (compared to downloading git repo and running a shell script...). And I'd like to know what changes specifically lead to this result?

[geerlingguy]

Acting as a router, my Internet speed test results are actually a bit worse: https://www.speedtest.net/result/11480334630

Monitoring with atop I see the interrupts pretty much kill the CPU on one core still.

So while it seems the patch does help in terms of hitting one interface, total router throughput (between the two interfaces) still seems to suffer quite dramatically...

iperf3 does give me 930 Mbps now, through the router, but actual Internet traffic gets queued in IRQ interrupts.

hping test gives 36,773 packets per second, which is in line with my previous testing.

So it seems the optimizations in the Seeed repo only really affect iperf3 performance numbers?

[bobafetthotmail]

Since Raspi is quadcore, you can try enabling multicore routing or "packet steering" by
uci set network.globals.packet_steering='1'
uci apply network
service network restart
the Luci Web interface description for this option (in Interfaces > GLobal Network Options tab) says "Enable packet steering across all CPUs. May help or hinder network speed."

Maybe it will spread that 81% CPU load more on other cores too.

Another thing you can enable to get more performance is "Software Flow Offloading" that is a firewall option to bypass some intensive routing functions (hence why the option says that "may break SQM/QoS functionality", which is the Quality of Service packet prioritization, optional functionality not everyone uses)
It's in Network > Firewall

Or edit /etc/config/firewall to add option flow_offloading '1' to the defaults section

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading '1'

None of these options are specific for Raspberry, if you have other devices that have multiple CPU cores or you are not using QoS/SQM packages you can also try them.

[geerlingguy]

@bobafetthotmail - I'll test that out under OpenWRT soon—just a note that last time I was fiddling with NICs on the Pi, I was unable to get anything to work for distributing packets among cores. I don't remember which board it was, but I found that at least irqbalance would have no effect on the Pi whatsoever.

[geerlingguy]

Video is up today! https://www.youtube.com/watch?v=w7teLVwi408

[wulfy23]

just a short note on the above... ( re: irq )...

rps / xps (packet_steering) is said to assist with 'multi client , multi stream'... although YMMV varies depending on nic drivers ( onboard supports a bit... most usb not so much... )

biggest improvement imho is 'irq_affinity' for both eth0 interrupts ( 'c' or 'a' maybe )... the aim is to distribute eth0 interrupts across several cores... ( which afaik is also the only thing irqbalance is capable of also distributing )... usb interrupts are likely also statically somewhat bound... so we distrubute eth0 as this is a known distributable entity...