Test 4-interface Intel NIC I340-T4 #3
Comments
|
Looking forward to the results. I am curious if it will work with something like openwrt and make a killer home router. Cheers |
|
@annunity you and me both! I just got it plugged in for my initial "I should be getting to bed but I'll test this instead" routine :) |
So far so good. One of the interfaces using |
|
PCIe interface seems happy with the default BAR space: |
|
However, not seeing the new interfaces by default: |
|
Glancing around in To download the .zip archive, you have to do it in a browser since it requires the manual acceptance of a download license. That's annoying, because I had to download it to my Mac then copy it over to the Pi via SCP. |
|
All right, so after digging into that process and realizing that downloads like the entire universe of Windows and Linux drivers in a giant archive, I found that the Intel Linux base driver page mentions:
So I downloaded that driver and manually copied it to the Pi, then: |
|
After seeing noseka1/linuxband#12, I monkey-patched the However, now it's complaining about permissions when it tries to copy over manfiles: So trying with |
|
Rebooted and... BINGO! Now, time to pull out a gigabit switch and start testing the maximum bandwidth I can pump through everything with |
|
Hehehe this is so dumb but fills me with joy:
I don't even have enough ports on my little office switch, I'm going to have to go into storage to grab my bigger switch tomorrow. And after plugging in another connection to one of the jacks on the board: |
|
Haha nice. Probably still less messy this using 4 usb nics. This is definitely one of those because you can sort of projects. |
|
So one thing I would like to take a look at is OpenWRT. And just seeing how many bits I can push through the Pi per second using Edit: Some ideas to test: |
|
Just a heads up. Openwrt for the pi 4 is still using daily snapshots or you can compile it yourself. The snapshots work fine if you are just testing but you will have trouble updating the packages after a while. The main appeal for me is to use load balancing or vlans in Openwrt. Beats having a large expensive switch. |
|
Just testing one interface with iperf: On the Pi: On my Mac: Just doing a quick test on my Mac with two interfaces, I found that I got: So 964 Mbits/sec total, which just shows that my local 1 Gbps switch has, as expected, 1 Gbps of total capacity. Now, I think what I'll do to try to test all five GigE ports at once is to have my Pi Dramble (4x Pi 4s) connecting directly to each of the ports on the Intel NIC, with a hardcoded IP address for each connection. Then have my Mac connected over the network to the wired Ethernet on the CM4. Then set up five instances of This could get messy. I know the cabling at least will get messy. |
|
For each Pi, I prepped the This is a tedious and annoying process, but this is research for science so it is worth it. The next step will be building a small Ansible playbook that creates little networks for each of the interfaces, using the same IP and netmask between the individual Pis and the Intel NIC interfaces... or maybe doing it all by hand. It's one of those XKCD 1319 issues. (Edit: I went manual for this part.) |
|
Now I added a mapping of static IPs to And rebooted. It looks like all the links are up, so it's time to start seeing if I can hit multiple interfaces at the same time and get more than 1 Gigabit pumping through this Pi! EDIT: That mapping doesn't work, because the independent networks each need different IP ranges, so I set up a new mapping: And I ran the iperf server on all the interfaces on the CM4: |
|
For each of the Dramble Pis, I had to get Then I set up five iperf servers (one on each interface on the CM4), and manually triggered a bunch of |
|
Er... I think I need to use a different IP range for each of the interfaces. Ugh, my little text file mapping is going all bonkers now. As I said, tedious! |
|
So first quick result, firing off each benchmark one by one:
Total: 3.06 Gbps Then I tried just hitting three interfaces on the NIC only, and got:
Total: 2.82 Gbps Then hitting all four interfaces on the NIC:
Total: 3.07 Gbps Hitting three on the NIC and one on the Pi:
Total: 2.84 Gbps So it seems like there's an upper ceiling around 3 Gbps for total throughput you can get through a Compute Module 4's PCIe slot... and when you couple that with the onboard network interface, it seems like there must be a tiny bit more overhead (maybe the Broadcom Ethernet chip isn't quite as efficient, or maybe having the kernel have to switch between the Intel chip and the Broadcom chip results in that 8% bandwidth penalty?). I ran all of these sets of tests twice, and they were extremely close in each case. Now I'm interested to see if I'd be able to pump any more bits through this thing with a 10GbE adapter, or if 2.5 GbE over any individual interface is about as high as it'll go before we hit limits. |
So yeah, there is overhead loss somewhere (3.1 Gbps limit atm). To improve this, you might look at those cards that offload from kernel to card driver (see next paragraph). Or it might just be that the drivers are not optimised when working together (on ARM). Do you get any connection over 940-ish Mbps (that seems to be about the limit per connection), which seems pretty good under the circumstances (and on par with other platforms). I predict 6 months worth of work for actual real-world use (kernel and driver patches/updates), would have this running reliably at max speeds. But you would obviously need a reason for this real-world use case (and probably an income at the end :) - eg High Performance Dramble Cluster 5, 1xCM4 + 4xRPi4 with 4x1Gb PCIe ethernet kit (and no need for a switch). I am wondering what sort of reliability comes from sustained-use-testing (24/48 hours) particularly thermals, ie can the CM4 and host-board handle it under sustained high-load conditions, or does something slowdown or heatup too much |
Heh, don't tempt me! I've definitely considered building myself a custom Pi switch / router with this card... though I have some other much more pressing projects right now, especially after the response from my last video.
In the few hours that I was testing it today, it got slightly warm, but not hot (fan blowing over the whole rig throughout). The SATA card I'm testing got much hotter, though I keep forgetting to pull out my Seek thermal cam to take a few pictures on the front and back of the board. |
|
Another note on the build: if running the 64-bit beta of Pi OS, it seems that you might run into: And that seems to be because even if you install the headers with |
|
@geerlingguy https://www.amazon.com/QNAP-Dual-Band-Wireless-Expansion-QWA-AC2600/dp/B07DD86XW4 (atk10k driver) https://www.amazon.com/QNAP-QXG-10G1T-Single-Port-Low-Profile-Full-Height/dp/B07CW2C2J1 Or some Mellanox ConnectX-3 cards from Ebay (mlx4_core kernel module) |
|
iperf can be quite cpu limited since it is a single threaded process. You might be running into a bottleneck because of lacking cpu performance on the pi cm4. The client in iperf has to do more cpu work than the server so you are getting as good of performance as you can get in terms of your setup with the cm4 being the server. I find iperf3 to be more performant than iperf in general so you might want to try that. Also, the -Z, --zerocopy of iperf3 can allow for better performance as well (because it uses less cpu cycles) |
|
one more thing: pcie 2 at 1X is limited to 500MB/s (or 4gb/s) in each direction. might be interesting to do a role reversal (-R) on a couple of the clients to see what happens in terms of total bandwidth utilized (RX+TX). since right now all the bandwidth is in one direction |
|
@theofficialgman - Good suggestions! I have the rig running currently and I'm going to do a few more tests with some suggestions I've gotten in the comments. |
|
Aha! We might be getting closer. The CPU itself still seems to have headroom, but I noticed the process I just created a new Ansible playbook to run these tests a lot more quickly/easily so I can better observe the board while the tests are running (I'll add it to an 'extras' dir in this repository soon), and I'm now seeing that running everything at exactly the same time (I had a small lag when I wasn't using Ansible's immediate forks) gets me:
...which is still 2.81 Gbps total. In the words of the young'ns playing Among Us, that
As good an answer as any from @luciang:
|
|
Ooh... I finally successfully cross-compiled Raspberry Pi OS Linux from my Mac (inside VirtualBox) over to one of the Pi 4s. Adding a Vagrantfile and README to assist others with that process very soon... should probably also do a video on it, or talk about the process in the follow-up networking video. Edit: But it seems like it might not be working — plugging it direct into the CM4 I get DOWN still. Might have to do a little more digging, was hoping for a quick win, but it's never easy, is it? Edit 2: Hahaha, it helps if you plug in the network card, doesn't it? |
|
Latest result with MTU 9000: 3.36 Gbps |
|
Does that number includes the onboard NIC? Quite impressive since there folks pointing out it could be the bottleneck of the pcie x1 |
|
Oops, MTU was still 1500 on the Intel side. Must've reverted after a few reboots and re-plugs today. Had to run: Ran the benchmark playbook again, and what do you know?
Testing just the four interfaces on the card, I get:
Testing just three of the card's interfaces:
And testing three of the card's interfaces plus the onboard:
So, seems logical that PCIe 2.0 x1 lane is maxing out right around 3.2 Gbps. Also confirmed that with jumbo packets / MTU 9000, the irq CPU usage was hovering around 50% maximum with all 5 interfaces going full throttle. It was 99% before, causing the slowdown. And also, I can confirm the Pi Compute Module 4 can be overclocked to 2.20 GHz, just like the Pi 400. Though it needs better cooling to stay running that fast ;) 4.15 Gbps ain't too shabby on a single Pi. I remember back in my early Pi cluster days when I salivated over getting 200-300 Mbps... 😂 |
|
Posted a new video: 4+ Gbps Ethernet on the Raspberry Pi Compute Module 4 |
Using VM is always slower than run under host. |
So about this STH has this adapter reviewed (https://www.servethehome.com/syba-dual-2-5-gigabit-ethernet-adapter-review/) that could make it possible to do two 2.5Gbps ports. I don't know how much the CPU can handle but that would effectively give you two 2.5Gbps ports for routing with a cheap switch (https://www.anandtech.com/show/15916/at-last-a-25gbps-consumer-network-switch-qnap-releases-qsw11055t-5port-switch) and the original port for ISP ... In fact I actually would be interested in making a board with just that. Compute module, the ASMedia PCI switch connected to two Intel 2.5 NICs, the third gigabit port, a Micro SD slot for those without EMMC, and a barrel connector so that you can find cheap power adapters. Currently same can be achieved with that adapter and compute module board but would be great to fit all that in size of EdgeRouterX box. |
|
FYI I'm testing a 2.5GbE adapter in #40. |
Just checked that thread ... awesome thanks ... |
|
@chinmaythosar - So... I also decided it would be fun to give the 2-port 2.5 GbE NIC a try; see #46 |
|
Closing issues where testing is at least mostly complete, to keep the issue queue tidy. |
Hey Jeff! Very interesting and inspiring. Actually, this Ethernet Server Adapter allows for HW timestamping and therefore more precise values when looking at latency performance, will maybe be another use-case where this setup could be interesting. But my question is, if it would be possible for the raspberry to also handle at the same time traffic coming from the USB3.0 port? Jordi |
|
@jordibiosca - Unfortunately, when you go through USB 3.0, the bus overhead slows things down and you get a lot less bandwidth/throughput. |
|
I'm currently debating changing my current home server setup, which is just one Intel i7-7700K based PC, to a combination of a "main server" of a Pi 4 Compute 8 GB RAM, with the i7-7700K being a secondary server only used for some heavier loads (game servers), which would be in sleep most of the time and only woken up when it's actually needed. Towards this goal I would need the Pi to have 4 Ethernet ports (1 to connect to the outside internet, 2 to share the connection to the existing other PCs, and the fourth for the i7-7700K which would join the "inside network"), which is why this Intel board is interesting to me. But for this change to actually make any sense for me, what I'm mostly interested in is the power consumption of
And how much does that consumption change between idle and full load, bot for the Pi's CPU and also for the Intel Ethernet card when under heavy network load. I don't really remember seeing power consumption being featured in your Youtube videos for the Pi based systems, but that is one thing that would be interesting at least for me. If you were to do some power consumption testing, then also be aware of some (cheap?) power meters not really taking into account the power factor, which might throw off the results at least if the measured power supply does not do much or any power factor correction. (I've gotten some super funky results from some PC systems with certain power meters like the PM-300). |
|
@maruohon - I haven't done a power meter with this card installed, but typically for other devices (I tested with 2.5G and dual 2.5G), the Pi uses a maximum of 15-20W (10-15W at idle, or 5-10W without a PCIe card). Only hard drives and more intense storage-related cards seem to really push that up higher. |
…on for Intel NIC.
and others
I ordered a Intel I340-T4 PCIe x4 4-port Gigabit Network Adapter (specs, and am waiting for it to arrive. I would like to see if this adapter also requires the BAR space expansion that the GPU required (see #2), and also whether I can see all four interfaces over the PCIe x1 2.0 bus, or whether I can only see some of them.
The card itself is an x4 card, meaning it won't slot into the stock PCIe x1 slot on the Compute Module. I could hack into it with a dremel, but instead, I bought an x16 to x1 riser/adapter: PCI-e PCI Express 1X to 16X Extension Cable.
We'll see if I can get it working! I have the 1x to 16x cable already, just waiting on the GigE adapter to come in.
Edit: Here's the card plugged into the IO Board:
The text was updated successfully, but these errors were encountered: