Test Radxa Taco 5x SATA NAS with 2.5G and 1G Ethernet #268
Comments
|
Oh man, I missed out on this one hard. |
|
This is really the ultimate rpi NAS board! Can't wait to see that review :) Let's see if it's time to replace my old Intel i7 w/ 4 sata disk @ Mergerfs/Snapraid setup. |
|
The Radxa Taco board looks like an awesome NAS solution. I'm really happy they are designing it! |
|
This is VERY exiting! |
|
How would a case, for a setup with SSD, ideally look like? Greetings from the north of Germany, too! |
|
I looked at the previous issue #202, where @andyattebery mentioned one issue:
I have played with RTL8125B for some time (on x86 of course) and met the same problem. I found a solution on Ask Ubuntu which suggests kernel newer than 5.9 supports the NIC out of the box. Maybe there is more need to be done for Raspberry Pi. Another approach is to try Realtek's driver. Also, I remember Jeff once tested a 2.5GbE NIC in the Pi vs ASUSTOR video. I don't know whether this card uses RTL8125 or RTL8125B. From my understanding, the former has better compatibility than the latter. |
|
Since I have one on hand, let's get the specifics:
The RTL8125B is the same chip used in other 2.5G network cards I've tested (notably the Rosewill: #40). To get it working, you'll have to cross-compile the Pi kernel with the following option enabled in Raspberry Pi OS is not likely to add support for various NICs out of the box since it's built to work only with the hardware Raspberry Pi currently builds-in, and very minimal other hardware (I got them to add SATA support but that's pretty universal/generic—these NIC drivers are not). |
|
I was able to get the Realtek driver working without having to recompile the kernel with the module on the newest version of Raspberry Pi OS. Jeff mentions in his post about 2.5GbE he had trouble compiling it, so maybe something has changed.
|
|
@andyattebery - Good to know! Did you use 32-bit Pi OS or 64-bit? A lot has changed in the past year, it might install easily on both now. |
|
Both 32-bit and 64-bit work. On one 32-bit installation |
|
@mjeshurun @bydorfler
which do you think not important and want to replace? The idea was to build a NAS with NVMe as cache and fast WiFi streaming. |
Hi @hipboi , I think all the PCIe switches you mentioned are important. |
|
@mjeshurun - With a 4-way PCIe switch, you only get 4 'devices' (the ones listed above). Unfortunately, you can't take one device (the 5x SATA controller) and hot-wire a PCIe port on top of it—you'd need another PCIe switch (or switch to a PCIe switch chip that handles more like an 8x or 12x, and those cost more money and I would assume are larger in board space). |
|
@geerlingguy are you talking suggestions for tests? I'd be interested to know if all that PCIe switching limits throughput in the (probably very unlikely) scenario that all interfaces are exhausted simultaneously. Testing the bandwidth of the block devices on their own and all at the same time would be easy to do with some fio, the 2.5 GbE NIC could be loaded using iperf. Testing the full speed of a Wifi 6 card might be a thing that is more difficult to achieve... I don't have experience with current Raspberry Pi's and SATA or NVME disks, so I don't have any idea if the PCIe switching or the CPU itself would be the bottleneck. Looking forward to your review and availability of this board! |
|
@markwort - Don't worry, been doing a lot of tests with both PCIe switches, just NVMe, NVMe RAID, SATA RAID, 10G/dual 2.5G Ethernet, multiple 1G Ethernet, etc. — the bottleneck is always in the CPU + x1 lane, unfortunately. Maximum we'll get is 3.6 Gbps or so through the bus, if going to the CPU. What would be interesting to see is if some of the devices (network drivers are like this sometimes) can bump things up for traffic that doesn't have to route through the CPU itself. But I'm planning on putting the pedal to the metal. The Penta board I have only has M-key, so I'll have to decide whether to put in an NVMe SSD, or an A+E adapter to mount a WiFi 6 card on it. |
Thanks for the explanation 🙏 |
|
@geerlingguy thanks for the quick answer! I didn't think that you might have dealt with the same PCIe switches before, but now I remember a remark about these issues from one of your videos.
You're probably thinking of RDMA, which can be supported by something like NFS or SMB ("SMB-Direct"), but that usually requires special NICs, and I see you've stumbled across issues with that in the past already. Have you tried io_uring for any benchmarks? In your Pi Dramble repo I only found calls to fio using libaio. I wouldn't know what to use Wifi on a NAS like that for. To get adequate speeds it can't be any more than a short cable run away from the access point. |
|
@markwort - I imagine one use case would be if you only have WiFi available in the spot where you want your NAS—with WiFi 6 and a really good router, and not too much distance (20' max probably), you can achieve gigabit speeds, which is good enough for many. But I go wired 99% of the time when I care about access speed/performance. Even the best wireless devices are going to run into issues where wired performance will be more stable. I haven't done anything with io_uring yet. |
|
@geerlingguy |
|
@mjeshurun - Sure, but it's limited to a maximum of around 70 Mbps (90 if you're lucky and have a really, really clear signal). That's good enough for some use cases :) |
|
First boot—used this 12V 8A power supply with 2.5mm barrel plug, booted into Pi OS 64-bit lite, on a 4 GB CM4 Lite with a Sandisk Extreme 32GB microSD card. A few initial observations:
Click to expand outputAfter Click to expand outputSo after upgrading to the latest 64-bit OS release, SATA seems to get the AHCI driver, and the drive slots should hopefully work. I noticed the board gets a bit toasty. I'll have to measure temps and see if a fan is required for both the topside and underside (or just generally good ventilation—most likely 'yes'). Don't have time to test out NVMe, SATA drives, or the 2.5G Ethernet yet. Looks like at least for the latter, I'll need to install the driver (or recompile the kernel, heh... hopefully the driver just installs gracefully). |
|
Getting Realtek 2.5G NIC working, attempt number 1:
Looks like the link is up! Next comment: speed test. Will have to re-plumb my wires a bit to make sure I'm testing over 2.5G and not a 1G switch. I should really get another 10G switch set up in my office so I don't have to go back to my rack for this stuff...
|
|
Yeah, got tired of not being able to reach one of my servers (an Odroid H2+ with the same chip) after a kernel update, so I decided to do the |
|
Testing on 2.5 Gbps—one thing I noticed right away, and this may be a bug: neither the amber connection light nor the green activity light on the 2.5 Gbps port light up when I connect it to my 10G switch. Connected to my 1G switch, I get a blinking amber light, but no green light. So maybe some wires are crossed with the port LEDs to the NIC? It might support multi-mode LEDs for 1/2.5G indication and maybe the board schematic is crossing some connections somewhere. Anyways, a trip to Click to expand full outputSome performance testing: Click to expand full outputInterestingly, I'm hitting the same issue I did a few months back, where throughput from the Taco to the Mac is a full 2.35 Gbps, but the opposite direction is around 300 Mbps and fluctuates a lot. I've seen this problem before, where Ethernet was slower only in one direction on one device, and it turned out the issue was with a FLYPROFiber SFP-10G-T-30M transceiver. I'm going to check if that's the case here. Edit: lol, yep, that was the model on that port. Off to order more transceivers! |
|
Now comes the fun part... let's see if a Sabrent Rocket 8TB NMVe SSD works in the M.2 slot...
It fits, even though it's a double-sided card (luckily the M.2 port is nice and tall).
The M.2 slot uses an M2.5 screw (and my board didn't come with one). That seemed a little odd since most of my M.2 devices/slots seem to come with M2-size screws (M2 meaning the measurement of the ISO metric screw thread, not 'screw meant for M.2'). I was lucky I had a pack of said screws from my Pi PoE HATs!
I formatted and mounted the drive to Click to expand full outputRunning my disk-benchmark.sh: Results:
|
|
And next... 5x Samsung 870 QVO 8TB SATA SSDs (and no, that's not a typo... lol):
I noticed that all the drive status LEDs are not only glaringly bright blue (aaah!!), they are also all lined up directly under the middle slot... so good luck seeing them. Not sure if the official case will use some very creative light pipes to show drive status, or if you'll just operate the drives without blinkenlights.
I also noticed the preinstalled headers on the board's edges are basically mm away from the drives once installed. It would be nice to have a smidge more clearance—and 3.5" drives will definitely not fit on this board with those headers in place. I formatted one drive ( I ran my disk-benchmark.sh on one drive: Results:
So in all respects, slightly slower than NVMe, but that's to be expected. Way more IOPS on the NVMe drive too. I guess that makes some sense, considering that thing cost twice as much! |
|
So this is cool... apparently support was added recently, so if you want an even quicker fix, you can run |
|
I can't get my board to detect my microSD/ SATA SSD. Any ideas? |
|
@iandk - Do you mean you're trying to boot with a USB hard drive, and not microSD? Or you're trying to boot of a microSD card with Pi OS on it? It's not clear from your comment what you're trying to do—and are you doing this on a Taco board or some other CM4 board? |
|
I’ve tried booting off a microSD as well as a SATA SSD which I connected directly to the SATA slots. |
|
Is the CM4 you're using a Lite version or one with eMMC installed? If using an eMMC compute module, it will never allow any kind of microSD card boot. SATA boot support is not implemented on the Raspberry Pi currently; see raspberrypi/firmware#1653 |
|
oh, I thought I had the lite version without emmc but just checked it indeed has emmc. That means I probably have to get another board which allows me to flash PiOS the emmc flash? |
|
@iandk - Yeah, it looks like (afaict so far) the Taco doesn't have the ability to boot the Pi into usbboot/rpiboot and allow flashing eMMC compute modules. |
|
yeah, this is the issue we did not consider for the first version. Now for the new hw revision, we added a small button for the usbboot. |
|
Video is live! https://www.youtube.com/watch?v=G_px298IF2k |
Thx. Where did you buy the Radxa Taco board and for how much? |
|
@vebmaster - The one I used in the video was provided by Radxa; it should hopefully be available soon on their website. |
This comment has been minimized.
This comment has been minimized.
|
I can confirm that the Samba performance dropped with the kernel upgrade from 5.4 to 5.10. |
|
@geerlingguy apologies for asking in the wrong issue, then deleting the comment here while you were already answering. AFAIK neither And as already mentioned block sizes matter in a world where network stacks implement auto tuning of settings. You would either need a network sniffer or a tool that has control over the block size to get what you test (Helios Lantest for example). BTW: I'm in the wrong issue since my concerns are not about 'RPi samba performance degradation' but about the measured differences between the RTD1296 thing and the Taco. Without checking details like the negotiated block size (or smbd tunables) it's hard to interpret the numbers or attribute them to 'hardware' while they're in reality just different settings. |
|
@ThomasKaiser heh, we just keep going back and forth! I've moved my comment body over to #162, let's keep the discussion there since it has more of the detailed benchmarking in that issue :) |
|
@iandk just glanced through the referenced rpi issue. The drop in SMB write performance after some seconds looks like 'buffer full' situations. In case you can still reproduce it this way with more recent kernels than 5.4 in combination with Samba I would run on the RPi in a terminal
|
|
I think the current state is, that it’s always stuck to around 70-80mb/s Write, instead of the 112mb/s stable with kernel 5.4 |
|
Well, I fired up my RPi 4B (early model with just 1GB) to check for this (5.10 kernel). No Windows involved but with macOS 12 as SMB client I get ~100 MB/s in both directions which is perfectly fine to me... |
|
You should remove all of those cache settings, no send/ receive file, no caching. |
Why exactly should I do that? |
|
The Pi 4 was (with kernel 5.4) able to saturate the 1Gbit/s port without a problem, you don't need a cache. |
|
Additionally lots of users have exactly the same issue, the current performance is quite a bit slower than a few months ago. I stillt think that this is related to the kernel update 5.4 -> 5.10 |
Yeah, I heard this. Jeff also said the same. And everybody refuses active benchmarking and looking into settings. You said above
I gave it a try and have to report: nope, it's not. I truly believe that you had more before and that 70-80 MB/s are pathetic especially given how Windows Explorer accelerates SMB transfers since Windows 7. But what's the point in using bad settings and not actively benchmarking stuff to get a clue what has changed? :) As for "remove all of those cache settings, no send/ receive file, no caching" and "you don't need a cache. This will just result in performance drops over time, as you transfer larger files." I don't understand what you mean (but it doesn't matter that much anyway since I'm fine with the performance I get with '5.10' or as I would call it 'appropriate settings'). There's the |
|
I did lot's of testing and benchmarking a few months ago. But again, there's no reason to use a cache, as it was always running perfectly fine and hitting the Gbit limits without one. |
Have you had a look with e.g. |
|
I have a Radxa Taco and I would like to put it in a case. Since no official case exists yet, my idea is to buy a small PC case made for mATX or ITX mobo NAS, and use extender cables to connect the drives from the bays to the Radxa Taco. Do the two holes in the Radxa Taco board line up so that I can screw the board in place inside of a case made for mATX or ITX? |
|
@geerlingguy Thanks for the videos about the Radxa Taco! I have mine fitted with a CM4 and it is running very stable so far, however I have had problems with the screws supplied for the CPU cooling being too short and I can't get the RTC to work. Has anyone got the RTC chip to work? |
|
I'm late to this discussion but will we ever be able to boot from M.2 NMVe on the Radxa Taco? |
and others
See original issue: #202
I have a Taco (well, the Penta main board that goes inside) and would like to do some testing on it; run some benchmarks, test compiling ZFS, etc.
Things to test:
The text was updated successfully, but these errors were encountered: