Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs

[geerlingguy]

Pineberry Pi is a joint venture between Michał Gapiński and Mirosław Folejewski — both of whom have a history of making some pretty cool Pi products.

They just announced their "HatDrive!" Line of HATs (one for the top, one for the bottom) for the Raspberry Pi 5.

I have one of each and will be testing it, along with their impedance-controlled FFCs (they make two lengths):

• HatDrive! Top for 2230/2242 M.2 E-key
• HatDrive! Bottom for 2230/2242/2280 M.2 E-key

I should also redirect the page https://pipci.jeffgeerling.com/hats/mirko-hat5m1-hat.html to these new HAT pages. See #550

[geerlingguy]

Redirect added, and pages are up here:

• https://pipci.jeffgeerling.com/hats/pineberry-pi-hatdrive-top.html
• https://pipci.jeffgeerling.com/hats/pineberry-pi-hatdrive-bottom.html

[Bouni]

Do you know if any RPi5 case that can be used with the HatDrive! Bottom?
Pineberry has a picture on their website but it shows the HatDrive! Top variant I think.

[zytegalaxy]

I can't seem to find exact board outline dimensions for the bottom board. I was wondering if it would fit my enclosure...Is that published somewhere? I order a set btw and can verify that once they arrive.

[geerlingguy]

A few quite notes testing the Top Hat:

• I noticed my Pi 5 bootloader was from September (see below). I ran sudo rpi-eeprom-update -d -a to upgrade to the latest bootloader and rebooted. Apparently older bootloaders could have issues booting off NVMe...
• I didn't get the full hardware kit, just the PCB, so I had to improvise some spacers. It seems like you don't need GPIO at all if you just want to use NVMe. I think GPIO connections on the Top Hat are just for signaling (some sort of power measurement?)
• You still have to enable PCIe like I document here: NVMe SSD boot with the Raspberry Pi 5

pi@pihat:~ $ sudo rpi-eeprom-update
*** UPDATE AVAILABLE ***
BOOTLOADER: update available
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    LATEST: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
            Use raspi-config to change the release.

pi@pihat:~ $ sudo rpi-eeprom-update -d -a
*** INSTALLING EEPROM UPDATES ***

BOOTLOADER: update available
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    LATEST: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
            Use raspi-config to change the release.
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    UPDATE: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
    BOOTFS: /boot/firmware
Using recovery.bin for EEPROM update

EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".

pi@pihat:~ $ sudo reboot

[geerlingguy]

Another note: I could not get the Pi to boot off NVMe if I used rpi-clone. I was able to do that on an earlier revision of the Pi 5 bootloader with the Pi prototype board... but a LOT has changed since then. It would try to boot, then drop me into a BusyBox initramfs shell. Very strange.

It works best if you flash Pi OS directly to the NVMe SSD, using the Pi with a USB adapter, Pi Imager direct, or on another computer.

[geerlingguy]

[geerlingguy]

A few questions about the board that I have now that I've put in a few hours...

• How is the 5V power input on the 'Bottom' version of the board to be used? Will some sort of adapter cable be included?
• How do you access the power monitoring on the 'Top' version?
• Will the full production kit include a GPIO extender or some other means to get GPIO pins to extend into or through the 'Top' version?
• Are there any cases that are compatible with either version specifically? (The Pi 5 Case is a bit tight, and airflow is a concern on it).

[jack-ma]

What is the pitch of the FPC cable connecting the PCIe signal? It looks like 0.5mm?

[mikegapinski]

1. We'll stock it but I have not found a device that needs it yet. We are playing it safe in case someone needs it, that is why we have power monitoring in the first place.
2. If you want to access the current monitoring chip via I2C (HatDrive! TOP only for now!!!):

dtparam=i2c_vc=on

You should see the INA with i2cdetect:

mgapinski@coralpi:/dev $ i2cdetect 0
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x08-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: 40 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: 50 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --    

3. We have ordered GPIO extenders for all of our TOP hats in the 1st batch.
4. I am working on adapting a acrylic design (2 elements on top/bottom, sides exposed, it will also work with other boards we are developing now) , it won't be ready in 2 weeks but I'll drop you something once we are done. We are not dealing with enough volume to do plastic injection but a lot can change in a month.

[Cristianistrate]

The official active cooler can be used with the TOP Hat ?
Doe's significatively impact the cooling of the fan ?

[mikegapinski]

Yes, and it does not seem like it is having a huge impact. It did not throttle when building the kernel

[ThomasKaiser]

@geerlingguy as for your disk benchmark mentioned on X.

The 777 MB/s do not represent the device's capabilities since a) hdparm uses a rather small block size when benchmarking (it's a tool from last century originally meant to deal with spinning rust) and b) you already showed that ~900 MB/s are possible with Gen3 speeds on RPi 5.

So either testing methodology or SSD in question is to blame for these low numbers (most probably both) :)

As for the 'testing methodology'... PiBenchmarks' Storage.sh relies on hdparm (limited block size, in the past it was 128K, with more modern versions it looks like 1M) or dd with just 4K blocksize. Quick test with a SATA SSD should show the problem:

hdparm -Tt --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   754 MB in  2.00 seconds = 376.91 MB/sec
 Timing O_DIRECT disk reads: 1318 MB in  3.00 seconds = 439.28 MB/sec

dd if=/dev/zero of=test bs=4k count=130k conv=fsync
133120+0 records in
133120+0 records out
545259520 bytes (545 MB, 520 MiB) copied, 1.63406 s, 334 MB/s

Storage.sh tells us about 334 MB/s sequential write speeds and 439.28 MB/s sequential reads. While when testing with e.g. 16M blocksize we're at 490/520 MB/s in reality:

iozone -e -I -a -s 100M -r 1024k -r 16384k -i 0 -i 1

          kB  reclen    write  rewrite    read    reread
      102400    1024   107457   436871   441420   439296                                                                                  
      102400   16384   486231   493974   519500   521398

No idea how you measured your 'nearly 900 MB/sec' on the Pi 5... but if it was only with Storage.sh I would revisit this :)

[geerlingguy]

A few test results from Pineberry, of various SSDs:

Model	PiBenchmarks.com score
Sabrent 500GB Rocket PCIe 4.0 2280	47002
Crucial P3 Plus 1TB 2280	40118
PNY CS1030 1TB 2280	42203
Lexar NM620 512GB Gen 3 NVMe 2280	44542
Western Digital SN530 512GB	45735
Samsung 970 Evo Plus 1TB 2280	27960

Edit: Also noticed they have a ton more up on PiBenchmarks.com: https://pibenchmarks.com/user/pineberrypi/

[geerlingguy]

No idea how you measured your 'nearly 900 MB/sec' on the Pi 5... but if it was only with Storage.sh I would revisit this :)

@ThomasKaiser - I only use PiBenchmarks.com script because it's a fun way of gamifying the testing—for my actual reporting/testing I use my own script: https://github.com/geerlingguy/pi-cluster/blob/master/benchmarks/disk-benchmark.sh

It runs fio at 1024k block size 4 GB test, then iozone 1024k and 4k random read and write tests. It's quick but effective for most cases. I am looking at expanding the benchmarks a little now that so many devices are reaching into "GB/sec" range, and trying to go for more gigabytes could help squash any additional caching that's happening and throwing off the results.

[geerlingguy]

The official active cooler can be used with the TOP Hat ?

Yep, see the video I just posted! https://www.youtube.com/watch?v=EXWu4SUsaY8

[ThomasKaiser]

It runs fio at 1024k block size 4 GB test

Well, just 1024k is IMO nothing that represents today's sequential access patterns any more. Even network protocols like AFP or SMB dynamically tune their internal blocksizes to more than this in a LAN environment for over a decade now. See my SSD example above: with 1M it's 430/440 MB/s write/read while it needs 16M to get closer to the real sequential transfer rates: 490/520 MB/s

And a write test 4GB in size will result with el cheapo SSDs in an amount of data written to fast cache and once the cache capacity is exceeded a drastic drop in write performance can be observed which – with a 4GB test size – will result in a MB/s number neither representing the 'cache write performance' nor the 'sustained write performance once cache is full' but a crude mixture.

Firing up fio with just 1GB data size but 4 times in a row generates both numbers and also an idea of how large the cache area of a specific SSD approximately is. Needless to say that once the cache area is exceeded the later iozone tests will also suffer from this and generate almost meaningless numbers (at least 4k random I/O) :)

[AEW2015]

I want to try my FPGA load tester and see what the upper limit is. I can get about 25 Watts on the compute blade with the CM4. It was important to cool the CM4 to supply stable power otherwise the CM4 got to 80C very fast.

[lijoseph]

Does the pineberry hat resolve wifi issue with the prototype raspberry pi m.2 nvme hat?

[mikegapinski]

Jeff used the old eeprom firmware with the OG prototype HAT. It had tons of issues

[balupton]

Does the bottom nvme board fit within the raspi case, with the cooler on the top?

[FriendlyNGeeks]

Now you didn,t hear this from me, a birdie told me if you buy a spark fun qicc shim and bypass the voltage regulator you can i2c 5volts from pi to any accessories you want(pisugar, m.2[bottom] board, etc..)

[arekm]

Thanks. M2.5x6mm standoffs worked with original "top hat" cable but barely (for example with sdcard inserted cable will be too short). Replacing cable with longer variant will be better, for nvme cooling, too.

[mikegapinski]

The short cable is not really meant for BOT mounting; people went crazy with the FPC bundle recently and it's out of stock. We should have them shortly, together with complete accessory kits (standoffs, new m.2 mount etc)

[arekm]

What's the standard length of "bottom" hat standoffs? (trying to fit this my small "monster" into unmodified cases designed for pi5+bottom hat).

Edit: Remixed one nice case. Works great with top hat (mounted at bottom) ! https://www.printables.com/model/777582-pi-hat-drive-top-variant-of-great-rpi5-enclosure

[mikegapinski]

We use M2.5 at 6mm, you can use the step files on our GitHub as reference to see if it'll fit

[kitopopo]

Good evening dear friends,

I have discovered how to boot my nvme in gen 3 always successfully. It seems incredible but the tests work for me and in my particular case. I have performed 30 shutdowns and 30 startups and a 100% satisfactory result. I don't really understand the reason but the problem seems to be with the FFC cable. Simply putting your finger on top of the FFC cable without pressing and just touching. Just above the numbers 2349 (just the touch of my finger is enough for me). After putting my finger on the hose with the Raspberry off after turning it on, I get 100% success on all startups. It seems like an impedance problem since from what I see it is a special type of cable that works with controlled impedance, of which I am completely unaware of how this type of cable works, but perhaps when you put your finger on the cable, the impedance is altered, although that It's already slipping out of my hands. (It is important to mention that performing the same test and touching the hose with a piece of plastic or some insulating material, the nvme boot does not work in most cases, so I rule out possible no good contact phases in the 2 connectors and the test reaffirms that it is of a variation of impedance)
As you mentioned, perhaps the firmware of most nvms also has something to do with it, is possible but in my opinion, if on desktop computers or laptops these nvmes work perfectly and in this project, my opinion is that there could be another added cause.
I encourage all of you who do not get a correct boot in gen3 to try it and I also encourage the development department to carry out tests, use other types of cables or cables without impedance control , using a cable 25mm, 50mm or 100mm in lenght. Currently I only have the Pineberri Pi cable, so I can carry out more tests but I hope and wish that they can find a solution and add it to some v2 revision of the board or FFC cable.

Note:

In my particular case, if the nvme boot fails 2 times in a row, it is necessary to connect the nvme to a computer using a dock or USB adapter to change to gen 2 in config.txt. In gen 2 it always starts correctly, once it starts, we have to change back to gen 3 and turn off the raspberry. Put your finger back on the cable every time you start (it's a shame we can't restart the raspberry remotely due to is impossible put my finger) I repeat that in my particular case it works with nvm lexar 620.

I hope it works for you and you will comment on the tests. I hope I have helped and I just wanted to share my experience . A cordial greeting and thank you all.

Our ribbon cables are thick and you might have separated the ground plane when you tested the board repeatedly by bending them back and forth.

Please drop me an email with your order number - I will ship you some FPC bundles and an engineering sample of a different cable in the same length. Curious about your results - I don't know what revision your HatDrive is from but we are on a third one at the moment.

Dear friend @mikegapinski , I have been doing tests these weeks and I get perfect boot with flat cables without impedance control less than 4cm, but with my v1 flat cables supplied with the pineberri board (impedance control) I can never start unless I touch it with my finger (just like I mentioned before) but I am very happy with my pineberri botton board and gen 3 work.
I would like to take this opportunity to comment that in the raspberry pi 5 log an error occurs, PCIe Bus Error: severity=Corrected, which is solved by adding pcie_aspm=off to cmdline.txt. This parameter solves almost 99% of the errors in gen 3 but from time to time a sporadic error appears.
Is there a way to test the new v2 or v3 cables or those that you told me about to test if the errors disappear completely? When I bought the boards I did not place the order directly to Pineberrybut place it with a Spanish Pineberri supplier. how could we do? Can I buy a cable in pineberry and can you send me the rest for testing? do you have any other ideas for that i can receive this cables test? thanks in advance, receive a cordial greeting

[bmoss706]

Pimoroni Base + RPi5. Bookworm has ASPM turned on.

Netac NV2000 Maxio MAP1202 controller

• $ sudo lspci -vvv shows that ASPM is not supported.
• Gen 3 ASPM on: System Journals show AER error counts as high as 46069 causing kernel panics.
• Gen 3 ASPM off at command line. No AER errors except when using the Pimoroni microSD extension which interferes with the FFC.
• ASPM off does not suppress error reporting. It shuts down the system causing the errors. This is a firmware issue.
• Pibenchmarks score average around 50000.

Lexar NM790 Maxio MAP1602 controller.

• Max power 3.7W, within the RPi5 PCIe FFC 1A, 5W limitation.
• Supports ASPM mode L1.
• Gen 3 no AER errors except when using the Pimoroni microSD extension.
• Pibenchmarks score average around 58500.

Recommendation: If you have issues with your NVMe drive and it does not support ASPM, turn it off.

[kitopopo]

Pimoroni Base + RPi5. Bookworm has ASPM turned on.

Netac NV2000 Maxio MAP1202 controller

• $ sudo lspci -vvv shows that ASPM is not supported.
• Gen 3 ASPM on: System Journals show AER error counts as high as 46069 causing kernel panics.
• Gen 3 ASPM off at command line. No AER errors except when using the Pimoroni microSD extension which interferes with the FFC.
• ASPM off does not suppress error reporting. It shuts down the system causing the errors. This is a firmware issue.
• Pibenchmarks score average around 50000.

Lexar NM790 Maxio MAP1602 controller.

• Max power 3.7W, within the RPi5 PCIe FFC 1A, 5W limitation.
• Supports ASPM mode L1.
• Gen 3 no AER errors except when using the Pimoroni microSD extension.
• Pibenchmarks score average around 58500.

Recommendation: If you have issues with your NVMe drive and it does not support ASPM, turn it off.

Thanks for your anwer. I have added pcie_aspm=off to cmdline.txt but some times the error appear. Plugin off an plugin off the raspberry several times the pcie error corrected appesr in my raspberry pi 5. The "pcie_aspm=off" seems to have no effect for me.
I suspect my fpc cable without shielding and without impedance control since with the original v1 from pineberry the nvm never doesn't boot correctly.

[kitopopo]

Hi dear friends,

For those who are interested, I have discovered a way how we can completely eliminate the error "PCIe Bus Error: severity=Corrected" in the raspberry pi 5 log without having to add "pcie_aspm=off" in cmdline.txt. So far in 70 boots I have achieved 100% success with my hardware. I'm doing more shutdowns and restarts before publishing the temporary solution that works for me. I will contact here again soon if I get 100% success, I need to do more tests. Thank you. Best regards

[mikegapinski]

I'd suggest moving the discussing about that Lexar drive elsewhere since it is not strictly related to the board itself.

I can share a small update from us about drive compatibility:

• All HatDrive variants ordered after 11.03 include with HAT+ power management (HatDrive! Top and Top Lite boards are launching now, they also have an updated footprint)
• We updated the HatDrive! Bottom in January but have not mentioned it since there were 3 revisions in circulation. V2 had SUSCLK that we determined was not necessary moving forward. V3 has HAT+ power management and we ditched the HAT ID / INA power monitoring IC since I2C on the FPC connector did not make it to the final spec
• Upcoming HatDrive! Bottom revision will ditch the 5V power plug, we honestly could not find a use case for it - both the spec and our testing show that the FPC connector can deliver enough power for everything we throw at it
• Current bootloader does not require PCIE_PROBE and changing the boot order, it will boot from NVMe if something is connected. Even V0 prototypes we made in October work, since Mirko figured out how auto detection could work. For a second we thought EEPROM would need to be connected to the GPIO header but it is not the case.
• Power management control on the bootloader level and recent bootloader updates resolved issues that we had with WD Drives. Your mileage may vary, but we are not getting any reports from customers that purchase the boards at the moment. Most of people don't read the docs and ignore the warnings we gave before. SN530 is very popular since it is cheap on Amazon (OEM variant)

Some other news that could mean something for the future: the embedded version of the Raspberry Pi Imager that boots up if no bootable media is present now recognizes and flashes NVMe drives connected behind a PCIe switch. This could mean booting from them may start working at some point @geerlingguy. It seems like RPI has solved most of the problems with the bootloader now and it aligns perfectly with their release schedule for the official M.2 HAT :)

[geerlingguy]

@mikegapinski good news indeed!

[kitopopo]

Sorry @mikegapinski , but it is not a problem with lexar unit or the nvme firmwares, it is a problem with the v1 flat cable. Many people with both Pineberry Pi and the competition have a PCIe Corrected error with other nvme brands. My nvme works correctly and without errors with another flat cable and without errors.
But if no one is interested in knowing the solution, I will no longer continue explaining the solution. kind regards

[mightymos]

I have a Kioxia KBG40ZNS256G NVMe drive that boots on 2/16/2024 bootloader with the Hatdrive Top (ordered 2/5/2024).
However, I am also getting PCIe corrected errors.
I was having freezes previously but this was on alternative OSes so I am not sure if that is a coincidence.

I ordered a 3cm cable from Hangzhou WildChip Tech Co., Ltd. Store to try out which is linked in the thread.
@kitopopo Is that a cable you've tried out with your drive(s)?

[mikegapinski]

I’ve seen some reports about this drive being inconsistent on the NVMe Base in the past. Try Gen 2 or disabling aspm. Switching cables to something without impedance control is not a good idea Wysyłane z aplikacji Outlook dla systemu iOS<https://aka.ms/o0ukef>

…

________________________________ Od: Jonathan Armstrong ***@***.***> Wysłane: Sunday, March 10, 2024 4:29:08 AM Do: geerlingguy/raspberry-pi-pcie-devices ***@***.***> DW: Michał Gapiński ***@***.***>; Mention ***@***.***> Temat: Re: [geerlingguy/raspberry-pi-pcie-devices] Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs (Issue #559) I have a Kioxia KBG40ZNS256G NVMe drive that boots on 2/16/2024 bootloader with the Hatdrive Top (ordered 2/5/2024). However, I am also getting PCIe corrected errors. I was having freezes previously but this was on alternative OSes so I am not sure if that is a coincidence. I ordered a 3cm cable from Hangzhou WildChip Tech Co., Ltd. Store to try out which is linked in the thread. @kitopopo<https://github.com/kitopopo> Is that a cable you've tried out with your drive(s)? — Reply to this email directly, view it on GitHub<#559 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAHDCBZV4EC4CTZFR7IL6ATYXPHQJAVCNFSM6AAAAAA7NGHADKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGA2TKOJUGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kitopopo]

Dear @mightymos ,

I have tried those 3cm cables from that store, with those cables I get a correct boot with my nvme but I get too many pcie severity:corrected errors. The title of those cables gave me the final idea to eliminate the pcie corrected errors mentioned above. If you look closely it says "shielding FPC cable".

I use normal white cables without impedance control and 40mm, perfect start but with errors in gen 3.

So a week ago it occurred to me to shield my 40mm cable with aluminum foil or silver paper, the kitchen paper of the entire life. After giving a few layers of paper to the cable, I inserted a cable in the middle to connect gnd to the gpio connectors of the raspberry, and to cover this cable I gave it more layers of paper which I finally wrapped with tape so that it does not move.

Attention, you have to be careful because if the aluminum foil is not perfectly fixed, it can move and short circuit the cable pins on both sides. If you do the test, do it at your own risk.

After this procedure I get 0 PCI corrected errors working in gen3 for more than a week, testing every day and powering off the raspberry pi 5 more of 200 times, rebooting... these errors have disappeared. You can also eliminate the pcie_aspm=off line from cmdline.txt since in my case it does not work since I kept getting errors.

Note: My experiment only works with 40mm cables, I have tested 50mm but again errors. My opinion is that the problem has always been in the FPC cable.

Now I just have to buy new cables to pineberry pi revisions v2 or v3 and verify these errors with new cables since my experiment is provisional. For this reason, several messages ago, I asked @mikegapinski for several test cables but he ignored my message.

I don't reject @mikegapinski advice and it is most likely that it is convenient to use cables with impedance control, they are engineers and it is their project, I simply explain how in my particular case I have managed to eliminate all the PCI errors and start my NVME correctly .
If someone tries it, we will discuss the results. It is difficult to get normal 40mm fpc cables, I think there is only one seller in the Asian store.

Kind regards

[mikegapinski]

@kitopopo

The cables we ship with the FPC bundle on the website are the same that you got with the board. We have a few thousand of them. Every HatDrive we ever shipped to customers has the same FPC. There are no revisions of those other than a few dozen prototypes from 2023/10 that never made it out of our lab.

The 100mm one currently comes from a new production batch (2024). We also have 60mm and 80mm for the HatBrick Commander and they could be potentially available separately soon.

60mm cable is currently only shipping with our Ethernet boards.

[kitopopo]

Well then I will have to continue using my experiment to achieve stability en Gen3. Thank you

[mightymos]

Thanks @mikegapinski I tried setting Gen 2 and disabling ASPM but still getting AER corrected errors.
I also just saw my first uncorrected (fatal) error so that is fun lol.

Apparently the aliexpress cable is claimed to be impedance controlled, so at worst I might have wasted some money.
I can stick with the stock cable for now.

I still seem to require PCIE_PROBE to boot with 2/16/2024 firmware.
Out of curiosity, since dtparam=nvme or similar is no longer explicitly required in config.txt where is it set and at what speed?
How do I know that if it is set in config.txt that the setting there overrides whatever is apparently now default?

Thanks for the help. If there is anything else I can report let me know.

[mightymos]

Regarding the Kioxia KBG40ZNS drives I was able to attempt updating firmware with this tool (Windows only):
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=tknm3

However I apparently have the latest firmware already.

PHYSICALDRIVE2: Firmware download and activation command completed successfully.
[Old-revision:10410106] [New-revision:10410106]  (Same revision)
>>> Command success but firmware versions are same...
### Return code: 4000 ###

To summarize to /boot/firmware/config.txt added:

dtparam=pciex1_gen=2

To /boot/firmware/cmdline.txt added:

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

[vin100ross]

I just wanted to add my 2 cents. Note that my HAT is not from Pineberry, so this might or might not be relevant. However, to add some weight to @kitopopo's experiments, I also had several corrected AER errors with my setup. Disabling bluetooth/wifi on my Pi5 reduced them by more than 90% (the remaining errors might be from a nearby router). So radio interference and FPC shielding are definitely things worth looking at. The Aliexpress cable linked previously is also advertised as "shielded", and I have one arriving in a couple of days, so I'm curious to see how it goes.

[kitopopo]

I just wanted to add my 2 cents. Note that my HAT is not from Pineberry, so this might or might not be relevant. However, to add some weight to @kitopopo's experiments, I also had several corrected AER errors with my setup. Disabling bluetooth/wifi on my Pi5 reduced them by more than 90% (the remaining errors might be from a nearby router). So radio interference and FPC shielding are definitely things worth looking at. The Aliexpress cable linked previously is also advertised as "shielded", and I have one arriving in a couple of days, so I'm curious to see how it goes.

Dear @vin100ross , thanks for your answer.

I also have the router nearby, and by deactivating Bluetooth and Wi-Fi the errors improved. I'm sorry to tell you that in my case the 3cm ffc hose from the Asian store seems not to be very armored since I still have PCIe corrected errors.

I am preparing a new shield that I have thought of, at the moment it works with a 5cm flat cable and without errors, I hope my ideas will help you if you decide to prepare something similar. I hope to upload the photos today.

_For non-believers in physics and electricity here I leave the following:

The Faraday cage is a conductive structure that protects its interior from external electric fields. Named after the physicist Michael Faraday, this cage works by redistributing external electric charges along its surface, neutralizing electric fields inside.

Key features of a Faraday cage:

Conductive material: It must be made of a conductive material, such as metal. Galvanized steel is commonly used.

No large openings: The openings in the cage should be smaller than the wavelength of the radiation to be blocked.

Ground connection: The cage must be grounded to allow the discharge of accumulated electric charges.

Faraday cages are used in various applications, from laboratories to protect sensitive equipment to building construction for shielding against electromagnetic radiation. The main idea is to create an electromagnetically isolated environment inside the conductive structure._

Best regards

[mikegapinski]

@kitopopo @vin100ross Hmm. This is now something we can work with. In a decently isolated environment it is easy to create a test that is easy to reproduce. You can fire up a hotspot with hostapd on either 2.4ghz or 5ghz and connect to it from a different device. After that you can do large transfers and check the logs with a sensitive M.2 device (those MAXIO controllers are perfect).

I think you might have found a valid explanation why Gen 3 is not fully certified by RPI. The Wi-Fi antenna is located next to the FPC connector and using a impedance controlled FPC in a shape that is covering it directly (Pimoroni NVMe base?) could make things even worse.

We'll do a brainstorm and figure out if there is a way for us to create an even better FPC cable since it is just a flexible printed circuit.

[kitopopo]

Good afternoon,

This is another type of shielding that works better than the previous one since I can eliminate errors with gen3 using a simple 5cm fpc cable
I have used copper tape or desoldering tape that is thicker and better than aluminum foil.

It is something provisional, I hope that @mikegapinski and his team manage to make a new FPC cable that solves these errors for everyone. Many of us will be interested in buying this new cable from Pineberry Pi.

I can't help more, best regards

[mikegapinski]

So... The official RPI HAT FPC cable is also not shielded, just impedance-controlled like ours. https://twitter.com/pineberrypi/status/1767244283025711559?s=61&t=Lh0E7mQyw_B9UkJjFAZW_Q

This makes me even more excited about trying to prototype this approach. It'll be expensive and the only thing we could try is Copper Layer Shields, but that would make the cable very stiff. Other more accessible shielding approaches would mean no impedance control and this is a no-no. You can read more here: https://blog.epectec.com/emi-and-rf-shielding-options-for-flexible-circuits

It might not be feasible for the HatDrive line but it could make more sense with something like the HatBRICK! Commander or uPCIty. Since we are already selling cables and they are very, very popular there could be a way to make a test batch at some point.

@geerlingguy another thing to play with that could be potentially improved, we didn't get any issues with 100mm on uPCIty but our office is in a basement and we usually use ethernet.

[vin100ross]

A quick Google shows wifi/Bluetooth interference with NVME on the Pi5 creeping up once in a while, so we're not the only ones it seems. I for one would buy into any solution you guys can come up with, even if it means a stiffer/thicker cable.

Anyway with aspm=off, and Bluetooth/wifi off, I went from a corrected error every minute on average to one every two days or so... Not perfect but manageable.

[kitopopo]

@vin100ross , In my case sometimes the errors remained stable for quite a long time.
To speed up and reproduce the problem, I restarted my raspberry ,sometimes even restarting is not enough and it is necessary to turn off the power. If you look at the log after each boot with "journalctl -n 3000" you will see that they start to appear every 10 seconds, the way to solve it is to restart again and it is stable again with sporadic errors. The key is to shut down or restart. I think that at boot time is when the error occurs and there is no way to stop it. Greetings

[kitopopo]

Interesting and complicated armor blog, I hope you find a solution. I also prefer the cable to be a little more rigid but without errors. Thanks in advance

[matty67]

I use the HatDrive top.
When I activate the option POWER_OFF_ON_HALT=1 and shutdown the raspberry, it still needs 0,8Watt and the ACT Led ist blinking 1 sec on and 1 sec off.
Is there another option to poweroff the HatDrive completely?

[mikegapinski]

I use the HatDrive top. When I activate the option POWER_OFF_ON_HALT=1 and shutdown the raspberry, it still needs 0,8Watt and the ACT Led ist blinking 1 sec on and 1 sec off. Is there another option to poweroff the HatDrive completely?

HatDrive Top (2024/V2) or HatDrive Top Lite supports sleep, V1 was designed when this feature was not available (we started selling the board in early November and the spec/firmware support for HAT+ was released a month later.

HatDrive Bottom (2024/V3) that supports sleep is shipping for over 2.5 months now

[mightymos]

Well the Aliexpress cable did not appear to work at all...nothing showed up from lspci whereas drives would normally at least show up with stock cable. I believe my cable was seated properly.

I then returned to the stock cable and was able to write a lite image to an SN530 NVMe drive, which is listed as compatible - unlike my Kioxia drive. However rebooting the Pi would still not boot with the NVMe. I then booted via USB adapter with SSD drive and lspci did not show SN530 any longer despite it just successfully writing.

It seems like I am out of options for more experimenting within budget so would plan to boot by USB adapter in the future. I suppose the benefit here is that USB is fairly simply to swap to another system if needed. In any case I'm not sure if this helps anyone but has been my experience so far.