Test Broadcom MegaRAID 9460-16i storage controller #72

geerlingguy · 2021-02-12T22:47:18Z

Now this is an interesting one. Full story to come soon, but I've been testing Broadcom's MegaRAID 9460-16i for a bit (thanks to @joshsinykin at Broadcom for helping me get set up!), and (spoilers) I actually have SAS RAID storage working with the Pi now.

Here's a picture of the card:

I'll be posting the basic details soon, and once I can polish up the process for getting drivers working, I'll also add in that detail (right now it requires a bit of tweaking that I might not be necessary soon!).

I also have it on good authority the 9560-16i also works with the Pi :)

geerlingguy · 2021-02-17T15:12:24Z

$ sudo lspci -vvvv
...
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3516 (rev 01)
	Subsystem: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3516
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 255
	Region 0: Memory at 600200000 (64-bit, prefetchable) [disabled] [size=1M]
	Region 2: Memory at 600300000 (64-bit, prefetchable) [disabled] [size=1M]
	Region 4: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=1M]
	Region 5: I/O ports at <unassigned> [disabled]
	[virtual] Expansion ROM at 600100000 [disabled] [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [148 v1] Power Budgeting <?>
	Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [168 v1] #19
	Capabilities: [254 v1] #16
	Capabilities: [284 v1] Vendor Specific Information: ID=0002 Rev=1 Len=100 <?>
	Capabilities: [384 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Capabilities: [3bc v1] #15

geerlingguy · 2021-02-17T16:10:17Z

All right, you must be joking.

Spoilers—everything's already working, and I'm just taking some time to compile the info to put on this issue at some point... but I've been scratching my head as to why I've only been able to get ~20 MB/sec for network copies (mostly testing with SMB and NFS) through the Pi with either the SSD or SAS drives attached.

It turns out, I had both WiFi and Wired networks enabled on my Mac, and even though I have all the network priority set up for wired, for some reason SMB, NFS, and rsync/SSH copies were all being performed over WiFi! It was when I noticed all copies were about the same speed (~144 Mbps) that I started suspecting this to be the problem.

Ugh, why can't multiple network interfaces ever be easy?!

geerlingguy · 2021-02-17T20:10:52Z

Documenting here the entire bringup process:

Prerequisite: You'll need a driver from Broadcom directly that supports ARMv7 / ARM64—the changes in the driver I used won't be available from the public download pages for a while.

Flash Pi OS 64-bit to the CM4 eMMC or a microSD card.
Boot the Pi.

Recompile the Pi OS kernel with CONFIG_IRQ_POLL=y:

sudo apt-get install -y git bc bison flex libssl-dev make
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux
make ARCH=arm64 bcm2711_defconfig
nano .config  # Set CONFIG_IRQ_POLL=y in here...
make -j4 ARCH=arm64 Image modules dtbs  # This took ~1 hour on my CM4...
sudo make ARCH=arm64 modules_install
sudo cp arch/arm64/boot/Image /boot/kernel8.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/

Install ctags (prerequisite to compiling the Broadcom driver): sudo apt-get install -y ctags

Expand and compile the driver:

tar -xvf megaraid_sas-07.718.00.00-src.tar.gz
cd megaraid_sas-07.718.00.00/
./compile.sh

Enable the module: sudo insmod megaraid_sas.ko (assuming you have the card plugged in and a backplane or storage devices plugged into the card already).

At this point, you can use StorCLI to work on volumes and drives connected to the card—if it initialized correctly.

Here's what it looks like in dmesg once the card is successfully initialized:

[  138.129230] megaraid_sas: loading out-of-tree module taints kernel.
[  138.135261] megasas: 07.718.00.00
[  138.135732] pci 0000:00:00.0: enabling device (0000 -> 0002)
[  138.135767] megaraid_sas 0000:01:00.0: enabling device (0000 -> 0002)
[  138.136656] megaraid_sas 0000:01:00.0: BAR:0x0  BAR's base_addr(phys):0x0x0000000600200000  mapped virt_addr:0x00000000238916d8
[  138.136677] megaraid_sas 0000:01:00.0: Waiting for FW to come to ready state
[  139.400492] megaraid_sas 0000:01:00.0: FW now in Ready state
[  139.400515] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
[  139.402988] megaraid_sas 0000:01:00.0: firmware supports msix	: (128)
[  139.403516] megaraid_sas 0000:01:00.0: requested/available msix 5/5
[  139.403531] megaraid_sas 0000:01:00.0: current msix/online cpus	: (5/4)
[  139.403543] megaraid_sas 0000:01:00.0: RDPQ mode	: (enabled)
[  139.403559] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 3053	 LDIO threshold: 0
[  139.605418] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
[  139.605440] megaraid_sas 0000:01:00.0: FW supports sync cache	: Yes
[  139.605452] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  139.716509] megaraid_sas 0000:01:00.0: FW provided supportMaxExtLDs: 1	max_lds: 64
[  139.716518] megaraid_sas 0000:01:00.0: controller type	: MR(4096MB)
[  139.716523] megaraid_sas 0000:01:00.0: Online Controller Reset(OCR)	: Enabled
[  139.716530] megaraid_sas 0000:01:00.0: Secure JBOD support	: Yes
[  139.716530] megaraid_sas 0000:01:00.0: NVMe passthru support	: Yes
[  139.716539] megaraid_sas 0000:01:00.0: FW provided TM TaskAbort/Reset timeout	: 6 secs/60 secs
[  139.716543] megaraid_sas 0000:01:00.0: PCI Lane Margining support	: No
[  139.716550] megaraid_sas 0000:01:00.0: JBOD sequence map support	: Yes
[  139.745384] megaraid_sas 0000:01:00.0: NVME page size	: (4096)
[  139.745532] megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[  139.745532] megaraid_sas 0000:01:00.0: INIT adapter done
[  139.747717] megaraid_sas 0000:01:00.0: Snap dump wait time	: 15
[  139.747735] megaraid_sas 0000:01:00.0: pci id		: (0x1000)/(0x0014)/(0x1000)/(0x9460)
[  139.747740] megaraid_sas 0000:01:00.0: unevenspan support	: no
[  139.747740] megaraid_sas 0000:01:00.0: firmware crash dump	: no
[  139.747740] megaraid_sas 0000:01:00.0: JBOD sequence map	: enabled
[  139.747898] megaraid_sas 0000:01:00.0: Max firmware commands: 3052 shared with nr_hw_queues = 4
[  139.747904] scsi host0: Broadcom SAS based MegaRAID driver
[  139.769816] scsi 0:0:97:0: Enclosure         LSI      VirtualSES       03   PQ: 0 ANSI: 7
[  139.778076] scsi 0:0:97:0: Attached scsi generic sg0 type 13
[  139.789480] scsi 0:2:0:0: Direct-Access     AVAGO    MR9460-16i       5.16 PQ: 0 ANSI: 5
[  139.790170] sd 0:2:0:0: [sda] 700121088 512-byte logical blocks: (358 GB/334 GiB)
[  139.790216] sd 0:2:0:0: [sda] Write Protect is off
[  139.790223] sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
[  139.790295] sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  139.790410] sd 0:2:0:0: Attached scsi generic sg1 type 0
[  139.790606] sd 0:2:0:0: [sda] Optimal transfer size 196608 bytes
[  139.791735] scsi 0:2:1:0: Direct-Access     AVAGO    MR9460-16i       5.16 PQ: 0 ANSI: 5
[  139.792089] sd 0:2:0:0: [sda] Attached SCSI disk
[  139.792341] sd 0:2:1:0: Attached scsi generic sg2 type 0
[  139.792558] sd 0:2:1:0: [sdb] 1754529792 512-byte logical blocks: (898 GB/837 GiB)
[  139.793181] sd 0:2:1:0: [sdb] Write Protect is off
[  139.793196] sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
[  139.793281] sd 0:2:1:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  139.793594] sd 0:2:1:0: [sdb] Optimal transfer size 786432 bytes
[  139.799745] sd 0:2:1:0: [sdb] Attached SCSI disk

You'll note this log is from after I had already used StorCLI to build two volumes, which are attached as sda and sdb once the driver loads.

We were originally having issues where we'd get the following messages after a long delay midway through initialization:

[  372.867846] megaraid_sas 0000:01:00.0: Init cmd return status FAILED for SCSI host 0
[  373.054122] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6747

geerlingguy · 2021-02-17T20:11:15Z

I should also mention the exact same process is required if you want to run this card on 32-bit Raspberry Pi OS.

geerlingguy · 2021-02-19T17:36:56Z

Just to shed a little more light on the problems we had with the Pi OS 64-bit kernel:

It seems like the BAR allocations and memory access is a little strange, where some accesses are 32-bit for 64-bit-configured BAR space.
On 64-bit Pi, writeq/readq exist, but seem to be disabled/not working, so that causes confusion in some drivers
Other drivers and Pi-based projects that run into this issue seem to implement Pi-specific workarounds
It might be something that can be more easily solved in the Pi's kernel fork, by adding a config option or implementing some form of writeq that works correctly.

geerlingguy · 2021-02-19T18:21:38Z

Linking back to the kernel bug report: raspberrypi/linux#4158

geerlingguy · 2021-02-26T17:36:16Z

fio command and config for 1MB random read:

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio

$ cat 1024K-rand-read.fio 
[global]
buffered=0
random_generator=tausworthe64
scramble_buffers=1
sync=0
direct=1
percentile_list=1:5:10:20:30:40:50:60:70:80:90:95:99:99.5:99.9:99.95:99.99
offset=0
ioengine=libaio
cpus_allowed=0-3
norandommap
random_distribution=random:100
do_verify=0
randrepeat=0
continue_on_error=none
nice=-5
blockalign=4096
blocksize=1024k
readwrite=randread
rwmixread=100
percentage_random=100

[Target0]
filename=/dev/sda
numjobs=1
flow_id=0

sda (Kingston SSDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio
Target0: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=399MiB/s][r=399 IOPS][eta 00m:00s]
Target0: (groupid=0, jobs=1): err= 0: pid=760: Fri Feb 26 17:38:50 2021
  read: IOPS=395, BW=397MiB/s (417MB/s)(17.5GiB/45106msec)
    slat (nsec): min=0, max=2519.9k, avg=177039.27, stdev=56358.83
    clat (nsec): min=0, max=833010k, avg=161201258.61, stdev=81075354.71
     lat (nsec): min=0, max=833306k, avg=161379074.55, stdev=81077781.78
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   87], 10.00th=[   96], 20.00th=[  110],
     | 30.00th=[  124], 40.00th=[  148], 50.00th=[  157], 60.00th=[  161],
     | 70.00th=[  165], 80.00th=[  174], 90.00th=[  207], 95.00th=[  317],
     | 99.00th=[  550], 99.50th=[  634], 99.90th=[  751], 99.95th=[  810],
     | 99.99th=[  818]
   bw (  KiB/s): min=    0, max=423936, per=99.87%, avg=406278.97, stdev=9308.35, samples=90
   iops        : min=    0, max=  414, avg=396.74, stdev= 9.10, samples=90
  lat (msec)   : 50=0.08%, 100=12.66%, 250=80.93%, 500=5.34%, 750=1.25%
  lat (msec)   : 1000=0.10%
  cpu          : usr=0.57%, sys=6.69%, ctx=30338, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=17856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=397MiB/s (417MB/s), 397MiB/s-397MiB/s (417MB/s-417MB/s), io=17.5GiB (18.8GB), run=45106-45106msec

Disk stats (read/write):
  sda: ios=131038/0, merge=0/0, ticks=16892098/0, in_queue=16892098, util=99.93%

sdb (HP Proliant 10K SAS HDDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio
Target0: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][0.9%][r=106MiB/s][r=106 IOPS][eta 01h:38m:56s]
Target0: (groupid=0, jobs=1): err= 0: pid=780: Fri Feb 26 17:45:24 2021
  read: IOPS=144, BW=146MiB/s (153MB/s)(6650MiB/45448msec)
    slat (nsec): min=0, max=512241, avg=271826.49, stdev=31943.28
    clat (nsec): min=0, max=1416.9M, avg=439770563.63, stdev=276100098.44
     lat (nsec): min=0, max=1417.1M, avg=440043348.41, stdev=276100397.68
    clat percentiles (msec):
     |  1.00th=[  129],  5.00th=[  155], 10.00th=[  176], 20.00th=[  209],
     | 30.00th=[  247], 40.00th=[  292], 50.00th=[  355], 60.00th=[  426],
     | 70.00th=[  518], 80.00th=[  651], 90.00th=[  852], 95.00th=[ 1028],
     | 99.00th=[ 1334], 99.50th=[ 1351], 99.90th=[ 1385], 99.95th=[ 1385],
     | 99.99th=[ 1418]
   bw (  KiB/s): min=    0, max=174080, per=99.99%, avg=149818.16, stdev=12032.10, samples=90
   iops        : min=    0, max=  170, avg=146.19, stdev=11.76, samples=90
  lat (msec)   : 4=0.06%, 10=0.02%, 100=0.02%, 250=30.83%, 500=38.00%
  lat (msec)   : 750=17.58%, 1000=8.87%
  cpu          : usr=0.31%, sys=4.70%, ctx=6795, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=6587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=6650MiB (6973MB), run=45448-45448msec

Disk stats (read/write):
  sdb: ios=15751/0, merge=0/0, ticks=5444861/0, in_queue=5444861, util=99.87%

geerlingguy · 2021-02-26T17:37:30Z

fio command, config, and results for 1MB random write:

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio

$ cat 1024K-rand-write.fio 
[global]
buffered=0
random_generator=tausworthe64
scramble_buffers=1
sync=0
direct=1
percentile_list=1:5:10:20:30:40:50:60:70:80:90:95:99:99.5:99.9:99.95:99.99
offset=0
ioengine=libaio
cpus_allowed=0-3
norandommap
random_distribution=random:100
do_verify=0
randrepeat=0
continue_on_error=none
nice=-5
blockalign=4096
blocksize=1024k
readwrite=randwrite
rwmixread=0
percentage_random=100

[Target0]
filename=/dev/sda
numjobs=1
flow_id=0

sda (Kingston SSDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio
Target0: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][w=311MiB/s][w=311 IOPS][eta 00m:00s]
Target0: (groupid=0, jobs=1): err= 0: pid=786: Fri Feb 26 17:46:48 2021
  write: IOPS=274, BW=276MiB/s (289MB/s)(12.2GiB/45185msec); 0 zone resets
    slat (nsec): min=0, max=1126.6k, avg=426122.06, stdev=102943.94
    clat (nsec): min=0, max=1605.8M, avg=232241941.35, stdev=206265049.39
     lat (nsec): min=0, max=1606.3M, avg=232669227.52, stdev=206267504.39
    clat percentiles (msec):
     |  1.00th=[   58],  5.00th=[   65], 10.00th=[   71], 20.00th=[   88],
     | 30.00th=[  113], 40.00th=[  161], 50.00th=[  188], 60.00th=[  203],
     | 70.00th=[  220], 80.00th=[  275], 90.00th=[  527], 95.00th=[  701],
     | 99.00th=[ 1003], 99.50th=[ 1116], 99.90th=[ 1502], 99.95th=[ 1569],
     | 99.99th=[ 1569]
   bw (  KiB/s): min=    0, max=372736, per=99.86%, avg=281749.71, stdev=89010.67, samples=90
   iops        : min=    0, max=  364, avg=275.07, stdev=86.94, samples=90
  lat (msec)   : 50=0.01%, 100=26.46%, 250=50.74%, 500=12.33%, 750=7.00%
  lat (msec)   : 1000=2.95%
  cpu          : usr=7.33%, sys=4.14%, ctx=25666, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,12387,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=12.2GiB (13.1GB), run=45185-45185msec

Disk stats (read/write):
  sda: ios=15/97990, merge=0/0, ticks=5/20626166, in_queue=20626172, util=99.68%

sdb (HP Proliant 10K SAS HDDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio
Target0: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][0.6%][w=68.0MiB/s][w=68 IOPS][eta 02h:30m:02s]
Target0: (groupid=0, jobs=1): err= 0: pid=803: Fri Feb 26 17:48:17 2021
  write: IOPS=95, BW=96.8MiB/s (102MB/s)(4418MiB/45623msec); 0 zone resets
    slat (nsec): min=0, max=735129, avg=475278.54, stdev=77991.79
    clat (nsec): min=0, max=1285.4M, avg=664899695.04, stdev=94807677.18
     lat (nsec): min=0, max=1285.7M, avg=665376073.45, stdev=94807409.77
    clat percentiles (msec):
     |  1.00th=[  510],  5.00th=[  558], 10.00th=[  584], 20.00th=[  600],
     | 30.00th=[  617], 40.00th=[  634], 50.00th=[  651], 60.00th=[  667],
     | 70.00th=[  684], 80.00th=[  709], 90.00th=[  776], 95.00th=[  852],
     | 99.00th=[ 1020], 99.50th=[ 1133], 99.90th=[ 1267], 99.95th=[ 1267],
     | 99.99th=[ 1284]
   bw (  KiB/s): min=    0, max=129024, per=98.82%, avg=97987.56, stdev=16416.15, samples=91
   iops        : min=    0, max=  126, avg=95.59, stdev=16.03, samples=91
  lat (msec)   : 500=0.99%, 750=88.79%, 1000=10.54%
  cpu          : usr=2.43%, sys=2.92%, ctx=4587, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,4355,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=96.8MiB/s (102MB/s), 96.8MiB/s-96.8MiB/s (102MB/s-102MB/s), io=4418MiB (4633MB), run=45623-45623msec

Disk stats (read/write):
  sdb: ios=53/13528, merge=0/0, ticks=133/6968760, in_queue=6968894, util=99.87%

Those SAS drives are getting slower :D

geerlingguy · 2021-02-26T17:52:35Z

Marking this as 'testing complete' (for now... I have some things in mind for this card in the future!).

I published a video on the card / backplane today: Hardware RAID for the fastest Raspberry Pi CM4 NAS!.

Blog post, too: Hardware RAID on the Raspberry Pi CM4.

xdays · 2021-06-25T02:07:38Z

Hi @geerlingguy can you share the the patched driver source code with me? I got a chance to own this card, I'd like to give it a try, thanks!

geerlingguy · 2021-06-28T16:21:45Z

@xdays - Sure! You'd need to use the 32-bit Pi OS for it to work with the publicly-available source. I can send you the patch—send me an email and reference this conversation :)

xdays · 2021-06-30T01:39:48Z

@xdays - Sure! You'd need to use the 32-bit Pi OS for it to work with the publicly-available source. I can send you the patch—send me an email and reference this conversation :)

Thanks! email: [redacted]

xdays · 2021-08-28T07:13:28Z

Hi @geerlingguy When I start testing this controller I always get this error:

Aug 28 03:29:31 raspberrypi kernel: brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
Aug 28 03:29:31 raspberrypi kernel: brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
Aug 28 03:29:31 raspberrypi kernel: brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x06ffffffff -> 0x00c0000000
Aug 28 03:29:31 raspberrypi kernel: brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
Aug 28 03:29:31 raspberrypi kernel: brcm-pcie fd500000.pcie: link down

I've tried 12V 3.5A and 12V 5A adapter, neither works.

I also re-compile the kernel and install the patched version of megaraid_sas driver, doesn't work. And this should work after pcie link is up.

At the same time the controller card is hot, so I assume it's up and running and just don't reply pi.

BTW, with same X1 to X16 cable, I can access nvme drive and one old lsi raid controller.

Thanks!

geerlingguy · 2021-08-28T22:10:52Z

If you're getting link down that means the PCIe bus isn't even seeing the card, so I wonder if there could be some strange issue with power or the adapter, even if it works with some other devices.

Note that the 9460-16i does use a hefty amount of power—it worked with this 12V 5A adapter but wouldn't work with a 12V 3A adapter I tested.

geerlingguy added testing in progress not-on-site-yet and removed not-on-site-yet labels Feb 12, 2021

geerlingguy added a commit that referenced this issue Feb 16, 2021

Issue #72: Add Broadcom MegaRAID 9460-16i storage controller.

e773a3d

joshsinykin mentioned this issue Feb 19, 2021

writeq() on 64bit 5.10.XX branch does not issue PCIe cycle, switching to two writel() does work. raspberrypi/linux#4158

Open

geerlingguy mentioned this issue Feb 19, 2021

Building a custom Kernel to improve compatibility #73

Closed

geerlingguy added testing complete and removed testing in progress labels Feb 26, 2021

geerlingguy closed this as completed Feb 26, 2021

This was referenced Aug 12, 2021

Test LSI 9305-16i PCIe SAS HBA #195

Closed

Test LSI 9405W-16i SAS/NVMe HBA #196

Closed

melanj pushed a commit to melanj/raspberry-pi-pcie-devices that referenced this issue Nov 10, 2021

Issue geerlingguy#72: Add Broadcom MegaRAID 9460-16i storage controller.

f3a6239

This was referenced Apr 28, 2022

Patched LSI/Broadcom mpt3sas driver for 64-bit BCM2711 raspberrypi/linux#5013

Closed

Patched LSI/Broadcom mpt3sas driver for 64-bit BCM2711 geerlingguy/linux#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Broadcom MegaRAID 9460-16i storage controller #72

Test Broadcom MegaRAID 9460-16i storage controller #72

geerlingguy commented Feb 12, 2021 •

edited

Loading

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 17, 2021 •

edited

Loading

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 19, 2021

geerlingguy commented Feb 19, 2021

geerlingguy commented Feb 26, 2021 •

edited

Loading

geerlingguy commented Feb 26, 2021 •

edited

Loading

geerlingguy commented Feb 26, 2021

xdays commented Jun 25, 2021

geerlingguy commented Jun 28, 2021

xdays commented Jun 30, 2021 •

edited by geerlingguy

Loading

xdays commented Aug 28, 2021

geerlingguy commented Aug 28, 2021

Test Broadcom MegaRAID 9460-16i storage controller #72

Test Broadcom MegaRAID 9460-16i storage controller #72

Comments

geerlingguy commented Feb 12, 2021 • edited Loading

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 17, 2021 • edited Loading

geerlingguy commented Feb 17, 2021

geerlingguy commented Feb 19, 2021

geerlingguy commented Feb 19, 2021

geerlingguy commented Feb 26, 2021 • edited Loading

sda (Kingston SSDs)

sdb (HP Proliant 10K SAS HDDs)

geerlingguy commented Feb 26, 2021 • edited Loading

sda (Kingston SSDs)

sdb (HP Proliant 10K SAS HDDs)

geerlingguy commented Feb 26, 2021

xdays commented Jun 25, 2021

geerlingguy commented Jun 28, 2021

xdays commented Jun 30, 2021 • edited by geerlingguy Loading

xdays commented Aug 28, 2021

geerlingguy commented Aug 28, 2021

geerlingguy commented Feb 12, 2021 •

edited

Loading

geerlingguy commented Feb 17, 2021 •

edited

Loading

geerlingguy commented Feb 26, 2021 •

edited

Loading

geerlingguy commented Feb 26, 2021 •

edited

Loading

xdays commented Jun 30, 2021 •

edited by geerlingguy

Loading