Test performance of a variety of microSD cards on Raspberry Pi - sequential and random read/write

[geerlingguy]

I have an old 64GB USB 3.0 SSD I pulled from an old MacBook Air, and it gets 150-300MB/sec transfer, with quite low latency. It would definitely saturate the USB 2.0 bus on the Pi, but it might be a heck of a lot faster as the data store for MySQL than using the normal mount inside the cheap Kingston microSD card!

It's worth investigating at least, and maybe in the database playbook, the configuration could allow for internal vs. external drive configuration.

Format a USB-connected drive on the Raspberry Pi in Raspbian

1. Get the drive's path using $ sudo fdisk -l
2. Enter fdisk to edit the disk's partition table: $ sudo fdisk /dev/sda1*
   1. d, n, p, 1, <enter>, <enter>, w
3. Format the partition: $ sudo mkfs -t ext4 /dev/sda1*
4. Create a directory (e.g. /ssd) and mount the partition: $ sudo mount /dev/sda1 /ssd
5. (If you want the disk mounted on boot) Edit fstab and add the drive: $ sudo nano /etc/fstab

See this post for more tips.

We're assuming the device is sda1. It could be something else, depending on your system's configuration/hardware.

Test the raw read/write speed of the disk with hdparm and dd

Install hdparm first, with $ sudo apt-get install -y hdparm.

Read speed, SSD:

$ sudo hdparm -t /dev/sda1
 Timing buffered disk reads:  72 MB in  3.05 seconds =  23.59 MB/sec

$ sudo hdparm -T /dev/sda1
 Timing cached reads:   464 MB in  2.01 seconds = 231.31 MB/sec

Read speed, internal microSD (Kingston Class 10):

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  38 MB in  3.02 seconds =  12.57 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   488 MB in  2.00 seconds = 244.02 MB/sec

Read speed, internal microSD (SanDisk Ultra Class 10):

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  54 MB in  3.07 seconds =  17.59 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   460 MB in  2.00 seconds = 229.54 MB/sec

Read speed, internal microSD (Transcend Premium 300x 32GB):

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  54 MB in  3.11 seconds =  17.38 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   498 MB in  2.00 seconds = 249.00 MB/sec

Read speed, internal microSD (Cheapo Class 4 4GB):

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  42 MB in  3.14 seconds =  13.37 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   430 MB in  2.00 seconds = 214.55 MB/sec

Write speed, SSD:

$ sudo dd if=/dev/zero of=/ssd/output bs=8k count=10k; sudo rm -f /ssd/output
83886080 bytes (84 MB) copied, 2.31056 s, 36.3 MB/s

Write speed, internal microSD (Kingston Class 10):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output

83886080 bytes (84 MB) copied, 2.02959 s, 41.3 MB/s

Write speed, internal microSD (SanDisk Ultra Class 10):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 2.03785 s, 41.2 MB/s

Write speed, internal microSD (Transcend Premium 300x 32GB):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 1.84696 s, 45.4 MB/s

Write speed, internal microSD (Cheap Class 4 4GB):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 0.492591 s, 170 MB/s

...looks like I need to do some more benchmarking and see whether the internal microSD card can hold its own against a fast SSD over USB 2.0! Maybe the Pi 2 is even better than the Pi in terms of latency/sustained speed for the directly-connected microSD slot?

Test random I/O and other things with iozone

Install iozone on the Raspberry Pi

1. $ wget http://www.iozone.org/src/current/iozone3_430.tar
2. $ cat iozone3_430.tar | tar -x
3. $ cd iozone3_430/src/current
4. $ make linux-arm
5. ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2

Test results

Internal Kingston 8GB microSD

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     1029     1204     4895     4797     4245       82
102400     512     9854    10237    13693    13661    13640     1116
102400   16384    10326    10368    13852    13826    13850    10363

Internal SanDisk Ultra 16GB microSD

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     1269     1467     4311     4311     4252      764
102400     512     9102     8252    18529    18467    18517     1571
102400   16384     7485    10029    18840    18843    18837     9605

Internal Transcend Premium 300x 32GB microSD

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     1369     1107     5047     5042     4280      933
102400     512     9349    10955    18766    18667    18732     5007
102400   16384    11067    11802    19046    19047    18961    11710

Internal Cheapo Class4 4GB microSD

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
TODO

External USB 3.0 64GB SSD

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /ssd/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     3877     5004     5184     5193     4553     4943
102400     512    29181    29630    30053    30036    29394    29554
102400   16384    31528    31540    32025    32153    32236    31446

Further notes on iozone usage

• -f filename "Used to specify the filename for the temporary file under test. This is useful when
  the unmount option is used. When testing with unmount between tests it is necessary for
  the temporary file under test to be in a directory that can be unmounted. It is not possible
  to unmount the current working directory as the process Iozone is running in this directory."
• SD card benchmark on the Raspberry Pi
• iozone.org
• Linux File System Read Write Performance Test

Comparison with 2013 MacBook Air

This is just to demonstrate the incredibly wide gap between a modest modern workstation's I/O capabilities and the Raspberry Pi (for anyone who believes that optimizing a Raspberry Pi for modern high performance computing applications is potentially worth the effort for anything besides education and fun).

Write speed, internal SSD (PCIe):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=100k; sudo rm -f /tmp/output
838860800 bytes transferred in 1.295498 secs (647519884 bytes/sec) (647 MB/sec)

Write speed, external SSD (USB 3.0):

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=100k; sudo rm -f /tmp/output
838860800 bytes transferred in 5.186630 secs (161735230 bytes/sec) (162 MB/sec)

Internal SSD (PCIe):

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4   349051   378714  2057008  2052849  1757784   384525
102400     512   378961   381589  1633720  1831942  1602355   397143
102400   16384   345783   255575  1726816  1740601  1561742   384016

External SSD (USB 3.0):

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /ssd/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4   106728    93730  1954872  1809223  1739037   164208
102400     512   140001   124136  1746128  1798353  1691446   160632
102400   16384    81202   101984  1545561  1809853  1528714   112968

External old-fashioned 2TB spinning disk (USB 3.0):

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /hdd/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4    92336    90505  1969492  2007127  1834830    86063
102400     512    99231    82251  1759390  1654712  1496792    95612
102400   16384    96631    87459  1532236  1525795  1325369    92766

(To build iozone on Mac OS X, I had to run make macosx instead of make linux-arm in step 4 above).

[geerlingguy]

Just for comparison, doing the same dd test on my little MacBook Air, I got 620.4 MB/s. Someday the Pi will have SATA at least, and/or USB 3.0. Someday...

[geerlingguy]

Transcend 300x card on Raspberry Pi 2

Using the same Transcend 300x card as with the B+, I re-ran all the tests on a new Raspberry Pi 2 model B:

hdparm - raw read speeds

(Prerequisite: sudo apt-get install -y hdparm).

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  54 MB in  3.07 seconds =  17.58 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   866 MB in  2.00 seconds = 432.48 MB/sec

dd - raw write speeds

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=25k; sudo rm -f /tmp/output
209715200 bytes (210 MB) copied, 4.2601 s, 49.2 MB/s

iozone - practical read/write scenarios

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     1593     1594     5752     5785     5507      991
102400     512     8143    12418    19002    18770    18919     6963
102400   16384    13021    13030    19191    19190    19093    10729

A few quick observations:

• Cached reads were more than double the speed of the B+.
• All disk IO operations seem to have improved slightly; though the limiting factor for small file random I/O is still the memory card/medium itself.
• The processor on the Pi 2 blows away the processor on the B+.
• Sequential writes with dd were even improved a little over the B+ results.

[geerlingguy]

SanDisk Extreme Pro 8GB card on Raspberry Pi 2

Prep instructions:

1. Boot Pi
2. Expand filesystem with sudo raspi-config (might need to sudo swapoff --all prior to this, and delete /var/swap file)
3. Reboot Pi
4. Install hdparm (sudo apt-get install -y hdparm)
5. Install ioperf
6. Run tests

hdparm - raw read speeds

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  56 MB in  3.06 seconds =  18.27 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   848 MB in  2.00 seconds = 423.66 MB/sec

dd - raw write speeds

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=25k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 0.279167 s, 300 MB/s

iozone - practical read/write scenarios

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     2600     2702     6584     6574     6602     1206
102400     512    17538    17628    18993    19005    19001     2895
102400   16384    17639    17832    19192    19195    19194    17824

[geerlingguy]

SanDisk Extreme 16GB card on Raspberry Pi 2

hdparm - raw read speeds

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  56 MB in  3.06 seconds =  18.30 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   852 MB in  2.00 seconds = 425.94 MB/sec

dd - raw write speeds

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 0.274278 s, 306 MB/s

iozone - practical read/write scenarios

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /home/pi/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     2314     2291     6857     6882     6601     1988
102400     512    17646    18209    19004    19004    18998    17771
102400   16384    18464    18401    19205    19156    19155    18394

[geerlingguy]

Samsung 16GB EVO card on Raspberry Pi 2

hdparm - raw read speeds

$ sudo hdparm -t /dev/mmcblk0
 Timing buffered disk reads:  54 MB in  3.11 seconds =  17.39 MB/sec

$ sudo hdparm -T /dev/mmcblk0
 Timing cached reads:   850 MB in  2.00 seconds = 424.32 MB/sec

dd - raw write speeds

$ sudo dd if=/dev/zero of=/tmp/output bs=8k count=10k; sudo rm -f /tmp/output
83886080 bytes (84 MB) copied, 0.278235 s, 301 MB/s

iozone - practical read/write scenarios

$ ./iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /home/pi/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     1759     1986     5639     5621     5364     1056
102400     512     9642    11599    18921    18839    18902     6533
102400   16384    11612    11607    19104    19104    18880     9680

[geerlingguy]

...and here are some rather depressing read/write stats (depressing because they show how much faster a proper USB 3.0 bus is for these fancy cards) from my MacBook Air using a USB 3.0 microSD card reader:

SanDisk Extreme Pro 8GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4    80292    63909  1926436  1938478  1748930    80047
102400     512    77696    78564  1540404  1662500  1477503    80289
102400   16384    78283    76706  1354555  1458234  1181668    79245

SanDisk Extreme 16GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4    43757    52241  1894611  2015954  1582739    52435
102400     512    52109    52293  1903132  1783470  1502830    52304
102400   16384    51984    52338  1730854  1748188  1573843    52300

SanDisk Ultra 16GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4    11442     9014  2246112  2270165  1876278     9913
102400     512     9974     9850  1744433  1884916  1653208    10260
102400   16384     9665    10601  1672116  1864254  1473050    10523

Samsung EVO 16GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4    10998     9033  1893976  1899782  1661497    10372
102400     512     9669    11573  1825992  1621869  1501863    12064
102400   16384    12692    12402  1784938  1547510  1422219    12280

Transcend Premium 300x 32GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     6551     8203  2127877  2136270  1831122     9094
102400     512    10806    12816  1991482  1869166  1704590    14685
102400   16384    13404    14460  1711214  1870742  1399345    15028

Kingston Class 10 8GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     7066     6885  1828370  1639189  1482317     8684
102400     512    10381    10020  1620914  1564789  1660931    10023
102400   16384    10424    10101  1665013  1522791  1445413    10422

Cheapo Class 4 4 GB

$ iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2 -f /Volumes/Test/test
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4     8865     7924  2168936  2119958  1860704     7940
102400     512     7404     5932    13780    14530    14287     7147
102400   16384     8066     5887  1718002  1730338  1516635     5998

[geerlingguy]

I've compiled these benchmarks here:

https://github.com/geerlingguy/raspberry-pi-dramble/wiki/microSD-Card-Benchmarks

[geerlingguy]

I've also added some info about external USB drives here:

https://github.com/geerlingguy/raspberry-pi-dramble/wiki/External-USB-drives

[mikkorantalainen]

If you really want to test flash memory performance, you need to use fio. Using dd or some else software that is not designed for this purpose is not a valid replacement.