limited performance

[ktsaou]

Hi all,

I have a raidz2 array with 4x 2TB drives, non AF.
At the beginning of the disks there is a small ext4 md-raid1 partition and the rest of the disks are assigned to the raidz2 pool. I have used ashift=12 at pool creation.

My workstation is a quad core at 3GHz with 4GB ram, running gentoo 64bit.

For some reason, zfs reads and writes are limited to around 78MB/s.

The same array was running md raid-5 giving aroung 150MB/s write speed and 250MB/s read speed.

I did this test:

create an 8GB file in zfs:

# rm /data/test; sync; dd if=/dev/zero of=/data/test bs=4k count=$((8*256*1024))
2097152+0 records in
2097152+0 records out
8589934592 bytes (8,6 GB) copied, 109,603 s, 78,4 MB/s

read it back:

# dd if=/data/test of=/dev/null bs=4k count=$((8*256*1024))
2097152+0 records in
2097152+0 records out
8589934592 bytes (8,6 GB) copied, 109,499 s, 78,4 MB/s

test the read performance of disks and controller:

dd if=/dev/sda2 of=/dev/null bs=4k count=$((2*256*1024)) &
dd if=/dev/sdb2 of=/dev/null bs=4k count=$((2*256*1024)) &
dd if=/dev/sdc2 of=/dev/null bs=4k count=$((2*256*1024)) &
dd if=/dev/sdd2 of=/dev/null bs=4k count=$((2*256*1024)) &
wait
524288+0 records in
524288+0 records out
2147483648 bytes (2,1 GB) copied, 17,4306 s, 123 MB/s
524288+0 records in
524288+0 records out
2147483648 bytes (2,1 GB) copied, 17,8363 s, 120 MB/s
524288+0 records in
524288+0 records out
2147483648 bytes (2,1 GB) copied, 18,5661 s, 116 MB/s
524288+0 records in
524288+0 records out
2147483648 bytes (2,1 GB) copied, 23,6243 s, 90,9 MB/s

I guess this proves my controller can go higher.

I see though that my sdd is slower that the others.
I did try to get it out of the pool:

# zpool offline pool scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1168223-part2
# zpool status
  pool: pool
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: resilvered 28K in 0h0m with 0 errors on Sun Sep  8 14:03:53 2013
config:

        NAME                                                 STATE     READ WRITE CKSUM
        pool                                                 DEGRADED     0     0     0
          raidz2-0                                           DEGRADED     0     0     0
            scsi-SATA_SAMSUNG_HD204UIS2H7JD6B102943-part2    ONLINE       0     0     0
            scsi-SATA_SAMSUNG_HD204UIS2H7JD6B103015-part2    ONLINE       0     0     0
            scsi-SATA_ST2000DL003-9VT_5YD3JJYZ-part2         ONLINE       0     0     0
            scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1168223-part2  OFFLINE      0     0     0

errors: No known data errors

# dd if=/data/test of=/dev/null bs=4k count=$((8*256*1024))
2097152+0 records in
2097152+0 records out
8589934592 bytes (8,6 GB) copied, 110,078 s, 78,0 MB/s

The read performance of the 3 disks gives exactly the same performance of the 4 disks.

I did try the same test to my md raid-1 array (first partition of the same disks):

# rm /test; sync; dd if=/dev/zero of=/test bs=4k count=$((8*256*1024))
2097152+0 records in
2097152+0 records out
8589934592 bytes (8,6 GB) copied, 87,5838 s, 98,1 MB/s
# dd if=/test of=/dev/null bs=4k count=$((8*256*1024))
2097152+0 records in
2097152+0 records out
8589934592 bytes (8,6 GB) copied, 71,0247 s, 121 MB/s

Any ideas why zfs is limiting my I/O to about 78MB/s?

Kind regards,

Costa

[ryao]

What is happening is that ZFS is not coalescing the 4K writes to a file while md raid is coalescing the writes to the block device. Try a larger blocksize, like bs=128k or bs=1m.

By the way, md raid lacks much of the layering that exists in ZFS to provide a filesystem with transactional semantics. If you are interested in things like virtual machine storage, it would be better to use a zvol to avoid the filesystem overhead.

[ktsaou]

 # rm /data/test; sync; dd if=/dev/zero of=/data/test bs=1M count=8000
8000+0 records in
8000+0 records out
8388608000 bytes (8,4 GB) copied, 85,8561 s, 97,7 MB/s

write performance improved. I did see spikes of 122MB/s at zpool iostat.

 # dd if=/data/test of=/dev/null bs=1M
8000+0 records in
8000+0 records out
8388608000 bytes (8,4 GB) copied, 113,427 s, 74,0 MB/s

read performance dropped. The highest spike I saw in iostat was 85MB/s. Each disk was reading at 15 to 20MB/s. I did tried it a few times.

Note that I have limited arc with:

echo 536870912 >> /sys/module/zfs/parameters/zfs_arc_max

before that my system was freezing during these tests.

[ryao]

I updated my previous comment with some more information before your response.

Anyway, dd performance is the one thing in which md raid excels. It can avoid partial stripe writes by buffering. The disk heads themselves do not need to move more than one track at a time because they receive a sequential stream of writes. When you have a file system between md raid and what you write, this is no longer the case because you then need to update filesystem structures. Extent based file systems like XFS tend to minimize this and permit the data to be written in a stream. The downside is that do not checksum data / self-heal. Putting one between dd and md raid would impose some overhead. CoW filesystems like ZFS do not use extents because they must have small units of storage so that CoW overhead remains small. Each time a change is done, they must calculate checksums and propagate the checksum updates up a tree. This is done to relatively random locations to minimize the possibility of a head drop wiping out the meta data structures (which are stored twice). The downside is that sequential IO performance suffers. You could probably get higher sequential performance by switching to a zvol (with volblocksize=12k) and/or setting sync=disabled (which would give ZFS the opportunity to behave more like MD RAID), but I do not see much opportunity to improve this further without code modifications that have been discussed/planned for future versions.

[ktsaou]

By the way, md raid lacks much of the layering that exists in ZFS to provide a filesystem with transactional semantics. If you are interested in things like virtual machine storage, it would be better to use a zvol to avoid the filesystem overhead.

This is mainly a media library NAS, i.e. many large files shared to media players and encoders. If I can get Gbps read/write I'll keep it with zfs. For the moment however, real world samba and nfs read/write performance is awfull. It averages bellow 40MB/s for just one client. If 2 or 3 clients are reading/writting, it goes bellow 10MB/s for each. With md raid-5 I was always getting 1Gbps speed on this machine's ethernet, with any number of clients, and the disks had some idle time.

[ktsaou]

I created a zfs filesystem with sync=disabled and checksum=off.

# dd if=/dev/zero of=/aaa/test bs=1M count=8000
8000+0 records in
8000+0 records out
8388608000 bytes (8,4 GB) copied, 69,5444 s, 121 MB/s

write performance improved drastically.

# dd if=/aaa/test of=/dev/null bs=1M count=8000
8000+0 records in
8000+0 records out
8388608000 bytes (8,4 GB) copied, 116,892 s, 71,8 MB/s

read performance however lowers even more.

I do understand all you said for justifying write performance.
I can't get it though why reads are so slow.

[ryao]

That is a good question. I do not see an obvious explanation from what you have told me. How did you create your pool? The HD204UI drives are advanced format disks (that misreport sector size), so if you forced ashift=9, unaligned IO could be hurting performance. Doing partitioning yourself could also somewhat diminish performance because ZFS would not be setting the Linux IO scheduler to noop to avoid duplication of effort and could result in misalignment.

By the way, If you are familiar with IRC, you might want to ask for suggestions in #zfsonlinux on freenode. That should have a quicker response time than the issue tracker. I realize that I have been fairly quick in my responses, but that response time is abnormally short.

[ktsaou]

You have been perfect. Thank you very much.

The HD204UI drives are advanced format disks

I didn't know that. Sorry.

so if you forced ashift=9,

I forced ashift=12. I have done nothing for partition alignment though (and I don't know if this matters at all).
This is how my disks look:

# fdisk /dev/sda
Welcome to fdisk (util-linux 2.21.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2982c9d3

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048   104859647    52428800   fd  Linux raid autodetect
/dev/sda2       104859648  3907029167  1901084760   83  Linux

Should I somehow align partition start of sda2 for zfs to be optimal?

If you are familiar with IRC, you might want to ask for suggestions in #zfsonlinux on freenode.

It has been more than a decade since the last time I used IRC, but I'll give it a try.

Many many thanks for your time.

Regards,

Costa

[ktsaou]

Also, I created the pool like this:

zpool create pool -f -o ashift=12 -m /pool raidz2 /dev/disk/by-id/scsi-SATA_SAMSUNG_HD204UIS2H7JD6B102943-part2 /dev/disk/by-id/scsi-SATA_SAMSUNG_HD204UIS2H7JD6B103015-part2 /dev/disk/by-id/scsi-SATA_ST2000DL003-9VT_5YD3JJYZ-part2 /dev/disk/by-id/scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1168223-part2

[ryao]

This is aligned properly. For your workload, I would set primarcache=metadata. You could also experiment with the module option to disable prefetch.

[ktsaou]

primarcache=metadata

With this, writes are at 123MB/s and reads were improved a little, at 92MB/s.

I disabled prefetch with this:

echo 1 >>/sys/module/zfs/parameters/zfs_prefetch_disable

and reads went up to 96MB/s. I saw spikes at 120MB/s.
My disks have read-ahead enabled. Should I disable this too?

[ktsaou]

Further findings:

with primarcache=metadata read speed of a single SMB client dropped bellow 10MB/s.
I enabled it back and went to 30MB/s.

I enabled prefetch, and SMB read speed is now 45MB/s for a single client.

[ktsaou]

ryao many thanks for everything.

I am going back to md raid-6.
Unfortunatelly, zfs performance in real workload is less than half compared to md raid, at least for the NAS I need.
ZFS is an awesome filesystem.I hope that soon it will have all the linux performance issues settled.

Best regards,

Costa

[ryao]

It might be useful to keep this open so that we can track it.