Using Borg on (device-managed) SMR drives

[enkore]

~FAQ entry could look like this

Can I use Borg on SMR drives?
-----------------------------

SMR (shingled magnetic recording) drives behave very differently
from regular hard drives, meaning that applications have to
behave in certain ways or otherwise performance will be heavily
degraded.

Some Linux kernel versions between 3.19 and 4.5 had various bugs
handling device-managed SMR drives, leading to IO errors, unresponsive
drives and unreliable operation in general.

Borg 1.1.x ships with default settings suited for SMR drives.

(Removed, see below

To enhance performance, special filesystem flags can be used
(these are **experimental / under development kernel features**)
for ext4::

  mkfs.ext4 -E packed_meta_blocks=1,discard -O bigalloc,extent,sparse_super2 -b 4096 -C 8192 /dev/somePartition

)

Test was conducted using a Seagate Archive v2 8 TB (ST8000AS0002) which is pretty much the only model available to consumers. Kernel is Linux 4.9.13.

With an ext4 formatted like above and current master branch ~165 MB/s write speed are achieved when processing large files. The drive works continuously, seeking rarely, so apparently Borg and ext4 work nicely here together such that data is written directly to their zones and not first to the buffer and then later to the final zone.

About once every ~30 minutes either ext4/jbd or the drive, not sure who, decides to do something resulting in a short IO stall where the drive performs a lot of seeking for <10 seconds with immediate return to normal afterwards. I assume that this is related to ext4's journal.

When the drive remains idle for some time (5-10 mins) it seems to flush the internal buffer. In this case it seems to be done after less than a minute.

Read performance in "borg check" is good (130-195 MB/s, about ~140 MB/s average).

------------------------------------------------------------------------------
Archive name: first-smr-archive
Archive fingerprint: dcdbe5451d2f9ce732eb1d16f627b1ab557985fa400ded9c404d391f23820990
Time (start): Sat, 2017-03-04 17:10:08
Time (end):   Sat, 2017-03-04 18:09:08
Duration: 58 minutes 59.09 seconds
Number of files: 2391681
Utilization of max. archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              475.86 GB            394.27 GB            335.03 GB
All archives:              475.86 GB            394.27 GB            335.03 GB

                       Unique chunks         Total chunks
Chunk index:                 1466326              2346071
------------------------------------------------------------------------------
2417.24user 293.71system 59:08.89elapsed 76%CPU (0avgtext+0avgdata 753364maxresident)k
927481056inputs+655248640outputs (0major+287976minor)pagefaults 0swaps

Average repo write speed is 93 MB/s or 132 MB/s if you compensate for deduplication.

[ThomasWaldmann]

Maybe the last paragraph should be moved to the top of the entry. A lot of people run old distribution kernels and then they can stop reading after first paragraph and just buy a normal disk.

[enkore]

1.0 (5 MB segments): first ~14 GB okay-ish (40 MB/s), then starts to stall, jbd (ext4 journal) and borg iowait increases a lot (90-99 %), disk starts to seek a lot. Average write speed is erratic and varies in the <10 MB/s area. After aborting + umounting FS and some idle time drive goes, acoustically speaking, nuts. Seems to flush it's buffer again. Edit: Took over an hour before the drive was done.

[enkore]

1.1 on vanilla ext4: FS creation, mount, umount take some time. Performance seems to be almost identical to options given above (which does not mean that they don't make a difference - just not for Borg. This could be caused by Borg already being well-matched to the characteristics of these drives, so Borg doesn't need special FS options).

[ThomasWaldmann]

So, what's the bug?

[enkore]

Fat fingers

[enkore]

Here's a graph (made using JSON output + with timestamps + matplotlib + numpy gradient):

(Note: different settings + source as in other tests, not directly comparable)

Blue = instantaneous numerical differentiation of transfer amount, therefore noisy as expected, orange = running linear average (nominal 1 minute window) over that, red = latency

Notice how the latency regularly spikes, sometimes to very large levels (around minute 80) and for longer times.

I'm not sure why it's a bit faster with less spread over the first ~240 GB.

The intervals where transfer rate = 0 but latency = no spike is where duplicates where in the source, nothing unusual.

latency := running difference between progress updates from borg minus the nominal rate, so anything that blocks borg will show up here - but there shouldn't be anything except the SMR drive causing it (source is a regular, idle drive, no network).

Edit: Fancier graph: