New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Borg on (device-managed) SMR drives #2252

Closed
enkore opened this Issue Mar 4, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@enkore
Copy link
Contributor

enkore commented Mar 4, 2017

~FAQ entry could look like this

Can I use Borg on SMR drives?
-----------------------------

SMR (shingled magnetic recording) drives behave very differently
from regular hard drives, meaning that applications have to
behave in certain ways or otherwise performance will be heavily
degraded.

Some Linux kernel versions between 3.19 and 4.5 had various bugs
handling device-managed SMR drives, leading to IO errors, unresponsive
drives and unreliable operation in general.

Borg 1.1.x ships with default settings suited for SMR drives.

(Removed, see below

To enhance performance, special filesystem flags can be used
(these are **experimental / under development kernel features**)
for ext4::

  mkfs.ext4 -E packed_meta_blocks=1,discard -O bigalloc,extent,sparse_super2 -b 4096 -C 8192 /dev/somePartition

)

Test was conducted using a Seagate Archive v2 8 TB (ST8000AS0002) which is pretty much the only model available to consumers. Kernel is Linux 4.9.13.

With an ext4 formatted like above and current master branch ~165 MB/s write speed are achieved when processing large files. The drive works continuously, seeking rarely, so apparently Borg and ext4 work nicely here together such that data is written directly to their zones and not first to the buffer and then later to the final zone.

About once every ~30 minutes either ext4/jbd or the drive, not sure who, decides to do something resulting in a short IO stall where the drive performs a lot of seeking for <10 seconds with immediate return to normal afterwards. I assume that this is related to ext4's journal.

When the drive remains idle for some time (5-10 mins) it seems to flush the internal buffer. In this case it seems to be done after less than a minute.

Read performance in "borg check" is good (130-195 MB/s, about ~140 MB/s average).

------------------------------------------------------------------------------
Archive name: first-smr-archive
Archive fingerprint: dcdbe5451d2f9ce732eb1d16f627b1ab557985fa400ded9c404d391f23820990
Time (start): Sat, 2017-03-04 17:10:08
Time (end):   Sat, 2017-03-04 18:09:08
Duration: 58 minutes 59.09 seconds
Number of files: 2391681
Utilization of max. archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              475.86 GB            394.27 GB            335.03 GB
All archives:              475.86 GB            394.27 GB            335.03 GB

                       Unique chunks         Total chunks
Chunk index:                 1466326              2346071
------------------------------------------------------------------------------
2417.24user 293.71system 59:08.89elapsed 76%CPU (0avgtext+0avgdata 753364maxresident)k
927481056inputs+655248640outputs (0major+287976minor)pagefaults 0swaps

Average repo write speed is 93 MB/s or 132 MB/s if you compensate for deduplication.

@enkore enkore added the documentation label Mar 4, 2017

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 4, 2017

Maybe the last paragraph should be moved to the top of the entry. A lot of people run old distribution kernels and then they can stop reading after first paragraph and just buy a normal disk.

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Mar 5, 2017

1.0 (5 MB segments): first ~14 GB okay-ish (40 MB/s), then starts to stall, jbd (ext4 journal) and borg iowait increases a lot (90-99 %), disk starts to seek a lot. Average write speed is erratic and varies in the <10 MB/s area. After aborting + umounting FS and some idle time drive goes, acoustically speaking, nuts. Seems to flush it's buffer again. Edit: Took over an hour before the drive was done.

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Mar 5, 2017

1.1 on vanilla ext4: FS creation, mount, umount take some time. Performance seems to be almost identical to options given above (which does not mean that they don't make a difference - just not for Borg. This could be caused by Borg already being well-matched to the characteristics of these drives, so Borg doesn't need special FS options).

@enkore enkore added the bug label Mar 5, 2017

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 6, 2017

So, what's the bug?

@enkore enkore removed the bug label Mar 6, 2017

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Mar 6, 2017

Fat fingers

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented Mar 9, 2017

Here's a graph (made using JSON output + with timestamps + matplotlib + numpy gradient):

figure_1-1

(Note: different settings + source as in other tests, not directly comparable)

Blue = instantaneous numerical differentiation of transfer amount, therefore noisy as expected, orange = running linear average (nominal 1 minute window) over that, red = latency

Notice how the latency regularly spikes, sometimes to very large levels (around minute 80) and for longer times.

I'm not sure why it's a bit faster with less spread over the first ~240 GB.

The intervals where transfer rate = 0 but latency = no spike is where duplicates where in the source, nothing unusual.

latency := running difference between progress updates from borg minus the nominal rate, so anything that blocks borg will show up here - but there shouldn't be anything except the SMR drive causing it (source is a regular, idle drive, no network).

Edit: Fancier graph:

figure_1-2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment