Improve zfs destroy performance with zio_t-free zio_free() #10034
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahrens
added
Status: Code Review Needed
pcd1193182
approved these changes
Feb 20, 2020
|
@pcd1193182 My first draft was simply |
sdimitro
approved these changes
Feb 20, 2020
Codecov Report
@@ Coverage Diff @@
## master #10034 +/- ##
==========================================
+ Coverage 79.19% 79.36% +0.16%
==========================================
Files 386 386
Lines 122448 122458 +10
==========================================
+ Hits 96977 97186 +209
+ Misses 25471 25272 -199
Continue to review full report at Codecov.
|
grwilson
reviewed
Feb 21, 2020
behlendorf
approved these changes
Feb 21, 2020
When "zfs destroy" is run, it completes quickly, and in the background we locate the blocks to free and free them. This background activity can be observed with `zpool get freeing` and `zpool wait -t free ...`. This background activity is processed by a single thread (the spa_sync thread) which calls zio_free() on each of the blocks to free. With even modest storage performance, the CPU consumption of zio_free() can be the performance bottleneck. Performance of zio_free() can be improved by not actually creating a zio_t in the common case (non-dedup, non-gang), instead calling metaslab_free() directly. This avoids the CPU cost of allocating the zio_t, and more importantly the cost of adding and later removing this zio_t from the parent zio's child list. The result is that performance of background freeing more than doubles, from 0.6 million blocks per second to 1.3 million blocks per second. Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
behlendorf
approved these changes
Feb 26, 2020
behlendorf
added
Status: Accepted
grwilson
approved these changes
Feb 28, 2020
ahrens
added a commit
to ahrens/zfs
that referenced
this pull request
Apr 29, 2020
…rformance with zio_t-free zio_free() When "zfs destroy" is run, it completes quickly, and in the background we locate the blocks to free and free them. This background activity can be observed with `zpool get freeing` and `zpool wait -t free ...`. This background activity is processed by a single thread (the spa_sync thread) which calls zio_free() on each of the blocks to free. With even modest storage performance, the CPU consumption of zio_free() can be the performance bottleneck. Performance of zio_free() can be improved by not actually creating a zio_t in the common case (non-dedup, non-gang), instead calling metaslab_free() directly. This avoids the CPU cost of allocating the zio_t, and more importantly the cost of adding and later removing this zio_t from the parent zio's child list. The result is that performance of background freeing more than doubles, from 0.6 million blocks per second to 1.3 million blocks per second. Reviewed-by: Paul Dagnelie <pcd@delphix.com> Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: George Wilson <gwilson@delphix.com> Signed-off-by: Matthew Ahrens <mahrens@delphix.com> Closes openzfs#10034
|
Perhaps an overly optimistic question, but is a similar optimization applicable to file deletion? That has an option to be applied lazily but can take quite a while. Perhaps not CPU-bound though, so not similar. |
|
@adamdmoss Yeah performance depends on a lot of things. As mentioned in the PR, the baseline (before this change) was 0.6 million blocks per second. So if you're getting much less than that, the bottleneck is likely elsewhere. |
jsai20
pushed a commit
to jsai20/zfs
that referenced
this pull request
Mar 30, 2021
When "zfs destroy" is run, it completes quickly, and in the background we locate the blocks to free and free them. This background activity can be observed with `zpool get freeing` and `zpool wait -t free ...`. This background activity is processed by a single thread (the spa_sync thread) which calls zio_free() on each of the blocks to free. With even modest storage performance, the CPU consumption of zio_free() can be the performance bottleneck. Performance of zio_free() can be improved by not actually creating a zio_t in the common case (non-dedup, non-gang), instead calling metaslab_free() directly. This avoids the CPU cost of allocating the zio_t, and more importantly the cost of adding and later removing this zio_t from the parent zio's child list. The result is that performance of background freeing more than doubles, from 0.6 million blocks per second to 1.3 million blocks per second. Reviewed-by: Paul Dagnelie <pcd@delphix.com> Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: George Wilson <gwilson@delphix.com> Signed-off-by: Matthew Ahrens <mahrens@delphix.com> Closes openzfs#10034
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
When "zfs destroy" is run, it completes quickly, and in the background
we locate the blocks to free and free them. This background activity
can be observed with
zpool get freeingandzpool wait -t free ....This background activity is processed by a single thread (the spa_sync
thread) which calls zio_free() on each of the blocks to free. With even
modest storage performance, the CPU consumption of zio_free() can be the
performance bottleneck.
Description
Performance of zio_free() can be improved by not actually creating a
zio_t in the common case (non-dedup, non-gang), instead calling
metaslab_free() directly. This avoids the CPU cost of allocating the
zio_t, and more importantly the cost of adding and later removing this
zio_t from the parent zio's child list.
The result is that performance of background freeing more than doubles,
from 0.6 million blocks per second to 1.3 million blocks per second.
How Has This Been Tested?
sudo zfs destroy -r domain0/recvd; time zpool wait domain0 1The filesystem is 1.14TiB, containing 536,823,751 blocks.
Baseline: 15m10s elapsed (1310MB/s, 589,000 blocks/sec)
With Changes: 6m45s elapsed (2951MB/s, 1,325,000 blocks/sec) (2.25x)
Types of changes
Checklist:
Signed-off-by.