Fast Clone Deletion

[shartse]

Video of talk at BSDCan

Note: I've updated the deletion performance figures and test script since this presentation as they weren't doing quite what I thought they were. Effectively, I was passing the stride option in dd in bytes instead of blocks. This meant I was writing more than I thought to the clone and in the sparse case I was writing off the end of the file. The old tests still demonstrated meaningful performance differences, just in a less realistic scenario.

Motivation and Context

Deleting zfs clones requires determining which parts of the block tree belong to the clone and which are still part of the original snapshot. The basic algorithm is to traverse the block tree, at each block checking when it was born. If it was created after the clone was created then free it and recursively check its child blocks. If the block was born before the clone was created then we know it is part of the snapshot. Otherwise, continue traversing the block tree.

This algorithm is non ideal in the case of sparsely modified clones (those with a small number of scattered writes). Even though there is very little written to the clones it takes a long time to scan through the block tree, i.e. deletion time for a clone with sparse writes is proportional to the size of the underlying block tree, which is the size snapshot. Our goal to change this be to proportional to the space used by the clone only.

The following figure shows the performance difference between deleting clones with with the same amount of modification, just with contiguous vs. sparsely distributed writes.

The script I wrote for this test is available here: lx_fast_delete

Description

Livelist Algorithm

Livelists use the same deadlist_t struct as Deadlists and are also used to track block pointers over the lifetime of a dataset. Livelists however, belong to clones and track the block pointers that are clone-specific (were born after the clone's creation). The exception is embedded block pointers which are not included in Livelists because they do not need to be freed.

When it comes time to delete the clone, the livelist provides a quick reference as to what needs to be freed. For this reason, livelists also track when clone-specific block pointers are freed before deletion to prevent double frees. Each block pointers in a livelist is marked as a FREE or an ALLOC and the deletion algorithm iterates backwards over the livelist, matching FREE/ALLOC pairs and then freeing those ALLOCs which remain. Livelists are also updated in the case when block pointers are remapped during removal: the old version of the blkptr is cancelled out with a FREE and the new version is tracked with an ALLOC.

Sublists

To bound the amount of memory required for deletion, livelists over a certain size are spread over multiple entries (made natural by the use of the deadlist_t data structure). . Entries are grouped by birth txg so we can be sure the ALLOC/FREE pair for a given block pointer will be in the same entry. This allows us to delete livelists incrementally over multiple syncs, one entry at a time. The threshold size at which we create a new sub-livelist is an important tunable for livelist performance - we had to balance the fact that larger sublists mean fewer sublists (decreasing the cost of insertion) against the consideration that sublists will be loaded into memory and shouldn't take up an inordinate amount of space. We settled on ~500000 entries, corresponding to roughly 128M

Condensing

We still have the issue that the livelist can grow arbitrarily large - continually gaining sub-livelists. However, because of the restriction that block pointers of the same txg must reside in the same sub-livelist, we can determine if a given block has been freed (in time proportional to the size of the sub-livelist instead of the total number of frees and allocs). This means we can periodically clear out ALLOC/FREE pairs from sublists and then merge the smaller lists together.

Disabling

Livelists are not always the most effective deletion method. We can approximate how much of a performance gain a livelist will give us based on the pecentage of blocks shared between the clone and its origin. Zero percent shared means that the clone has completely diverged and that the old method is maximally effective: every read from the block tree will result in lots of frees. Livelists give us gains when they track blocks scattered across the tree, when one read in the old method might only result in a few frees. Once the clone has been overwritten enough, writes are no longer sparse and we'll no longer get much of a benefit from tracking them with a livelist. We chose a lower limit of 75 percent shared (25 percent overwritten). Once the amount of shared space drops below this threshold, the clone will revert to the old deletion method.

Testing

Performance Results

Deletion

I repeated the first test discussed above for deleting sparsely written clones and found a dramatic performance improvement using the new algorithm. As the above figure shows, the old deletion time grows linearly with respect to the size of the snapshot, while the new deletion time grows at a much smaller rate. In the following figure, I compare the new deletion time to the size of the data written to the clone and we can see that it's now proportional to that.

I also examined performance changes to the old algorithm's "best case" scenario - that of contiguously overwritten clones. I wasn't expecting to be a performance improvement with the livelist strategy since the old method was already very efficient (because of the way the overwritten block pointers laid out in the block tree) and we can see from the figure below that the livelist does add some overhead. However, since the deletion time is still proportional to the amount of data written and real world scenarios are likely to be a hybrid of sparse and contiguous write patterns, I'm not concerned about this performance change.

Writes

One of the main drawbacks of the livelist deletion algorithm is the way that it imposes extra work done at write-time to track the modified block pointers for clones. I explored possible write performance degradation by constructing a "worst case" scenario for writes. This scenario is one in which a write modifying blocks tracked in many different livelist sublists and a FREE entry has to be appended to each one.

1. Create a snapshot of a large filesystem and a clone of that snapshot
2. Write X bytes of data in N different chunks to the clone such that each chunk is separated into it's own sublist.
3. Overwrite the modified region with a small write Y but spread evenly into N piece such that each touches a different sublist.
4. Measure the time the write in 3 took, including the sync time which is when the block pointers are flushed to the livelist.

The script I used for this test is available here: lx_write_test

The following figure shows the results from repeated experiments with N = 50, N = 1 and N = 0 (the old algorithm that doesn't use livelists). We can see that a larger number of sublists does result in degraded write performance. This motivates the reasoning for having the sublist size be as large as is reasonable, given memory constraints, as well as the condensing of sublists and disabling of livelists altogether once the clone has been overwritten past a certain point.

Note: the figure shows the average time of 1000 writes and the standard deviation.

Correctness

I added zfs tests to cover basic assumptions about livelists and well as edge cases for deletion, condensing and disabling.

This Illumos implementation feature has been active internally at Delphix for the past 10 months.

I've been running these changes through zloop for several days without hitting anything related to this.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.

[kithrup]

I'd seen the presentation about this before, so it was nice to see the code that went along with it. I don't have situations where I have lots of clones to delete frequently, but I can see the need for it. This looks like a marked improvement there.

Nothing obvious leaped out at me as problematical. I'll probably try to go over the changes again, but in the meanwhile it looks good.

[shartse]

Thanks for taking a look! I'll go through you comments soon.

[behlendorf]

@shartse can you please rebase this on master now that redacted send/recv has been merged.

[behlendorf]

Thanks for addressing the review feedback. Rebasing this one last time on master should resolve the kernel.org build failure. The cstyle checker also caught a few minor things. Then this looks good to me!

[shartse]

@behlendorf - Great thanks for you feedback! I updated this review with one more fix to address a race condition we found recently. It'd be great if you could take a look at that as well and let me know if you have any concerns.

[codecov]

Codecov Report

Merging #8416 into master will increase coverage by 0.24%.
The diff coverage is 92.72%.

@@            Coverage Diff             @@
##           master    #8416      +/-   ##
==========================================
+ Coverage   78.78%   79.02%   +0.24%     
==========================================
  Files         400      400              
  Lines      121004   121645     +641     
==========================================
+ Hits        95328    96130     +802     
+ Misses      25676    25515     -161

Flag	Coverage Δ
#kernel	79.64% <95.95%> (+0.07%)	⬆️
#user	66.59% <67.06%> (+0.07%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d274ac5...ee94c36. Read the comment docs.

[rrygl]

Can someone look at this corruption bug?
#11480
It's believed to be caused by this new feature. It's been there since Jan and no solution yet.