Fast Dedup: DDT Prefetch

[allanjude]

Motivation and Context

After pool import, it takes a long time for a dedup table to naturally be loaded into the cache, which reduces performance until enough of the table is in memory.

In production we have seen significantly reduces performance for at least 72 hours on a large DDT, reduces to minutes using this feature.

Description

This change adds a new zpool prefetch -t ddt $pool command which causes a pool's DDT to be loaded into the ARC. The primary goal is to remove the need to "warm" a pool's cache before deduplication stops slowing write performance. It may also provide a way to reload portions of a DDT if they have been flushed due to inactivity.

Replaces #9464

How Has This Been Tested?

Tests added.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[don-brady]

Address review feedback

• Make ioct for prefetch generic so future types can be added
  

• Cleanup dmu_object_cached_size()
  

• Fix panic due to incorrect prefetch max value
  

• Rewrote dmu_prefetch_wait to avoid decompressing data during prefetch
  

[don-brady]

rebased after zap shrinking was landed

[shodanshok]

Maybe I missed something, but I have two questions:

• what happens if DDT overflows available ARC memory?
• can this be extended for generic pool metadata prefetch?

Thanks.

[amotin]

* what happens if DDT overflows available ARC memory?

Right now it will evict something prefetched earlier. Not very efficient...

* can this be extended for generic pool metadata prefetch?

As I pointed earlier, it should be done for BRT also, since they and their problems due to random access are very similar. "Genertic pool metadata" though is pretty vague, you should not want to prefetch all existing space maps, indirect blocks, extended attributes and other things you likely never need in a typical operation.

[robn]

[Fast dedup stack rebased to master 3c941d1]

[robn]

[Fast dedup stack rebased to master c98295e]

[tonyhutter]

@allanjude can you take a look at the checkstyle errors?

[allanjude]

@allanjude can you take a look at the checkstyle errors?

It was the file you had me remove, needed to be removed from the Makefile.am as well.
Should be good to go now.

[lundman]

Hellooo!

So this commit BSODs on Windows, it turns out to be:

https://github.com/KlaraSystems/zfs/blob/master/module/zfs/spa.c#L553
Here, err is not initialised, so Windows deem it worthy of $rand.
We skip the alloc as nvp is not NULL.

Before we get here, nvp is set to outnvl in the call to:
https://github.com/KlaraSystems/zfs/blob/master/module/zfs/zfs_ioctl.c#L7934

We leave somewhat early due to

 .... spa->spa_pool_props_object == 0)
		goto out;

and in out we do:

out:
	mutex_exit(&spa->spa_props_lock);
	dsl_pool_config_exit(dp, FTAG);
	if (err && err != ENOENT) {
		nvlist_free(*nvp);
		*nvp = NULL;
		return (err);

So nvp is now freed, we return error. We bubble up through:
https://github.com/KlaraSystems/zfs/blob/master/module/zfs/zfs_ioctl.c#L3051-L3057
here, nvp is free, but nothing in outnvl reflects that, so finally:

https://github.com/KlaraSystems/zfs/blob/master/module/zfs/zfs_ioctl.c#L7974

Boom, we are rogering around with the freed outnvl.

So, first fix is err = 0 naturally, but we should probably also handle the error case where we have freed nvp.

I could do a PR, but I have 332 commits to pull in still, so if someone gets one in before me, I will be happy :)

[robn]

@lundman wow, good find. I've got half a patch worked up already; I'll finish and post it tonight.