Some improvements to metaslabs eviction

[amotin]

Motivation and Context

Quite often analyzing dbgmsg logs from different systems I see repeating metaslabs load/unload cycles during period of low activity. It seems to be a waste of time.

Description

• Add old eviction for special and dedup metaslab classes. Those vdevs may be potentially big and fragmented with large metaslabs, while their asynchronous write pattern is not really different from normal class. It seems an omission to not evict old metaslabs from them.
• If we have metaslab preload enabled, which means we are not too low on memory, do not evict active metaslabs even if they are not used for some time. Eviction of active metaslabs means we won't be able to write anything until we load them, that may take some time, that is directly opposite to metaslab preload goals. For small systems the memory saving should be less important after recent reduction in number of allocators and so active metaslabs.

How Has This Been Tested?

Without the patch on almost idle system I see ZFS loading and then unloading about ~18 metaslabs for each single activity spike (~4x2 active and 10 preload metaslabs). With the patch I see load/unload cycle only covering the 10 preload metaslabs, which while still may be a waste, should not affect payload performance.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

Makes sense. We could potentially also check arc_reclaim_needed() here and allow the metaslab to be evicted if memory is tight and metaslab_preload_enabled is set. That said, I suspect it would only matter on very low memory machines so I'm good with this.