Clarify 'zpool remove' restrictions

[behlendorf]

Motivation and Context

Make the documentation clear about the restrictions of device removal. #7880

Description

Update zpool(8) to clarify what type of vdevs may be safely removed and that the existence of any top-level raidz device which is part of the primary pool will prevent device removal.

How Has This Been Tested?

Proofread.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.
• Change has been approved by a ZFS on Linux member.

[DeHackEd]

While I don't condone non-redundant pools, disks in a pure RAID-0 configuration can also be removed.

[behlendorf]

Good point, refreshed and slightly reworded.

     zpool remove [-np] pool device...
             Removes the specified device from the pool.  This command supports
             removing hot spare, cache, log, and both mirrored and non-redundant
             primary top-level vdevs, this includes dedup and special vdevs.  When
             the primary pool storage includes a top-level raidz vdev only hot
             spare, cache, and log devices can be removed.

[dweeezil]

As long as this is being clarified, I think it might be a good idea to refer to the device_removal feature by name:

$ diff -u zpool.8 zpool.8.mod
--- zpool.8	2018-09-13 14:15:02.443896936 -0500
+++ zpool.8.mod	2018-09-13 14:21:56.936155588 -0500
@@ -1940,8 +1940,10 @@
 the background.
 The removal progress can be monitored with
 .Nm zpool Cm status.
-This feature must be enabled to be used, see
-.Xr zpool-features 5
+The
+.Sy device_removal
+feature flag must be enabled to remove a top-level vdev, see
+.Xr zpool-features 5 .
 .Pp
 A mirrored top-level device (log or data) can be removed by specifying the top-level mirror for the
 same.

Even some more additional text might be prudent to make it clear that removing a top-level vdev has compatibility ramifications as opposed to removing a log or cache device.

[Lady-Galadriel]

LGTM, and @dweeezil's comment about restrictions is probably valid as well.

[behlendorf]

Refreshed to include @dweeezil's additions.

[ahrens]

I think it would flow better with "this includes" -> "including"

[behlendorf]

Sure, that flows a little better.

[ahrens]

I get this, but IMO it would be more easily understood to say When the primary pool storage includes a raidz vdev, other top-level vdevs may not be removed.

[behlendorf]

I considered a wording very similar, but decided against it in favor of this clunkier more precise wording. My concern was that it implied spare, cache, and log devices, which look like top-level vdevs in the zpool status output, could not be removed. But I'd love a better way to phase this.