Fix 'zpool add' handling of nested interior VDEVs

[loli10K]

Description

When replacing a faulted device which was previously handled by a spare multiple levels of nested interior VDEVs will be present in the pool configuration: this is safe and allowed, but get_replication() needs to handle this situation gracefully to let zpool add new devices to the pool otherwise we will suffer the following failure:

root@linux:~# zpool status
  pool: testpool
 state: DEGRADED
  scan: resilvered 20.1M in 0 days 00:00:07 with 0 errors on Tue Dec 26 18:47:37 2017
config:

	NAME                         STATE     READ WRITE CKSUM
	testpool                     DEGRADED     0     0     0
	  mirror-0                   DEGRADED     0     0     0
	    /var/tmp/file-vdev1      ONLINE       0     0     0
	    spare-1                  DEGRADED     0     0     0
	      replacing-0            DEGRADED     0     0     0
	        /var/tmp/file-vdev2  UNAVAIL      0     0     0  cannot open
	        /var/tmp/file-vdev3  ONLINE       0     0     0
	      /var/tmp/file-spare1   ONLINE       0     0     0
	spares
	  /var/tmp/file-spare1       INUSE     currently in use

errors: No known data errors
root@linux:~# zpool add $POOLNAME spare $TMPDIR/file-spare2
nvlist_lookup_string(cnv, ZPOOL_CONFIG_PATH, &path) == 0
ASSERT at zpool_vdev.c:884:get_replication()Aborted (core dumped)

Motivation and Context

Fix #6678

How Has This Been Tested?

Tested on local builder, script added to the ZTS; zpool add is now able to handle the following situation:

root@linux:/usr/src/zfs# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 20.2M in 0 days 00:00:00 with 0 errors on Tue Dec 26 18:41:38 2017
config:

	NAME                                                STATE     READ WRITE CKSUM
	testpool                                            DEGRADED     0     0     0
	  mirror-0                                          DEGRADED     0     0     0
	    /var/tmp/file-vdev1                             ONLINE       0     0     0
	    spare-1                                         DEGRADED   261     0     0
	      replacing-0                                   DEGRADED     0     0     0
	        /var/tmp/file-vdev2                         UNAVAIL      0     0     0  cannot open
	        spare-1                                     DEGRADED     0     0     0
	          replacing-0                               DEGRADED     0     0     0
	            /var/tmp/file-vdev3                     UNAVAIL      0     0     0  cannot open
	            spare-1                                 DEGRADED     0     0     0
	              replacing-0                           DEGRADED     0     0     0
	                /var/tmp/file-vdev4                 UNAVAIL      0     0     0  cannot open
	                spare-1                             DEGRADED     0     0     0
	                  replacing-0                       DEGRADED     0     0     0
	                    /var/tmp/file-vdev5             UNAVAIL      0     0     0  cannot open
	                    spare-1                         DEGRADED     0     0     0
	                      replacing-0                   DEGRADED     0     0     0
	                        /var/tmp/file-vdev6         UNAVAIL      0     0     0  cannot open
	                        spare-1                     DEGRADED     0     0     0
	                          /var/tmp/file-vdev7       UNAVAIL      0     0     0  cannot open
	                          /var/tmp/file-spare6      ONLINE       0     0     0
	                        spare-2                     DEGRADED     0     0     0
	                          replacing-0               DEGRADED     0     0     0
	                            /var/tmp/file-vdev8     UNAVAIL      0     0     0  cannot open
	                            spare-1                 DEGRADED     0     0     0
	                              /var/tmp/file-vdev9   UNAVAIL      0     0     0  cannot open
	                              /var/tmp/file-spare8  ONLINE       0     0     0
	                            /var/tmp/file-vdev10    ONLINE       0     0     0
	                          /var/tmp/file-spare7      ONLINE       0     0     0
	                      /var/tmp/file-spare5          ONLINE       0     0     0
	                  /var/tmp/file-spare4              ONLINE       0     0     0
	              /var/tmp/file-spare3                  ONLINE       0     0     0
	          /var/tmp/file-spare2                      ONLINE       0     0     0
	      /var/tmp/file-spare1                          ONLINE       0     0   261
	spares
	  /var/tmp/file-spare1                              INUSE     currently in use
	  /var/tmp/file-spare2                              INUSE     currently in use
	  /var/tmp/file-spare3                              INUSE     currently in use
	  /var/tmp/file-spare4                              INUSE     currently in use
	  /var/tmp/file-spare5                              INUSE     currently in use
	  /var/tmp/file-spare6                              INUSE     currently in use
	  /var/tmp/file-spare7                              INUSE     currently in use
	  /var/tmp/file-spare8                              INUSE     currently in use
	  /var/tmp/file-spare9                              AVAIL   

errors: No known data errors
root@linux:/usr/src/zfs# truncate -s 512m $TMPDIR/file-spare-new
root@linux:/usr/src/zfs# zpool add $POOLNAME spare $TMPDIR/file-spare-new
root@linux:/usr/src/zfs# echo $?
0
root@linux:/usr/src/zfs# 

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.
• Change has been approved by a ZFS on Linux member.

[behlendorf]

LGTM, nice fix!

[behlendorf]

s/create/creates

[codecov]

Codecov Report

Merging #6996 into master will decrease coverage by 0.18%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #6996      +/-   ##
=========================================
- Coverage   75.49%   75.3%   -0.19%     
=========================================
  Files         296     296              
  Lines       95500   95500              
=========================================
- Hits        72095   71915     -180     
- Misses      23405   23585     +180

Flag	Coverage Δ
#kernel	74.47% <ø> (-0.39%)	⬇️
#user	67.46% <100%> (-0.41%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 823d48b...1e7caa4. Read the comment docs.