Add zpool status -d (delays) and -p (parseable) #7756

tonyhutter · 2018-07-30T21:36:37Z

Description

This patch adds a new delays (-d) column to zpool status to show IO delays for VDEVs. It also adds a new parsable (-p) flag to display exact values.

 	NAME         STATE     READ WRITE CKSUM DELAY
 	testpool     ONLINE       0     0     0     -
	  mirror-0   ONLINE       0     0     0     -
 	    loop0    ONLINE       0     0     0    20
 	    loop1    ONLINE       0     0     0     0

Motivation and Context

We often see bad disks or bad SAS links throwing up a number of IO delay errors. This allows us to see the number of delays without manually counting them in zpool events.

Also: #6885

How Has This Been Tested?

Added test case

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the ZFS on Linux code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
All new and existing tests passed.
All commit messages are properly formatted and contain Signed-off-by.
Change has been approved by a ZFS on Linux member.

ahrens · 2018-07-31T17:42:07Z

This is showing the "delay" latency injected via zinject -D ms:lanes, right? Are you displaying the number of "lanes" (parallel i/os)?

ahrens · 2018-07-31T17:44:07Z

man/man8/zpool.8

@@ -2035,6 +2035,8 @@ output.  See the
 option of
 .Nm zpool Cm iostat
 for complete details.
+.It Fl d
+Display the number of leaf VDEV IO delays.


Assuming this is talking about zinject -D, I think we could improve this documentation, to refer to that feature and to describe the output more precisely. It isn't clear what "a delay" is, but we could say something like "i/o's are delayed such that they will take at least this many milliseonds to complete". We should also reference the zinject functionality.

tonyhutter · 2018-07-31T19:24:40Z

This is showing the "delay" latency injected via zinject -D ms:lanes, right?
Are you displaying the number of "lanes" (parallel i/os)?

It's actually showing the number of ZIOs that didn't complete due to a timeout. So any ZIOs that don't finish in zio_delay_max amount of time. I'll make it clearer in the documentation.

behlendorf · 2018-07-31T19:51:23Z

It's actually showing the number of ZIOs that didn't complete due to a timeout.

Actually these IO's might have completed successfully, they just took an unreasonably long time.

tonyhutter · 2018-07-31T19:54:29Z

@behlendorf thanks, yea that's a good point. I'll add it to the documentation.

tonyhutter · 2018-08-07T00:30:56Z

Documentation is updated in latest push.

behlendorf

I can see how this information would be useful. But I'm concerned that the mechanism for determining that an IO was delayed is currently too crude to be useful.

What if instead we used the existing latency histrograms to calculate a number of delayed IOs. For example, the total number of IOs which fall outside of 2 (or maybe 3) standard deviations. That would allow us to report a more meaningful delay count regardless of the underlying device performance. What we really want from this number is to be able to easily identify problematic drives which haven't failed in any way, but are not behaving consistently and are degrading overall pool performance.

behlendorf · 2018-08-16T23:33:42Z

cmd/zpool/zpool_main.c

+			VERIFY0(nvlist_lookup_nvlist(nv,
+			    ZPOOL_CONFIG_VDEV_STATS_EX, &nvx));
+			VERIFY0(nvlist_lookup_uint64(nvx,
+			    ZPOOL_CONFIG_VDEV_DELAYS, &delays));


You can use the fnvlist_lookup_nvlist() and fnvlist_lookup_uint64() here. We really shouldn't be calling VERIFY in the command line utilities.

cmd/zpool/zpool_main.c

module/zfs/zfs_fm.c

behlendorf · 2018-08-17T00:00:53Z

include/sys/fs/zfs.h

@@ -1029,6 +1032,8 @@ typedef struct vdev_stat_ex {
 	uint64_t vsx_agg_histo[ZIO_PRIORITY_NUM_QUEUEABLE]
 	    [VDEV_RQ_HISTO_BUCKETS];

+	uint64_t vsx_delays;


This can be moved in to vdev_stat_t since it's a single uint64_t which will pack nicely in to the nvlist uint64 array passed to user space.

tonyhutter · 2018-08-20T21:19:39Z

@behlendorf -

I can see how this information would be useful. But I'm concerned that the mechanism for
determining that an IO was delayed is currently too crude to be useful.

What if instead we used the existing latency histrograms to calculate a number of delayed IOs.

For reference, the current logic is "mark any IOs longer than zio_delay_max" as delayed, with
zio_delay_max defaulting to 30 seconds. Even at this crude default timeout, I've found the delay
event counts to be useful. I'll see the delay counts start spiking when hitting multipath
errors, for example.

I think we should get this patch in first using the existing zio_delay_max code. It's less
disruptive, and makes the test case really simple. Then, if we want to later change it to
work on standard deviations of latency, we'll have the working zpool status -d code already
in place to test it. I'm also a little hesitant to include the standard deviation code now,
since I don't know how it will it will work all hardware, especially on super low-latency devices
like 3d xpoint and nvdimm.

I'm working on fixing your other changes.

ahrens · 2018-08-29T19:47:19Z

I think it's confusing to call these "delays" or "delayed i/os". I think we should change the terminology to "slow i/os" or something similar (including renaming zio_delay_max and the other variables in the source code).

While we're at it, the units in which zio_delay_max are defined is a bit confusing:
#define ZIO_DELAY_MAX (30 * MILLISEC)
So the delay looks like 30 milliseconds, and there are no comments explaining otherwise, but you said above that it's actually 30 seconds. I think something like NSEC2MSEC(SEC2NSEC(30)) would make more sense (or better yet, define it in nanoseconds and use SEC2NSEC(30)).

tonyhutter · 2018-08-29T20:03:14Z

@behlendorf I incorporated all your changes and rebased
@ahrens I'm fine with the rename, I'll make the changes.

tonyhutter · 2018-08-29T20:09:46Z

@ahrens do you want me to rename the 'delay' event to 'slow_io' as well? Or keep it as 'delay' for compatibility?

ahrens · 2018-08-29T21:43:38Z

@tonyhutter I think slow_io would be a better name for the event but I'll leave it to you and Brian if you think it's OK to rename the existing user-visible event.

behlendorf

Since this event name is user visible renaming it to slow_io may to cause problems. For example, the ZEDs diagnosis engine currently ignores these events. Changing the event name would result in FMA cases being opened when there's a mismatched kernel/user space. Though I do like the name better, I think compatibility needs to win out this time.

Let's just update the zfs-events(5) man page to make it clear. While reading the man page I also noticed the zio_delay entry under PAYLOADS needs to be updated. The value returned in the payload is in nanoseconds not ticks.

#define ZIO_DELAY_MAX (30 * MILLISEC)

As for renaming the module option, changing it to zfs_slow_io_ms would be consistent with the others. There are 9 other module options expressed as milliseconds and non as nanoseconds. So let's leave this units here as-is, and simply redefine this internally as ZIO_SLOW_IO_MS 30000 which is consistent with the style used for the other module options.

I'm OK with refining what is considered a slow IO in a future patch.

cmd/zpool/zpool_main.c

behlendorf · 2018-08-29T22:22:33Z

tests/zfs-tests/tests/functional/Makefile.am

@@ -16,6 +16,7 @@ SUBDIRS = \
 	cp_files \
 	ctime \
 	deadman \
+	delays \


Let's not add another top level directory for this one test. Can you move it in to functional/cli_root/zpool_status/ with the other similar test cases.

That was the original plan, but I need to run my test as root, and functional/cli_root/zpool_status/* runs as a unprivileged user:

[tests/functional/cli_root/zpool_status] tests = ['zpool_status_001_pos', 'zpool_status_002_pos','zpool_status_003_pos', 'zpool_status_-c_disable', 'zpool_status_-c_homedir', 'zpool_status_-c_searchpath'] user = tags = ['functional', 'cli_root', 'zpool_status']

I see, that makes sense. What do you think about moving it to functional/fault/zpool_status_-d then?

tests/zfs-tests/tests/functional/delays/zpool_status_-d.ksh

behlendorf · 2018-08-29T22:29:55Z

tests/zfs-tests/tests/functional/delays/zpool_status_-d.ksh

+echo $OLD_DELAY > /sys/module/zfs/parameters/zio_delay_max
+
+DELAYS=$(zpool status -dp | grep "$DISK" | awk '{print $6}')
+if [ "$DELAYS" -gt "0" ] ; then


This isn't quite right, -gt won't work correctly when comparing strings instead of integers.

It works for me:

# ksh $ $ DELAYS=5 && if [ "$DELAYS" -gt "0" ] ; then echo "greater" ; fi greater $ DELAYS=5 && if [ "$DELAYS" -gt "6" ] ; then echo "greater" ; fi $

I'm fine with unquoting it though.

tonyhutter · 2018-10-18T16:16:10Z

Reviewers - can you please take another look at this? I think the test failures are unrelated.

Also, I decided to keep these named "delays" rather than "slow IOs" to not break compatibility. We log all zed events to splunk, and it would be a pain to have two event names for the same event. I'd also like to keep the naming consistent between events names and our variable names.

behlendorf · 2018-10-19T04:57:48Z

module/zfs/zfs_fm.c


-	zfs_ereport_start(&ereport, &detector, subclass, spa, vd,
+	rc = zfs_ereport_start(&ereport, &detector, subclass, spa, vd,
 	    zb, zio, stateoroffset, size);


It looks like this could result in 1 being returned since zfs_ereport_start() doesn't return errnos. How about:

int error = zfs_ereport_start(&ereport, &detector, subclass, spa, vd, zb, zio, stateoroffset, size); if (error) return (SET_ERROR(EINVAL));

behlendorf · 2018-10-19T05:00:25Z

module/zfs/zfs_fm.c


 	/* Cleanup is handled by the callback function */
-	zfs_zevent_post(ereport, detector, zfs_zevent_post_cb);
+	if (zfs_zevent_post(ereport, detector, zfs_zevent_post_cb) != 0)
+		return (EINVAL);


Rather than suppress the zfs_zevent_post() errno can we also return it.

behlendorf · 2018-10-19T05:11:08Z

module/zfs/zfs_fm.c

 #ifdef _KERNEL
 	nvlist_t *ereport = NULL;
 	nvlist_t *detector = NULL;

 	if (zfs_is_ratelimiting_event(subclass, vd))
-		return;
+		return (EBUSY);


It looks like SET_ERROR() should be used for some of these errnos.

behlendorf · 2018-10-19T05:13:55Z

module/zfs/zfs_fm.c

+/*
+ * Return 0 if the event was actually posted, return 1 if not.
+ */
+static int


Making the return type a boolean_t would be less misleading if we only care about success and failure. Otherwise with a return code of 0 I'd expect an errno to be returned on failure.

behlendorf · 2018-10-19T05:15:59Z

module/zfs/zfs_fm.c

 	zfs_ereport_start(&ereport, &detector, FM_EREPORT_ZFS_CHECKSUM,
 	    spa, vd, zb, zio, offset, length);

 	if (ereport == NULL)
-		return;
+		return (1);


Let's not mix errno return values with values like 1 (1 = EPERM).

codecov · 2018-10-23T08:30:06Z

Codecov Report

Merging #7756 into master will decrease coverage by 0.05%.
The diff coverage is 92.75%.

@@            Coverage Diff             @@
##           master    #7756      +/-   ##
==========================================
- Coverage   78.44%   78.38%   -0.06%     
==========================================
  Files         381      381              
  Lines      114550   114581      +31     
==========================================
- Hits        89854    89819      -35     
- Misses      24696    24762      +66

Flag	Coverage Δ
#kernel	`78.7% <89.13%> (-0.04%)`	⬇️
#user	`67.29% <76.31%> (+0.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9569292...793d569. Read the comment docs.

ahrens · 2018-10-23T20:28:02Z

man/man5/zfs-events.5

 \fBzio_delta\fR this does not include any vdev queuing time and is therefore
-solely a measure of the block layer performance.  On most modern Linux systems
-HZ is defined as 1000 making a tick equivalent to 1 millisecond.
+solely a measure of the block layer performance.


Do I understand correctly that you are actually not changing the meaning of the zio_delay payload, becuase it was in nanoseconds before, but the documentation was wrong so you're updating it?

That's correct. See comment: #7756 (review)

I'm being thrown off by the word required, I think because there's no concept of a requirement here. I think this would be more clear as something like The time elapsed (in nanoseconds) waiting for the block layer to complete the I/O.

ahrens · 2018-10-23T21:04:41Z

Also, I decided to keep these named "delays" rather than "slow IOs" to not break compatibility. We log all zed events to splunk, and it would be a pain to have two event names for the same event. I'd also like to keep the naming consistent between events names and our variable names.

The new functionality in zpool status elevates this concept to a much broader user base, compared to zed events. I would like to see us use a more precise name in zpool status and related documentation. Even if we need to retain a different event name for backwards compatibility.

The zfs-module-parameters manpage has a great example of the potential for confusion around the term delay:

       zfs_delay_min_dirty_percent (int)
                   Start to delay each transaction once there is  this  amount
                   of    dirty    data,   expressed   as   a   percentage   of
                   zfs_dirty_data_max. ...

       zfs_delay_scale (int)
                   This controls how quickly the transaction delay  approaches
                   infinity.   Larger  values  cause longer delays for a given
                   amount of dirty data.

                   See the section "ZFS TRANSACTION DELAY".

       zfs_delays_per_second (int)
                   Rate limit IO delay events to this many per second.
...
ZFS TRANSACTION DELAY
       We delay transactions when we've determined that  the  backend  storage
       isn't able to accommodate the rate of incoming writes.

One could easily misunderstand this as saying that we zfs_delays_per_second limits how frequently ZFS will "delay transactions". I realize we may be stuck with this term for the ZED events, but I think that many more people will interact with zpool status -d than with zed, so I think it's worth trying to contain this confusing terminology to only where it's really needed for backward compatibility.

tonyhutter · 2018-10-24T18:04:39Z

@ahrens I think you bring up a good point regarding slow IO 'delays' getting confused with TXG delays. I've pushed an additional commit that basically renames delays to slow IOs, with the exception of delay events. I also changed the flag from -d to -s for slow IOs. Can you let me know if this new rename commit is what you're looking for? If so, I'll squash it.

ahrens

Thanks for making the terminology change @tonyhutter!

ahrens · 2018-10-24T18:06:43Z

module/zfs/zio.c

@@ -77,7 +77,7 @@ uint64_t zio_buf_cache_allocs[SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT];
 uint64_t zio_buf_cache_frees[SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT];
 #endif

-int zio_delay_max = ZIO_DELAY_MAX;
+int zio_slow_io_ms = ZIO_DELAY_MAX;


should we also rename the macro?

ahrens · 2018-10-24T18:07:42Z

module/zfs/zio.c

-MODULE_PARM_DESC(zio_delay_max, "Max zio millisec delay before posting event");
+module_param(zio_slow_io_ms, int, 0644);
+MODULE_PARM_DESC(zio_slow_io_ms,
+	"Max zio completion time (milliseconds) before posting a delay event");


I'm assuming this also controls when we count a slow io in zpool status. Maybe we should also mention the zpool status impact, since that may impact more uses.

ahrens · 2018-10-24T18:10:43Z

module/zfs/vdev.c

-module_param(zfs_delays_per_second, uint, 0644);
-MODULE_PARM_DESC(zfs_delays_per_second, "Rate limit delay events to this many "
+module_param(zfs_slow_ios_per_second, uint, 0644);
+MODULE_PARM_DESC(zfs_slow_ios_per_second, "Rate limit delay events to this many "
 	"IO delays per second");


What would you think about renaming this to indicate that it's a limit on the events? e.g. zfs_slow_io_events_per_second_max? Or is that too verbose?

Presumably this does not limit the rate at which we count slow i/os for zpool status. I wonder if it would be helpful to clarify that anywhere?

ahrens · 2018-10-24T18:11:17Z

man/man5/zfs-events.5

@@ -95,7 +95,7 @@ information regarding "hung" I/O detection and configuration.
 .ad
 .RS 12n
 Issued when a completed I/O exceeds the maximum allowed time specified
-by the \fBzio_delay_max\fR module option.  This can be an indicator of
+by the \fBzio_slow_io_ms\fR module option.  This can be an indicator of
 problems with the underlying storage device.


should we also mention the rate limiting here?

ahrens · 2018-10-24T18:11:48Z

man/man5/zfs-module-parameters.5

-A zevent will be logged if a ZIO operation takes more than N milliseconds to
-complete. Note that this is only a logging facility, not a timeout on
+A delay zevent will be logged if a ZIO operation takes more than N milliseconds
+to complete. Note that this is only a logging facility, not a timeout on
 operations.


mention the zpool status -s implications too?

tonyhutter · 2018-10-25T21:26:09Z

@ahrens my latest commit has all your recommended fixes, plus a few other ones:

Rename test case from zpool_status_-d to zpool_status_-s to match the -d to -s rename.
Check for delay events in the test case.
Fix it so the slow_io counters still increment when delay events are being ratelimited.

behlendorf

Almost there, only a few comments. And I think the new zfs_ereport_is_valid() function helped clean this up a lot, good idea.

man/man5/zfs-module-parameters.5

module/zfs/zfs_fm.c

module/zfs/zio.c

cmd/zpool/zpool_main.c

man/man8/zpool.8

cmd/zpool/zpool_main.c

include/sys/spa.h

man/man8/zpool.8

module/zfs/vdev.c

module/zfs/zfs_fm.c

tests/zfs-tests/tests/functional/fault/zpool_status_-s.ksh

tonyhutter · 2018-11-01T18:34:34Z

@behlendorf thanks, those are really good clean-ups you suggested. I've fixed them in my latest push: 733a3ca

tonyhutter · 2018-11-01T22:07:20Z

I'll squash these commits after buildbot finishes

behlendorf · 2018-11-01T22:31:00Z

@tonyhutter sounds good, I'd suggest rebasing too. @ahrens this should be ready for another review.

ahrens · 2018-11-07T17:59:54Z

cmd/zpool/zpool_main.c

-		return (gettext("\tstatus [-c [script1,script2,...]] [-gLPvxD]"
-		    "[-T d|u] [pool] ... \n"
+		return (gettext("\tstatus [-c [script1,script2,...]] "
+		    "[-dgLpPvxD]  [-T d|u] [pool] ... \n"


ahrens · 2018-11-07T18:04:23Z

cmd/zpool/zpool_main.c

@@ -7138,6 +7164,9 @@ status_callback(zpool_handle_t *zhp, void *data)
 		    cbp->cb_namewidth, "NAME", "STATE", "READ", "WRITE",
 		    "CKSUM");

+		if (cbp->cb_print_slow_ios)
+			(void) printf(gettext("  SLOW"));


It would be easier to see that this is correctly aligned if we did " %5s", like we do with the other headers, and when printing the value

ahrens · 2018-11-07T18:05:44Z

include/sys/zio.h

@@ -161,7 +161,7 @@ enum zio_encrypt {
 /*
 * Default Linux timeout for a sd device.
 */
-#define	ZIO_DELAY_MAX			(30 * MILLISEC)
+#define	ZIO_SLOW_IO_MAX			(30 * MILLISEC)


Why do we use a macro for this, rather than initializing int zio_slow_io_ms = 30 * MILLISEC;? I'm not seeing the value in this additional indirection.

I think wherever 30 * MILLISEC appears, we'll want a comment saying that it's 30 seconds. (as opposed to assuming that 30 * MILLISEC means 30 milliseconds).

ahrens · 2018-11-07T18:10:46Z

man/man5/zfs-events.5

 \fBzio_delta\fR this does not include any vdev queuing time and is therefore
-solely a measure of the block layer performance.  On most modern Linux systems
-HZ is defined as 1000 making a tick equivalent to 1 millisecond.
+solely a measure of the block layer performance.


I'm being thrown off by the word required, I think because there's no concept of a requirement here. I think this would be more clear as something like The time elapsed (in nanoseconds) waiting for the block layer to complete the I/O.

ahrens · 2018-11-07T18:13:18Z

man/man5/zfs-module-parameters.5

-complete. Note that this is only a logging facility, not a timeout on
-operations.
+When an I/O operation takes more than \fBzio_slow_io_ms\fR milliseconds to
+complete is marked as a slow I/O.  Each slow I/O causes a delay zevent.  Slow


This talks about delay zevent, and zfs_slow_ios_per_second talks about slow I/O (delay) events. I think these are talking about the same thing, so we should use the same term in both places.

They're close to the same thing. "slow" refers to IOs, "delay" to the event. So an IO is marked as "slow" which in turn causes a "delay event". Technically zfs_slow_ios_per_second should be called zfs_delays_per_second, since it's rate-limiting the events, but we wanted to avoid confusing it with TXG delays, so I kept it as zfs_slow_ios_per_second.

I think both of these are referring to events (not the i/o itself). I would suggest you change the zfs_slow_ios_per_second description to say:

Rate limit delay zevents (which report slow i/os) to this many per second.

That way both places are using the term delay zevents to refer to the event.

That works, I'll make the change

ahrens · 2018-11-07T18:15:17Z

man/man8/zpool.8

@@ -2214,6 +2216,11 @@ Display a histogram of deduplication statistics, showing the allocated
 and referenced
 .Pq logically referenced in the pool
 block counts and sizes by reference count.
+.It Fl s
+Display the number of leaf VDEV slow IOs.  This is the number of IOs that
+didn't complete in \fBzio_slow_io_ms\fR milliseconds.  This does not


should we mention the default value here?

This patch adds a new slow I/Os (-s) column to zpool status to show the number of VDEV slow I/Os. This is the number of I/Os that didn't complete in zio_slow_io_ms milliseconds. It also adds a new parsable (-p) flag to display exact values. NAME STATE READ WRITE CKSUM SLOW testpool ONLINE 0 0 0 - mirror-0 ONLINE 0 0 0 - loop0 ONLINE 0 0 0 20 loop1 ONLINE 0 0 0 0 Signed-off-by: Tony Hutter <hutter2@llnl.gov> Closes: openzfs#6885

tonyhutter · 2018-11-07T23:27:59Z

Commits squashed and rebased. I also changed the commit message to say "slow IOs" instead of "delays".

This patch adds a new slow I/Os (-s) column to zpool status to show the number of VDEV slow I/Os. This is the number of I/Os that didn't complete in zio_slow_io_ms milliseconds. It also adds a new parsable (-p) flag to display exact values. NAME STATE READ WRITE CKSUM SLOW testpool ONLINE 0 0 0 - mirror-0 ONLINE 0 0 0 - loop0 ONLINE 0 0 0 20 loop1 ONLINE 0 0 0 0 Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Tony Hutter <hutter2@llnl.gov> Closes openzfs#7756 Closes openzfs#6885

ahrens reviewed Jul 31, 2018

View reviewed changes

tonyhutter force-pushed the delays2 branch 2 times, most recently from 7307f13 to 9a42494 Compare July 31, 2018 19:50

tonyhutter force-pushed the delays2 branch 3 times, most recently from b65e263 to 9960b16 Compare August 2, 2018 20:37

behlendorf reviewed Aug 17, 2018

View reviewed changes

behlendorf added the Status: Work in Progress Not yet ready for general review label Aug 19, 2018

tonyhutter force-pushed the delays2 branch 2 times, most recently from 58d325e to c9008f9 Compare August 29, 2018 00:16

behlendorf reviewed Aug 29, 2018

View reviewed changes

tonyhutter mentioned this pull request Sep 12, 2018

Feature request: Detect and warn slow disk performance #6885

Closed

tonyhutter force-pushed the delays2 branch 2 times, most recently from 8c76973 to c1ed0b9 Compare September 25, 2018 18:27

behlendorf added Status: Code Review Needed Ready for review and testing and removed Status: Work in Progress Not yet ready for general review labels Sep 25, 2018

tonyhutter force-pushed the delays2 branch 2 times, most recently from 7b1ec46 to 10b8498 Compare September 27, 2018 19:17

tonyhutter force-pushed the delays2 branch 2 times, most recently from f409b94 to fa1a953 Compare October 4, 2018 16:57

tonyhutter force-pushed the delays2 branch from a798b1a to 95983dd Compare October 9, 2018 17:46

tonyhutter force-pushed the delays2 branch from 95983dd to 676d7df Compare October 17, 2018 20:25

behlendorf requested changes Oct 19, 2018

View reviewed changes

tonyhutter force-pushed the delays2 branch from 676d7df to aa7f095 Compare October 22, 2018 22:04

tonyhutter force-pushed the delays2 branch from aa7f095 to efd4a80 Compare October 23, 2018 18:12

ahrens reviewed Oct 23, 2018

View reviewed changes

ahrens reviewed Oct 24, 2018

View reviewed changes

tonyhutter force-pushed the delays2 branch from e81e02e to f49062b Compare October 25, 2018 21:39

behlendorf approved these changes Oct 30, 2018

View reviewed changes

man/man5/zfs-module-parameters.5 Outdated Show resolved Hide resolved

module/zfs/zfs_fm.c Outdated Show resolved Hide resolved

module/zfs/zfs_fm.c Show resolved Hide resolved

module/zfs/zio.c Outdated Show resolved Hide resolved

behlendorf approved these changes Oct 30, 2018

View reviewed changes

behlendorf approved these changes Oct 31, 2018

View reviewed changes

behlendorf approved these changes Nov 1, 2018

View reviewed changes

ahrens reviewed Nov 7, 2018

View reviewed changes

tonyhutter force-pushed the delays2 branch from f218610 to 194a7f5 Compare November 7, 2018 21:09

ahrens approved these changes Nov 7, 2018

View reviewed changes

tonyhutter force-pushed the delays2 branch from 194a7f5 to 793d569 Compare November 7, 2018 23:26

behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Nov 9, 2018

behlendorf merged commit ad796b8 into openzfs:master Nov 9, 2018

Add zpool status -d (delays) and -p (parseable) #7756

Add zpool status -d (delays) and -p (parseable) #7756

Conversation

tonyhutter commented Jul 30, 2018 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

ahrens commented Jul 31, 2018

Choose a reason for hiding this comment

tonyhutter commented Jul 31, 2018

behlendorf commented Jul 31, 2018

tonyhutter commented Jul 31, 2018

tonyhutter commented Aug 7, 2018

behlendorf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyhutter commented Aug 20, 2018

ahrens commented Aug 29, 2018 • edited Loading

tonyhutter commented Aug 29, 2018

tonyhutter commented Aug 29, 2018

ahrens commented Aug 29, 2018

behlendorf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyhutter commented Oct 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 23, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrens commented Oct 23, 2018

tonyhutter commented Oct 24, 2018

ahrens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyhutter commented Oct 25, 2018

behlendorf left a comment

Choose a reason for hiding this comment

tonyhutter commented Nov 1, 2018

tonyhutter commented Nov 1, 2018

behlendorf commented Nov 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyhutter commented Nov 7, 2018

tonyhutter commented Jul 30, 2018 •

edited

Loading

ahrens commented Aug 29, 2018 •

edited

Loading

codecov bot commented Oct 23, 2018 •

edited

Loading