metamorphic: limit aggregate injected I/O latency

[jbowens]

Limit the amount of I/O latency injected into any individual run of the metamorphic test to improve worst-case test runtimes.

Informs #3681.
Informs #3733.

[cockroach-teamcity]

This change is 

[jbowens]

I'm unclear on how much of a contributor the I/O latency injection is to overall test time, but at least the failure in #3733 had it enabled. When target file sizes are small, or flush-split-bytes is low, or LBaseMaxBytes is low, it seems we can easily generate significant amounts of I/O.

Reviewable status: 0 of 6 files reviewed, all discussions resolved (waiting on @RaduBerinde)

[RaduBerinde]

Reviewable status: 0 of 6 files reviewed, 2 unresolved discussions (waiting on @jbowens)

---

vfs/errorfs/latency.go line 61 at r1 (raw file):

	mean  time.Duration
	limit time.Duration
	agg   time.Duration

this needs a comment

---

vfs/errorfs/latency.go line 88 at r1 (raw file):

	})
	time.Sleep(dur)
	rl.agg += dur

This needs to be an atomic or under a mutex, no?

[jbowens]

Reviewable status: 0 of 6 files reviewed, 2 unresolved discussions (waiting on @RaduBerinde)

---

vfs/errorfs/latency.go line 88 at r1 (raw file):

Previously, RaduBerinde wrote…

This needs to be an atomic or under a mutex, no?

Thanks, I need more sleep

[RaduBerinde]

Great idea.

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @jbowens)

[jbowens]

TFTR!