Do not wait on IO when attempting to grow caches

Growing cache can block on swap operations, which will force all
allocations to be emergency allocations until the swap operations have
finished. Each emergency allocation will wait a maximum of 1 second,
which means that we will spend a fairly substantial amount of time
waiting when growing blocks on swap operations and swap happens to be a
zvol. We disallow ___GFP_IO to prevent this.

At a glance, this looks safe both for systems without swap on zvols and
for systems without swap because we will try again when a failure
happens. Unfortunately, it also means that we can spin on memory for an
indeterminate amount of time. That will result in a deadlock-like state
when either the system lacks swap and nothing triggers the OOM-killer or
the system has swap and nothing else triggers routines to free memory by
either swapping or invoking the OOM-killer.

The deadlock-like state would happen without swap prior to this and will
likely happen with swap because of this. However, something else should
eventually run that will trigger the cleanup routines, breaking the
deadlock-like state. That means that we can spin for a rather long
period of time in the worse case. A counter probably could be used to
fix this. However, that should be a separate patch.

Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>

module/spl/spl-kmem.c

@@ -1830,7 +1830,7 @@ spl_cache_grow(spl_kmem_cache_t *skc, int flags, void **obj)
 
 		atomic_inc(&skc->skc_ref);
 		ska->ska_cache = skc;
-		ska->ska_flags = flags & ~__GFP_FS;
+		ska->ska_flags = flags & ~(__GFP_IO | __GFP_FS);
 		taskq_init_ent(&ska->ska_tqe);
 		taskq_dispatch_ent(spl_kmem_cache_taskq,
 		    spl_cache_grow_work, ska, 0, &ska->ska_tqe);