mutex: force serialization on mutex_exit() to fix races

[tuxoko]

It is known that mutex in Linux is not safe when using it to synchronize
the freeing of object in which the mutex is embedded:
http://lwn.net/Articles/575477/

The known places in ZFS which are suspected to suffer from the race condition
are zio->io_lock and dbuf->db_mtx:
zio uses zio->io_lock and zio->io_cv to synchronize freeing between zio_wait()
and zio_done().
dbuf uses dbuf->db_mtx to protect reference counting.

Add it's been reported multiple time that zio->io_lock and dbuf->db_mtx suffer
from corruption, which is exactly the symptom of the race.
openzfs/zfs#2523
openzfs/zfs#2897

This patch fix this kind of race by forcing serializaion on mutex_exit() with
a spinlock, making the mutex safer by sacrificing a bit of performance and
memory overhead.

Signed-off-by: Chunwei Chen tuxoko@gmail.com

[ryao]

This looks good to me.

[tuxoko]

Update: add comment.

[behlendorf]

@tuxoko Thanks for update (and fix). I've merged this patch which should resolve a whole slew of open issues. I've tried to list most of them in the commit message but I'm confident I missed several (likely many).

@ryao @dajhorn @FransUrbo I don't usually recommend cherry picking fixes between tags. But if your going to update your distributions packages I'd strongly suggest cherry picking this fix prior to the 0.6.4 tag. It's small, safe, and address the most commonly reported issue.