Newer Illumos code

[ryao]

This depends on openzfs/spl#300. It supersedes #1696, #1760 and #1772.

It includes some fixes that I wrote when porting the changes. I split them out and placed them before the Illumos commits to make backporting them easier. This is not quite ready to merge yet, but it is ready for comments.

I ran git log usr/src/uts/common/fs/zfs and reviewed all commits made since the start of 2012 to obtain a list of commits that we were missing. In addition, I used git blame to identify the causes of merge conflicts, which identified some even older commits that we were missing. The following commits have been omitted for any of the following reasons:

• They add new feature flags and I would rather not include them in this pull request.
• They don't make sense on Linux.
• They require additional improvements to the Illumos code before they can be merged (e.g. nop write)
• They have already been fixed in ZFSOnLinux (e.g. LZ4 stack overflow, txg commit callbacks don't work)

commit 0713e232b7712cd27d99e1e935ebb8d5de61c57d
Author: George Wilson <george.wilson@delphix.com>
Date:   Tue Oct 1 13:25:53 2013 -0800

    4101 metaslab_debug should allow for fine-grained control
    4102 space_maps should store more information about themselves
    4103 space map object blocksize should be increased
    4104 ::spa_space no longer works
    4105 removing a mirrored log device results in a leaked object
    4106 asynchronously load metaslab
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Sebastien Roy <seb@delphix.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

commit cf746768a898264fa491791a8252865e5338dc67
Author: Bryan Cantrill <bryan@joyent.com>
Date:   Sat Aug 3 16:27:57 2013 -0700

    1974 pages_pp_maximum/lotsfree discrepancy leads to mlock() failure
    Reviewed by: Keith Wesolowski <keith.wesolowski@joyent.com>
    Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

commit 810e43b2eb0e320833671a403fdda51917e8b036
Author: Bill Pijewski <wdp@joyent.com>
Date:   Thu Aug 22 19:03:54 2013 +0000

    2932 support crash dumps to raidz, etc. pools
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Dan McDonald <danmcd@nexenta.com>

commit 18e649782fcf9da52d755b2074b4b9382426c818
Author: Marcel Telka <marcel.telka@nexenta.com>
Date:   Tue Aug 13 07:32:14 2013 +0200

    4039 zfs_rename()/zfs_link() needs stronger test for XDEV
    Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
    Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
    Reviewed by: Saso Kiselkov <skiselkov.ml@gmail.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Dan McDonald <danmcd@nexenta.com>

commit a5b577712a34346841d970e0827b4920ace408af
Author: Dan McDonald <danmcd@nexenta.com>
Date:   Wed Jul 10 10:58:11 2013 -0400

    348 ZFS should handle DKIOCGMEDIAINFOEXT failure
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

commit 3cb69f734bc60bbb4d56a28c83706db862bec082
Author: Will Andrews <will@firepipe.net>
Date:   Tue Jun 11 09:13:55 2013 -0800

    3749 zfs event processing should work on R/O root filesystems
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Eric Schrock <eric.schrock@delphix.com>
    Approved by: Christopher Siden <christopher.siden@delphix.com>

commit b3d9f2e26021d3f55a281af30720589d303b9806
Author: Will Andrews <willa@spectralogic.com>
Date:   Tue Jun 11 09:13:51 2013 -0800

    3747 txg commit callbacks don't work
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Eric Schrock <eric.schrock@delphix.com>
    Approved by: Christopher Siden <christopher.siden@delphix.com>

commit d8fa96c3021236b21f47b84036cbe4f5d4854a06
Author: Sa?o Kiselkov <skiselkov.ml@gmail.com>
Date:   Wed Apr 24 08:45:43 2013 -0800

    3705 stack overflow due to zfs lz4 compression
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Christopher Siden <christopher.siden@delphix.com>

commit d5ee8a1311accef11ec2057f70da38d1dd687088
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   Sun Apr 7 18:40:39 2013 -0800

    3654 zdb should print number of ganged blocks
    3655 ::rrwlock and ::refcount print errors on non-debug kernels
    3656 remove unused function zap_cursor_move_to_key()
    3657 mdb_ctf_vread() should allow conversion of pointer to uintptr_t
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Christopher Siden <christopher.siden@delphix.com>
    Reviewed by: Dan McDonald <danmcd@nexenta.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

commit c61ea5668634a215fd1c9000beb7f6d2853b3468
Author: George Wilson <george.wilson@delphix.com>
Date:   Wed Feb 20 22:50:16 2013 -0800

    3557 dumpvp_size is not updated correctly when a dump zvol's size is changed
    3558 setting the volsize on a dump device does not return back ENOSPC
    3559 setting a volsize larger than the space available sometimes succeeds
    3560 dumpadm should be able to remove a dump device
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Christopher Siden <christopher.siden@delphix.com>
    Approved by: Albert Lee <trisk@nexenta.com>

commit 80901aea8e78a2c20751f61f01bebd1d5b5c2ba5
Author: George Wilson <george.wilson@delphix.com>
Date:   Tue Nov 13 14:55:48 2012 -0800

    3236 zio nop-write
    Reviewed by: Matt Ahrens <matthew.ahrens@delphix.com>
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Christopher Siden <chris.siden@delphix.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

commit cd1c8b85eb30b568e9816221430c479ace7a559d
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   Thu Aug 30 05:13:49 2012 -0700

    3112 ztest does not honor ZFS_DEBUG
    3113 ztest should use watchpoints to protect frozen arc bufs
    3114 some leaked nvlists in zfsdev_ioctl
    3115 poll(2) returns prematurely in presence of spurious wakeups
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Matt Amdur <Matt.Amdur@delphix.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Christopher Siden <chris.siden@delphix.com>
    Approved by: Eric Schrock <eric.schrock@delphix.com>

commit b420f3adeb349714478d1a7813d2c0e069d41555
Author: Richard Lowe <richlowe@richlowe.net>
Date:   Wed Aug 1 16:40:39 2012 -0400

    backout 3006: causes 3046 (panics after mounting root)

commit 94dd93aee32d1616436eb51fb7b58094b9a8d3e8
Author: George Wilson <gwilson@delphix.com>
Date:   Sun May 6 18:49:02 2012 -0700

    1618 zfs causing system to hang in vmem_xalloc()
    Reviewed by: Eric Schrock <eric.schrock@delphix.com>
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Matt Ahrens <matt@delphix.com>
    Reviewed by: Richard Elling <richard.elling@richardelling.com>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Reviewed by: Steve Gonczi <gonczi@comcast.net>
    Approved by: Gordon Ross <gwr@nexenta.com>

commit c08b16378bf1d808956fc23b8c8ac30f3172a034
Author: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Date:   Sat Feb 18 18:33:56 2012 -0500

    2130 zvol DKIOCFREE uses nested DMU transactions
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Dan McDonald <danmcd@nexenta.com>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Reviewed by: Richard Elling <richard.elling@richardelling.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>    

commit 654b400c387942fc00d4f6869539adbd7b25fbce
Author: Joshua M. Clulow <josh@sysmgr.org>
Date:   Sat Feb 4 16:38:08 2012 +1100

    763 FMD msg URLs should refer to something visible
    Reviewed by: Richard Lowe <richlowe@richlowe.net>
    Reviewed by: Yuri Pankov <yuri.pankov@gmail.com>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Approved by: Richard Lowe <richlowe@richlowe.net>

[ryao]

As an additional note, I seem to have gotten the userland tools building with C99, but as illumos 4047 demonstrated, the kernel modules are still being built as C89.

[ryao]

I did a rebase to fix a mistake where I gave myself authorship on one of the illumos commits... now the history being shown by github is incorrect (although my own local history seems right). All of my commits are supposed to precede the Illumos commits.

[behlendorf]

@ryao This is great! I'll make the time to get you feedback on this, I'd love it if the 0.6.3 caught us largely up with Illumos.

[behlendorf]

@dweeezil Since you're familiar with this issues involved in bring in changes from Illumos it would be great if you could also review these patches.

[ryao]

I have refreshed the pull request with the following changes:

1. It has been rebased on today's ZFSOnLinux HEAD. No significant changes were needed to do this, but people who pull this branch for local testing will not be missing recent commits.
2. The commit messages have been reformatted to be consistent with existing commits taken from Illumos. Specifically, a reference section has been added and a short subject line has been prepended.
3. Illumos 4061 has been added to the list of commits. I had missed this because it was a change specific to userland.
4. An off-by-one error in my commit to change zfs_unmount_snap() that @prakashsurya caught has been fixed.
5. A commit to define MUTEX_SPIN in libzpool has been dropped. It crept into this from something else that I have in development.
6. The two commits to fix missing copyright notices have been merged into one with additional details added.
7. I fixed some typos in the commit messages that I introduced by mistake.

With that said, someone still needs to go through and revert modifications that occurred because ZFSOnLinux only called the constructors/destructors once per SLAB. @nedbass' work to port 4045 does this and a few other places seem to do it too. I have not touched this yet, but the corresponding SPL pull request includes the modifications required to support this.

[ryao]

A minor refresh. I just did a review of the output of git log usr/src/lib/libzfs, git log usr/src/common/zfs and git log $(ls ./usr/src/man/man*/z*) on the illumos-gate repository. I found some commits in userland that we were also missing. Specifically, I have added:

• Illumos 3745, 3811
• Illumos 3818
• Illumos 3909

Also, I found that my above list is missing the following two commits:

commit 71dbfc287ce623fa58f67b024d14ef6924fa1b2a
Author: Paul B. Henson <henson@acm.org>
Date:   Fri Oct 26 13:33:52 2012 -0700

    3254 add support in zfs for aclmode=restricted
    Reviewed by: Albert Lee <trisk@nexenta.com>
    Reviewed by: Gordon Ross <gwr@nexenta.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

commit a3c49ce110f325a563c245bedc4d533adddb7211
Author: Albert Lee <trisk@nexenta.com>
Date:   Sat May 14 00:29:13 2011 -0400

    742 Resurrect the ZFS "aclmode" property
    664 Umask masking "deny" ACL entries.
    279 Bug in the new ACL (post-PSARC/2010/029) semantics
    Reviewed by: Aram Hăvărneanu <aram@nexenta.com>
    Reviewed by: Gordon Ross <gwr@nexenta.com>
    Reviewed by: Robert Gordon <rbg@openrbg.com>
    Reviewed by: Mark.Maybee@oracle.com
    Approved by: Garrett D'Amore <garrett@nexenta.com>

These might fall under the "do not make sense on Linux" category given that we don't have ACL support working yet.

[dweeezil]

@behlendorf I took a cursory glance at this wad as soon as it was posted and was planning on giving it a more thorough look this weekend.

[dweeezil]

I've still not had much time to look at this yet (it's huge!) but after fiddling with it a bit, I came up with a couple of patches in dweeezil/zfs@1a91683; one to deal with a nvlist allocation and the other to deal with a duplicate preprocessor definition. I've been simultaneously looking the code and trying to run it. Another observation is that I can't unload the zfs.ko module (if anything nontrivial has been done subsequent to its loading) without reverting the recent constructor/destructor patch to spl. @ryao, is this to be expected at this point? The rmmod process blocks in D state but I've not had a chance to figure out why or where it's blocking.

FInally, I'll note that trying to do a zfs destroy <whatever>@snapshot always fails saying that the snapshot (or filesystem, I forgot which) can't be found.

Hopefully I'll have more time to look at this throughout the coming week.

[ryao]

@dweeezil You seem to have found a couple of regressions. I am not sure what caused the module unloading issue, but the zfs destroy dataset@snapshot issue is caused by a slight error in "Fix incorrect usage of strdup() in zfs_unmount_snap()" where I used strncpy instead of strlcpy. I will push a revised version soon.

[dweeezil]

@ryao I suspect you'll want to remove the extern declaration of zfs_txg_synctime_ms in dsl_pool.h as part of the 4045 patch.

[prakashsurya]

@dweeezil @nedbass Has been working on porting and testing the 4045 changes over. I'd have to ask him to be sure, but my hunch is that parameter will disappear when that patch lands. I don't think it makes sense anymore with 4045.

[dweeezil]

@kpande Did you also adjust the length parameter? See dweeezil/zfs@29b2094.

[dweeezil]

@kpande Try dweeezil/zfs@fdffb9e. The failure condition you discovered was a result of the illumos 3744 patch.

[ryao]

@kpande that would be a different bug. Thanks for letting us know about it.

[ryao]

I have pushed a refresh. Here are the changes.

1. The snapshot destruction regression has been squashed. It was caused by changes to zfs_unmount_snap() when porting 3744 zfs shouldn't ignore errors unmounting snapshots.
2. zfs_unmount_snap() now uses strlcpy() instead of strncpy(), which guarentees proper termination.
3. I expanded on the commit message for 'Add missing code to zfs_debug.{c,h}'.

I spent the better part of the past week trying to get a new workstation working (sadly, the hardware is not cooperating), so there is not as much here as I would have liked to have. Here is what is still left to do:

1. If possible, 'Add missing code to zfs_debug.{c,h} ' should be broken into the original Illumos commits for consistency.
2. The module unloading regression needs to be tackled. From what I am told, it is probably caused by the SPL changes.
3. Look into @kpande's issue.
4. Resolve the issue building the kernel modules as C99. Determine if we will be making a switch to C99 with this pull request.