Permalink
Switch branches/tags
Commits on Oct 31, 2018
  1. Minor corrections to manpages

    lundman committed Oct 31, 2018
Commits on Oct 23, 2018
  1. vnop_fsync should skip zil_commit sometimes

    lundman committed Oct 23, 2018
    When we unlink files, we call vnode_recycle() and zil_commit, and
    XNU's VFS layer before calling vnop_reclaim() will also call vnop_fsync().
    This means all delete files are zil_commit()ed twice.
Commits on Oct 17, 2018
  1. Rename lua panic to avoid clash

    lundman committed Oct 17, 2018
    OSX panic is a #define making for some fun times.
  2. Tag zfs-1.8.1

    lundman committed Oct 17, 2018
  3. ZFS init in thread

    lundman committed Oct 17, 2018
    Wait for SPL to have loaded first.
Commits on Oct 11, 2018
  1. OpenZFS 7431 - ZFS Channel Programs

    cwill authored and lundman committed Feb 8, 2018
    Authored by: Chris Williamson <chris.williamson@delphix.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: John Kennedy <john.kennedy@delphix.com>
    Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
    Approved by: Garrett D'Amore <garrett@damore.org>
    Ported-by: Don Brady <don.brady@delphix.com>
    Ported-by: John Kennedy <john.kennedy@delphix.com>
    
    OpenZFS-issue: https://www.illumos.org/issues/7431
    OpenZFS-commit: openzfs/openzfs@dfc1153
    
    Porting Notes:
    * The CLI long option arguments for '-t' and '-m' don't parse on linux
    * Switched from kmem_alloc to vmem_alloc in zcp_lua_alloc
    * Lua implementation is built as its own module (zlua.ko)
    * Lua headers consumed directly by zfs code moved to 'include/sys/lua/'
    * There is no native setjmp/longjump available in stock Linux kernel.
      Brought over implementations from illumos and FreeBSD
    * The get_temporary_prop() was adapted due to VFS platform differences
    * Use of inline functions in lua parser to reduce stack usage per C call
    * Skip some ZFS Test Suite ZCP tests on sparc64 to avoid stack overflow
    
    OpenZFS 8605 - zfs channel programs fix zfs.exists
    
    Authored by: Chris Williamson <chris.williamson@delphix.com>
    Reviewed by: Paul Dagnelie <pcd@delphix.com>
    Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Approved by: Robert Mustacchi <rm@joyent.com>
    Ported-by: Don Brady <don.brady@delphix.com>
    
    zfs.exists() in channel programs doesn't return any result, and should
    have a man page entry. This patch corrects zfs.exists so that it
    returns a value indicating if the dataset exists or not. It also adds
    documentation about it in the man page.
    
    OpenZFS-issue: https://www.illumos.org/issues/8605
    OpenZFS-commit: openzfs/openzfs@1e85e11
    
    OpenZFS 8592 - ZFS channel programs - rollback
    
    Authored by: Brad Lewis <brad.lewis@delphix.com>
    Reviewed by: Chris Williamson <chris.williamson@delphix.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Robert Mustacchi <rm@joyent.com>
    Ported-by: Don Brady <don.brady@delphix.com>
    
    ZFS channel programs should be able to perform a rollback.
    
    OpenZFS-issue: https://www.illumos.org/issues/8592
    OpenZFS-commit: openzfs/openzfs@d46b5ed
    
    OpenZFS 8600 - ZFS channel programs - snapshot
    
    Authored by: Chris Williamson <chris.williamson@delphix.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: John Kennedy <john.kennedy@delphix.com>
    Reviewed by: Brad Lewis <brad.lewis@delphix.com>
    Approved by: Robert Mustacchi <rm@joyent.com>
    Ported-by: Don Brady <don.brady@delphix.com>
    
    ZFS channel programs should be able to create snapshots.
    In addition to the base snapshot functionality, this entails extra
    logic to handle edge cases which were formerly not possible, such as
    creating then destroying a snapshot in the same transaction sync.
    
    OpenZFS-issue: https://www.illumos.org/issues/8600
    OpenZFS-commit: openzfs/openzfs@68089b8
    
    OpenZFS 8677 - Open-Context Channel Programs
    
    Authored by: Serapheim Dimitropoulos <serapheim@delphix.com>
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Reviewed by: Chris Williamson <chris.williamson@delphix.com>
    Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
    Approved by: Robert Mustacchi <rm@joyent.com>
    Ported-by: Don Brady <don.brady@delphix.com>
    
    We want to be able to run channel programs outside of synching
    context. This would greatly improve performance for channel programs
    that just gather information, as they won't have to wait for synching
    context anymore.
    
    === What is implemented?
    
    This feature introduces the following:
    - A new command line flag in "zfs program" to specify our intention
      to run in open context. (The -n option)
    - A new flag/option within the channel program ioctl which selects
      the context.
    - Appropriate error handling whenever we try a channel program in
      open-context that contains zfs.sync* expressions.
    - Documentation for the new feature in the manual pages.
    
    === How do we handle zfs.sync functions in open context?
    
    When such a function is found by the interpreter and we are running
    in open context we abort the script and we spit out a descriptive
    runtime error. For example, given the script below ...
    
    arg = ...
    fs = arg["argv"][1]
    err = zfs.sync.destroy(fs)
    msg = "destroying " .. fs .. " err=" .. err
    return msg
    
    if we run it in open context, we will get back the following error:
    
    Channel program execution failed:
    [string "channel program"]:3: running functions from the zfs.sync
    submodule requires passing sync=TRUE to lzc_channel_program()
    (i.e. do not specify the "-n" command line argument)
    stack traceback:
                [C]: in function 'destroy'
                [string "channel program"]:3: in main chunk
    
    === What about testing?
    
    We've introduced new wrappers for all channel program tests that
    run each channel program as both (startard & open-context) and
    expect the appropriate behavior depending on the program using
    the zfs.sync module.
    
    OpenZFS-issue: https://www.illumos.org/issues/8677
    OpenZFS-commit: openzfs/openzfs@17a49e1
    
    Fix coverity defects: zfs channel programs
    
    CID 173243, 173245:  Memory - corruptions  (OVERRUN)
     Added size argument to lcompat_sprintf() to avoid use of INT_MAX
    
    CID 173244:  Integer handling issues  (OVERFLOW_BEFORE_WIDEN)
     Added cast to uint64_t to avoid a 32 bit overflow warning
    
    CID 173242:  Integer handling issues  (CONSTANT_EXPRESSION_RESULT)
     Conditionally removed unused luai_numisnan() floating point check
    
    CID 173241:  Resource leaks  (RESOURCE_LEAK)
     Added missing close(fd) on error path
    
    CID 173240:    (UNINIT)
    Fixed uninitialized variable in get_special_prop()
    
    CID 147560:  Null pointer dereferences  (NULL_RETURNS)
    Cleaned up bad code merge in dsl_dataset_promote_check()
    
    CID 28475:  Memory - illegal accesses  (OVERRUN)
    Fixed lcompat_sprintf() to use a size paramater
    
    CID 28418, 28422:  Error handling issues  (CHECKED_RETURN)
    Added function result cast to (void) to avoid warning
    
    CID 23935, 28411, 28412:  Memory - corruptions  (ARRAY_VS_SINGLETON)
    Added casts to avoid exposing result as an array
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Don Brady <don.brady@delphix.com>
    Closes #7181
    
    Add JSON output support to channel programs
    
    The changes piggyback JSON output support on top of channel programs
    (#6558).  This way the JSON output support is targeted to scripting
    use cases and is easily maintainable since it really only touches
    one function (zfs_do_channel_program()).
    
    This patch ports Joyent's JSON nvlist library from illumos to enable
    easy JSON printing of channel program output nvlist.  To keep the
    delta small I also took advantage of the fact that printing in
    zfs_do_channel_program() was almost always done before exiting
    the program.
    
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Reviewed-by: Tony Hutter <hutter2@llnl.gov>
    Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
    
    Add tunables for channel programs
    
    This patch adds tunables for modifying the maximum memory limit and
    maximum instruction limit that can be specified when running a channel
    program.
Commits on Oct 10, 2018
  1. Fix arc_release() refcount

    behlendorf authored and lundman committed Oct 10, 2018
    Update arc_release to use arc_buf_size().  This hunk was accidentally
    dropped when porting compressed send/recv, ZOL 2aa34383b.
    
    Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
    Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
  2. Add zfs_refcount_transfer_ownership_many()

    behlendorf authored and lundman committed Oct 10, 2018
    When debugging is enabled and a zfs_refcount_t contains multiple holders
    using the same key, but different ref_counts, the wrong reference_t may
    be transferred.  Add a zfs_refcount_transfer_ownership_many() function,
    like the existing zfs_refcount_*_many() functions, to match and transfer
    the correct refcount_t;
    
    This issue may occur when using encryption with refcount debugging
    enabled.  An arc_buf_hdr_t can have references for both the
    hdr->b_l1hdr.b_pabd and hdr->b_crypt_hdr.b_rabd both of which use
    the hdr as the reference holder.  When unsharing the buffer the
    p_abd should be transferred.
    
    This issue does not impact production builds because refcount holders
    are not tracked.
    
    Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
    Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Commits on Oct 4, 2018
  1. Refcounted DSL Crypto Key Mappings

    tcaputi authored and lundman committed Oct 4, 2018
    Since native ZFS encryption was merged, we have been fighting
    against a series of bugs that come down to the same problem: Key
    mappings (which must be present during all I/O operations) are
    created and destroyed based on dataset ownership, but I/Os can
    have traditionally been allowed to "leak" into the next txg after
    the dataset is disowned.
    
    In the past we have attempted to solve this problem by trying to
    ensure that datasets are disowned ater all I/O is finished by
    calling txg_wait_synced(), but we have repeatedly found edge cases
    that need to be squashed and code paths that might incur a high
    number of txg syncs. This patch attempts to resolve this issue
    differently, by adding a reference to the key mapping for each txg
    it is dirtied in. By doing so, we can remove many of the
    unnecessary calls to txg_wait_synced() we have added in the past
    and ensure we don't need to deal with this problem in the future.
    
    Reviewed-by: Jorgen Lundman <lundman@lundman.net>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
  2. Return errorcode in arc_read

    lundman committed Oct 4, 2018
Commits on Oct 2, 2018
  1. Tag zfs-1.8.0

    lundman committed Oct 2, 2018
Commits on Sep 28, 2018
  1. Always attempt to expand /dev/disk names

    lundman committed Sep 28, 2018
    Normally, getmntany() is called without a search argument, so will skip
    all the "if()" statements, call statfs2mnttab() which in turn calls
    expand_disk_to_zfs() to expand /dev/disk* names back into dataset name.
    
    However, when searching for a specific mount, by passing in a search argument,
    it would not expand the /dev/disk name, but compare dataset name to that
    of /dev/disk name.
    
    Now we always call statfs2mnttab() to attempt to expand names, and
    compare against the (possibly) expanded name.
    
    This should fix callers of zfs_is_mounted() when devdisk is involved.
Commits on Sep 20, 2018
  1. Remove abort() from devid() wrappers

    lundman committed Sep 20, 2018
    zpool abort()ing is ugly to users, it is better we return failure and
    let the callers handle that.
Commits on Sep 18, 2018
  1. Send rename event for /etc/zfs/zpool.cache

    lundman committed Sep 18, 2018
    Recent commit 99661cd accidentally broke the
    OsX specific "/etc/zfs/zpool.cache" commit, which passes to zed the event to
    rename the file as well as its name.
    
    A feature added in 6300b27
Commits on Sep 12, 2018
  1. Tag zfs-1.7.4

    lundman committed Sep 12, 2018
    META file updated.
Commits on Aug 23, 2018
Commits on Aug 22, 2018
  1. 9688 aggsum_fini leaks memory

    pcd1193182 authored and lundman committed Aug 22, 2018
    Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>
    Reviewed by: Matt Ahrens <matt@delphix.com>
    Reviewed by: Prashanth Sreenivasa <pks@delphix.com>
    
    aggsum_fini() does not free as_buckets. It should do so.
    
    Upstream bug: DLPX-58607
Commits on Aug 8, 2018
Commits on Aug 7, 2018
  1. many threads directly evicting dbufs makes a computer slow

    rottegift authored and lundman committed Jun 18, 2018
    On a low-latency pool (RAM disk, SSD, L2 backed spinny
    disk, or even a big hot ARC) with many many snapshots
    running multiple concurrent
    "zfs list -r -o name,creation -t snap pool"
    thrashes the dbuf cache, and decompressing the
    metadata blocks causes the kernel to eat all the available
    CPU, kiling userland responsiveness and depressing
    I/O throughput.
    
    Keep a simple atomic count of the number of threads
    directly evicting from the dbuf, and introduce an
    IODelay(1) -- which does a microsecond of REP;NOP aka PAUSE,
    yielding to the other hyperthread -- if the number of such
    threads is less than half of max_ncpus.   For any directly
    evicting thread above that threshold, do an IOSleep(1),
    which deschedules the thread entirely for approximately a
    millisecond, letting other threads (including the dbuf
    eviction thread) make progress.
Commits on Aug 6, 2018
Commits on Jul 24, 2018
  1. Reserve DMU_BACKUP_FEATURE for ZSTD

    allanjude authored and lundman committed Jun 12, 2018
    Reserve bit 25 for the ZSTD compression feature from FreeBSD.
    
    Signed-off-by: Allan Jude <allanjude@freebsd.org>
Commits on Jul 19, 2018
  1. Refactor arc_hdr_realloc_crypt()

    tcaputi authored and lundman committed Jul 19, 2018
    The arc_hdr_realloc_crypt() function is responsible for converting
    a "full" arc header to an extended "crypt" header and visa versa.
    This code was originally written with a bcopy() so that any new
    members added to arc headers would automatically be included
    without requiring a code change. However, in practice this (along
    with small differences in kmem_cache implementations between
    various platforms) has caused a number of hard-to-find problems in
    ports to other operating systems. This patch solves this problem
    by making all member copies explicit and adding ASSERTs for fields
    that cannot be set during the transfer. It also manually resets the
    old header after the reallocation is finished so it can be properly
    reallocated and reused.
    
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
Commits on Jul 14, 2018
  1. Revert "Temporary: fix arc_hdr_realloc_crypt() to zero released hdr"

    lundman committed Jul 14, 2018
    This reverts commit d282603.
    
    Replaced with proper fix:
    
    Refactor arc_hdr_realloc_crypt()
Commits on Jul 13, 2018
  1. Correct missing os_raw_receive

    lundman committed Jul 13, 2018
    Causing ASSERT/VERIFY when receiving raw
  2. Temporary: fix arc_hdr_realloc_crypt() to zero released hdr

    lundman committed Jul 13, 2018
    This is a temporary commit until the proper fix will come from upstream
  3. Separate the error code for already unloaded key

    lundman committed Jul 13, 2018
    Connected to the future commit: Adopt pyzfs from ClusterHQ
    from ZOL (85ce3f4fd1).
Commits on Jul 12, 2018
  1. Fix hash_lock / keystore.sk_dk_lock lock inversion

    behlendorf authored and lundman committed Feb 4, 2018
    The keystore.sk_dk_lock should not be held while performing I/O.
    Drop the lock when reading from disk and update the code so
    they the first successful caller adds the key.
    
    Improve error handling in spa_keystore_create_mapping_impl().
    
    Reviewed by: Thomas Caputi <tcaputi@datto.com>
    Reviewed-by: RageLtMan <rageltman@sempervictus>
    Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
  2. Raw receives must compress metadnode blocks

    tcaputi authored and lundman committed Feb 21, 2018
    Currently, the DMU relies on ZIO layer compression to free LO
    dnode blocks that no longer have objects in them. However,
    raw receives disable all compression, meaning that these blocks
    can never be freed. In addition to the obvious space concerns,
    this could also cause incremental raw receives to fail to mount
    since the MAC of a hole is different from that of a completely
    zeroed block.
    
    This patch corrects this issue by adding a special case in
    zio_write_compress() which will attempt to compress these blocks
    to a hole even if ZIO_FLAG_RAW_ENCRYPT is set. This patch also
    removes the zfs_mdcomp_disable tunable, since tuning it could
    cause these same issues.
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
  3. Raw receive should change key atomically

    tcaputi authored and lundman committed Feb 21, 2018
    Currently, raw zfs sends transfer the encrypted master keys and
    objset_phys_t encryption parameters in the DRR_BEGIN payload of
    each send file. Both of these are processed as soon as they are
    read in dmu_recv_stream(), meaning that the new keys are set
    before the new snapshot is received. In addition to the fact that
    this changes the user's keys for the dataset earlier than they
    might expect, the keys were never reset to what they originally
    were in the event that the receive failed. This patch splits the
    processing into objset handling and key handling, the later of
    which is moved to dmu_recv_end() so that they key change can be
    done atomically.
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
  4. ZFS send fails to dump objects larger than 128PiB

    loli10K authored and lundman committed Oct 26, 2017
    When dumping objects larger than 128PiB it's possible for do_dump() to
    miscalculate the FREE_RECORD offset due to an integer overflow
    condition: this prevents the receiving end from correctly restoring
    the dumped object.
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
    Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
  5. Prevent raw zfs recv -F if dataset is unencrypted

    tcaputi authored and lundman committed Feb 21, 2018
    The current design of ZFS encryption only allows a dataset to
    have one DSL Crypto Key at a time. As a result, it is important
    that the zfs receive code ensures that only one key can be in use
    at a time for a given DSL Directory. zfs receive -F complicates
    this, since the new dataset is received as a clone of the existing
    one so that an atomic switch can be done at the end. To prevent
    confusion about which dataset is actually encrypted a check was
    added to ensure that encrypted datasets cannot use zfs recv -F to
    completely replace existing datasets. Unfortunately, the check did
    not take into account unencrypted datasets being overriden by
    encrypted ones as a case.
    
    Along the same lines, the code also failed to ensure that raw
    recieves could not be done on top of existing unencrypted
    datasets, which causes amny problems since the new stream cannot
    be decrypted.
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
Commits on Jul 5, 2018
  1. Fix coverity defects: CID 176037

    tcaputi authored and lundman committed Jul 5, 2018
    CID 176037: Uninitialized scalar variable
    
    This patch fixes an uninitialized variable defect caught by
    coverity and introduced in 69830602
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>
  2. Add ASSERT to debug encryption key mapping issues

    tcaputi authored and lundman committed Jul 5, 2018
    This patch simply adds an ASSERT that confirms that the last
    decrypting reference on a dataset waits until the dataset is
    no longer dirty. This should help to debug issues where the
    ZIO layer cannot find encryption keys after a dataset has been
    disowned.
    
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Tom Caputi <tcaputi@datto.com>