-
Notifications
You must be signed in to change notification settings - Fork 10
Development
You should also familiarize yourself with the Project roadmap so that you can put the technical details here in context.
For dealing with panics.
Apple's documentation: https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KEXTConcept/KEXTConceptDebugger/debug_tutorial.html
Boot the target VM with:
sudo nvram boot-args="-v keepsyms=1 debug=0x144"Make it panic.
On your development machine, you will need the Kernel Debug Kit. Download it from Apple.
gdb /Volumes/Kernelit/mach_kernel
(gdb) source /Volumes/KernelDebugKit/kgmacros
(gdb) target remote-kdp
(gdb) kdp-reattach 192.168.30.133 # use the IP of your target / crashed VM
(gdb) showallkmods
Find the addresses for the ZFS and SPL modules.
Press Ctrl+Z to suspend gdb, or use another terminal:
sudo kextutil -s /tmp -n -k /Volumes/KernelDebugKit/mach_kernel -e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ ../spl/module/spl/spl.kext/Then resume gdb, or return to the gdb terminal:
fg
(gdb) set kext-symbol-file-path /tmp
(gdb) add-kext /tmp/spl.kext
(gdb) add-kext /tmp/zfs.kext
(gdb) bt
echo "settings set target.load-script-from-symbol-file true" >> ~/.lldbinit
lldb /Volumes/KernelDebugKit/mach_kernelFrom Yosemite onward, the kernel path may instead be something like:
/Library/Developer/KDKs/KDK_10.10_14B25.kdk/System/Library/Kernels/kernel
Then:
(lldb) kdp-remote 192.168.30.146
(lldb) showallkmods
(lldb) addkext -F /tmp/spl.kext/Contents/MacOS/spl 0xffffff7f8ebb0000
(lldb) addkext -F /tmp/zfs.kext/Contents/MacOS/zfs 0xffffff7f8ebbf000
addkext seems broken, so use:
(lldb) target modules add ../spl/module/spl/spl.kext/Contents/MacOS/spl
(lldb) target modules load --file spl --slide 0xffffff7f91e63000
Then follow the GDB flow above.
If you prefer to work in GDB, you can always panic the kernel with:
sudo dtrace -w -n "BEGIN{ panic();}"But this can also be revealing:
sudo /usr/libexec/stackshot -i -f /tmp/stackshot.log
sudo symstacks.rb -f /tmp/stackshot.log -s -w /tmp/trace.txt
less /tmp/trace.txtExample stack excerpt:
PID: 156
Process: zpool
Thread ID: 0x4e2
Thread state: 0x9 == TH_WAIT |TH_UNINT
Thread wait_event: 0xffffff8006608a6c
Kernel stack:
machine_switch_context (in mach_kernel) + 366 (0xffffff80002b3d3e)
0xffffff800022e711 (in mach_kernel) + 1281 (0xffffff800022e711)
thread_block_reason (in mach_kernel) + 300 (0xffffff800022d9dc)
lck_mtx_sleep (in mach_kernel) + 78 (0xffffff80002265ce)
0xffffff8000569ef6 (in mach_kernel) + 246 (0xffffff8000569ef6)
msleep (in mach_kernel) + 116 (0xffffff800056a2e4)
0xffffff7f80e52a76 (0xffffff7f80e52a76)
0xffffff7f80e53fae (0xffffff7f80e53fae)
0xffffff7f80e54173 (0xffffff7f80e54173)
0xffffff7f80f1a870 (0xffffff7f80f1a870)
0xffffff7f80f2bb4e (0xffffff7f80f2bb4e)
0xffffff7f80f1a9b7 (0xffffff7f80f1a9b7)
0xffffff7f80f1b65f (0xffffff7f80f1b65f)
0xffffff7f80f042ee (0xffffff7f80f042ee)
0xffffff7f80f45c5b (0xffffff7f80f45c5b)
0xffffff7f80f4ce92 (0xffffff7f80f4ce92)
spec_ioctl (in mach_kernel) + 157 (0xffffff8000320bfd)
VNOP_IOCTL (in mach_kernel) + 244 (0xffffff8000311e84)
It is a shame that this only shows kernel symbols and not SPL/ZFS internals, but you can ask it to load another symbol file:
sudo kextstat # grab the addresses of SPL and ZFS again
sudo kextutil -s /tmp -n -k /Volumes/KernelDebugKit/mach_kernel -e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ ../spl/module/spl/spl.kext/
sudo symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.spl.symExample:
0xffffff800056a2e4 (0xffffff800056a2e4)
spl_cv_wait (in net.lundman.spl.sym) + 54 (0xffffff7f80e52a76)
taskq_wait (in net.lundman.spl.sym) + 78 (0xffffff7f80e53fae)
taskq_destroy (in net.lundman.spl.sym) + 35 (0xffffff7f80e54173)
0xffffff7f80f1a870 (0xffffff7f80f1a870)
Then:
sudo symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.zfs.symExample:
0xffffff7f80e54173 (0xffffff7f80e54173)
vdev_open_children (in net.lundman.zfs.sym) + 336 (0xffffff7f80f1a870)
vdev_root_open (in net.lundman.zfs.sym) + 94 (0xffffff7f80f2bb4e)
vdev_open (in net.lundman.zfs.sym) + 311 (0xffffff7f80f1a9b7)
vdev_create (in net.lundman.zfs.sym) + 31 (0xffffff7f80f1b65f)
spa_create (in net.lundman.zfs.sym) + 878 (0xffffff7f80f042ee)
Voilà.
This section is only relevant to the older O3X implementation that used the zones allocator. Newer versions use a custom kmem allocator.
In some cases, you may suspect memory issues, for instance after a panic like this:
panic(cpu 1 caller 0xffffff80002438d8): "zalloc: \"kalloc.1024\" (100535 elements) retry fail 3, kfree_nop_count: 0"@/SourceCache/xnu/xnu-2050.7.9/osfmk/kern/zalloc.c:1826
To debug this, attach GDB and use zprint:
(gdb) zprint
ZONE COUNT TOT_SZ MAX_SZ ELT_SZ ALLOC_SZ TOT_ALLOC TOT_FREE NAME
0xffffff8002a89250 1620133 18c1000 22a3599 16 1000 125203838 123583705 kalloc.16 CX
0xffffff8006306c50 110335 35f000 4ce300 32 1000 13634985 13524650 kalloc.32 CX
0xffffff8006306a00 133584 82a000 e6a900 64 1000 26510120 26376536 kalloc.64 CX
0xffffff80063067b0 610090 4a84000 614f4c0 128 1000 50524515 49914425 kalloc.128 CX
0xffffff8006306560 1070398 121a2000 1b5e4d60 256 1000 72534632 71464234 kalloc.256 CX
0xffffff8006306310 399302 d423000 daf26b0 512 1000 39231204 38831902 kalloc.512 CX
0xffffff80063060c0 100404 6231000 c29e980 1024 1000 22949693 22849289 kalloc.1024 CX
0xffffff8006305e70 292 9a000 200000 2048 1000 77633725 77633433 kalloc.2048 CX
In this case, kalloc.256 is suspect.
Reboot the kernel with zlog=kalloc.256 on the command line, then use:
(gdb) findoldest
oldest record is at log index 393:
--------------- ALLOC 0xffffff803276ec00 : index 393 : ztime 21643824 -------------
0xffffff800024352e <zalloc_canblock+78>: mov %eax,-0xcc(%rbp)
0xffffff80002245bd <get_zone_search+23>: jmpq 0xffffff80002246d8 <KALLOC_ZINFO_SALLOC+35>
0xffffff8000224c39 <OSMalloc+89>: mov %rax,-0x18(%rbp)
0xffffff7f80e847df <zfs_kmem_alloc+15>: mov %rax,%r15
0xffffff7f80e90649 <arc_buf_alloc+41>: mov %rax,-0x28(%rbp)
List an index:
(gdb) zstack 394
--------------- ALLOC 0xffffff8032d60700 : index 394 : ztime 21648810 -------------
0xffffff800024352e <zalloc_canblock+78>: mov %eax,-0xcc(%rbp)
0xffffff80002245bd <get_zone_search+23>: jmpq 0xffffff80002246d8 <KALLOC_ZINFO_SALLOC+35>
0xffffff8000224c39 <OSMalloc+89>: mov %rax,-0x18(%rbp)
0xffffff7f80e847df <zfs_kmem_alloc+15>: mov %rax,%r15
0xffffff7f80e90649 <arc_buf_alloc+41>: mov %rax,-0x28(%rbp)
How often was zfs_kmem_alloc involved?
(gdb) countpcs 0xffffff7f80e847df
occurred 3999 times in log (100% of records)
How often was arc_buf_alloc involved?
(gdb) countpcs 0xffffff7f80e90649
occurred 2390 times in log (59% of records)
ZFS is designed to aggressively cache filesystem data in main memory. The result can be significant filesystem performance improvements.
Allocator selection was especially challenging on OS X. Over time, O3X evolved through several approaches:
- Direct calls to
OSMalloc: rejected because performance was slow and the minimum allocation size is one page (4 KB) - Direct calls to
zalloc: rejected because only part of system memory could be accessed, and exceeding the limit caused a kernel panic - Direct calls to
bmalloc: a home-grown slice allocator used in O3X 1.2.7 and 1.3.0; successful but space-inefficient - Full
kmem/vmemallocator stack from Illumos, plus memory pressure monitoring
O3X uses the Solaris Porting Layer (SPL), which provides the Illumos kmem.h API for use by ZFS.
By default, kmem / vmem require a certain level of performance from the OS page allocator. OS X's page allocator can be easy to overwhelm. O3X tuned vmem to use a KMEM_QUANTUM of 512 KB chunks from the page allocator rather than the smaller allocations vmem prefers.
As of 1.5.2, KMEM_QUANTUM was switched to 128 KB based on user feedback. That improved some cases but also caused reduced performance and beachballing on other machines.
Further investigation suggested that vmem needed to behave more like libumem in userspace, cooperating with an upstream allocator and returning memory under pressure more gracefully.
Reference:
Jeff Bonwick’s paper on kmem and vmem:
https://www.usenix.org/legacy/event/usenix01/full_papers/bonwick/bonwick_html/
The kmem allocator has an internal diagnostic mode. In this mode, heap memory is instrumented with markers so the allocator can detect common memory handling errors as the program runs.
This mode is disabled by default because it carries a significant performance penalty.
Examples of detectable errors:
- Modify after free
- Write past end of buffer
- Free of memory not managed by
kmem - Double free
- Various other corruptions
- Freed size != allocated size
- Freed address != allocated address
Enable debug mode by compiling with the preprocessor symbol DEBUG. At a minimum, spl-kmem.c and spl-osx.c need to see this define for the debugging features to be fully enabled.
In debug mode, choose whether kmem should log the fault and then panic, or just log it. In spl-kmem.c, set:
-
kmem_panic=0to log only -
kmem_panic=1to log and panic
Example test code added to spl_start():
{
...
int *p;
for (int i = 0; i < 20; i++) {
p = (int*)spl_kmem_alloc(1024);
spl_kmem_free(p);
*p = 0;
}
}Example logged output:
14/08/2015 5:09:47.000 PM kernel[0]: SPL: kernel memory allocator: buffer modified after being freed
14/08/2015 5:09:47.000 PM kernel[0]: SPL: modification occurred at offset 0x0 (0xdeadbeefdeadbeef replaced by 0xdeadbeef00000000)
14/08/2015 5:09:47.000 PM kernel[0]: SPL: buffer=0xffffff887a87d980 bufctl=0xffffff887a7ad840 cache: kmem_alloc_1152
14/08/2015 5:09:47.000 PM kernel[0]: SPL: previous transaction on buffer 0xffffff887a87d980:
14/08/2015 5:09:47.000 PM kernel[0]: SPL: thread=0 time=T-0.000001383 slab=0xffffff887a5ffe68 cache: kmem_alloc_1152
14/08/2015 5:09:47.000 PM kernel[0]: SPL: net.lundman.spl : _kmem_cache_free_debug + 0x227
14/08/2015 5:09:47.000 PM kernel[0]: SPL: net.lundman.spl : _kmem_cache_free + 0x173
14/08/2015 5:09:47.000 PM kernel[0]: SPL: net.lundman.spl : _zfs_kmem_free + 0x2c4
14/08/2015 5:09:47.000 PM kernel[0]: SPL: net.lundman.spl : _spl_start + 0x2bb
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN6OSKext5startEb + 0x40b
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN6OSKext4loadEhhP7OSArray + 0xdd
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN6OSKext4loadEhhP7OSArray + 0x3e1
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN6OSKext22loadKextWithIdentifierEP8OSStringbbhhP7OSArray + 0xf2
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZNK11IOCatalogue14isModuleLoadedEP12OSDictionary + 0xe0
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN9IOService15probeCandidatesEP12OSOrderedSet + 0x2c4
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN9IOService14doServiceMatchEj + 0x22a
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : __ZN15_IOConfigThread4mainEPvi + 0x13c
14/08/2015 5:09:47.000 PM kernel[0]: SPL: mach_kernel : _call_continuation + 0x17
If you want to compile O3X for a specific OS X version, for example compiling for 10.9 on a 10.10 host:
SPL:
./configure --with-kernel-headers=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/Kernel.framework/ CFLAGS=-mmacosx-version-min=10.9ZFS:
./configure --with-kernelsrc=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/Kernel.framework/ CFLAGS=-mmacosx-version-min=10.9Huge thanks to Brendan Gregg for so much of the dtrace magic.
To profile the kernel while running a command:
sudo dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacksThis runs for 60 seconds.
Convert it to a flamegraph:
./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svgExamples mentioned in the original page:
rsync -a /usr/ /BOOM/deletea/- Bonnie++ in various stages
- IOzone comparisons
An initial port of the standard ZFS test suite was created. It consists of scripts and utility programs that exercise the breadth of the ZFS filesystem.
The tests are best run in a virtual machine with a baseline setup captured in a snapshot. Because the tests are destructive, the VM should be reverted to the snapshot before future runs.
Test runs typically take 2–4 hours depending on the hardware setup.
The user zfs-test needs to be able to run sudo without entering a password. Add the following to sudoers:
zfs-tests ALL=(ALL) NOPASSWD: ALL
The sudo root environment must also preserve certain variables from zfs-test:
Defaults env_keep += "__ZFS_MAIN_MOUNTPOINT_DIR"
Modify /etc/bashrc to contain:
export __ZFS_MAIN_MOUNTPOINT_DIR="/"If your development directory is ~you/Developer, clone zfs, spl, and zfs-tests there:
cd ~you/Developer
git clone git@github.com:openzfsonosx/zfs-test.git
git clone git@github.com:openzfsonosx/zfs.git
git clone git@github.com:openzfsonosx/spl.gitBuild ZFS using the source-build instructions.
Ensure that /var/tmp has approximately 100 GB of free space.
Create three virtual hard drives of 10–20 GB each.
Set up the tests:
cd ~you/Developer/zfs-tests
./autogen.sh
./configure CC=clang CXX=clang++Edit the generated Makefile and change the test_hw recipe so that your three virtual disks are listed in the DISKS environment variable:
test_hw: test_verify test/zfs-tests/cmd
@KEEP="`zpool list -H -oname`" STF_TOOLS=$(abs_top_srcdir)/test/test-runner/stf STF_SUITE=$(abs_top_srcdir)/test/zfs-tests DISKS="/dev/disk3 /dev/disk1 /dev/disk2" su zfs-tests -c "ksh $(abs_top_srcdir)/test/zfs-tests/cmd/scripts/zfstest.ksh $$RUNFILE"Run the test suite:
sudo make test_hwThe test suite writes summary pass/fail information to the console as it runs. On completion, summary statistics are written to the console.
Test log files are stored in /var/tmp/<testrun>, where testrun is a unique-looking number. In that directory there is a log file and a directory per test. Each test directory contains detailed log information.
Quick comparison testing, mainly as an indicator rather than ideal benchmarking.
HFS+ and ZFS were created on the same virtual disk in VMware. The pool was created with:
sudo zpool create -f -o ashift=12 -O atime=off -O casesensitivity=insensitive -O normalization=formD BOOM /dev/disk1The HFS+ filesystem was created with the standard Disk Utility defaults.
IOzone was run in standard automode:
sudo iozone -a -b outfile.xlsThe original page included flamegraph and chart images for HFS+ and ZFS read/write comparisons.
See VFS.
Create two files, each 100 MB, to be used as block devices:
dd if=/dev/zero bs=1m count=100 of=vdisk1
dd if=/dev/zero bs=1m count=100 of=vdisk2Attach the files as raw disk images:
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount vdisk1
# /dev/disk2
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount vdisk2
# /dev/disk3Create a mirrored zpool:
sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive -O normalization=formD tank mirror disk2 disk3Show zpool status:
sudo zpool statusExample output:
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
disk2 ONLINE 0 0 0
disk3 ONLINE 0 0 0
errors: No known data errors
Then test ZFS features, find bugs, and so on.
Export the pool:
sudo zpool export tankDetach the raw images:
hdiutil detach disk2
hdiutil detach disk3This section outlines differences between the OS X port and ZFS on other platforms.
To avoid deadlocks, O3X does not call VFS with locks held. This differs somewhat from the original Illumos code.
In particular, when creating a znode in zfs_mknode and zfs_znode_alloc, the vnode is not attached there because the code is still inside a dmu_tx.
Instead, VNOPs such as zfs_create, zfs_mkdir, zfs_symlink, and zfs_make_xattrdir call zfs_znode_getvnode() after the dmu_tx has completed.
This creates a small window where another thread can call zget() on the same object before the vnode has been attached. That is detected in zget(), which delays until the vnode is attached.
There is also special handling in zil_lwb_commit() to ensure zfs_get_data() is called without locks and with the vnode attached.
Illumos has a delete fastpath. In zfs_remove, if a znode can be deleted immediately, it marks the vnode as free and directly calls zfs_znode_delete(). Otherwise it adds the znode to zfs_unlinked_add().
On OS X, there is no way to directly release a vnode. XNU retains full control of vnode lifetime. Even after vnode_recycle(), the vnode is not released until vnop_reclaim is called.
O3X therefore attempts vnode_recycle(), and only if it returns 1 does it know that reclaim has already happened and it can call zfs_znode_delete() directly:
zp->z_fastpath = B_TRUE;
if (vnode_recycle(vp) == 1) {
/* recycle/reclaim is done, so we can just release now */
zfs_znode_delete(zp, tx);
} else {
/* failed to recycle, so just place it on the unlinked list */
zp->z_fastpath = B_FALSE;
zfs_unlinked_add(zp, tx);
}There is also special lock handling in zfs_zinactive, since it can be called from inside vnode_create() while locks are already held.
There is no way to trigger a mount directly from the XNU kernel.
Apple uses a static nfsmount() helper that third-party code cannot call. O3X therefore creates a fake /dev/diskX entry for the snapshot. diskarbitrationd notices the new disk, probes it, and eventually calls zfs.util.
Automount is disabled here because there is no way to specify a mountpoint automatically. zfs.util therefore calls DADiskMount to mount the snapshot to the correct directory.
This means O3X needs extra VNOPs in zfs_ctldir.c to provide the right answers during the mount sequence.
There are two relevant calls in the OS X SPL.
Use spl_vn_rdwr() when zfs_onexit is involved, for example in dmu_send.c (zfs recv/send) and zfs_ioc_diff (zfs diff).
The XNU implementation of zfs_onexit, via getf and releasef, stores the internal struct fileproc inside a wrapper struct spl_fileproc, allowing spl_vn_rdwr() to do I/O on non-file-backed vnodes such as pipes or sockets.
Other code, such as vdev_file.c, should keep using the normal vn_rdwr().
XNU can ask for many fields in vnop_getattr, including VA_NAME, which Finder uses heavily, especially in the vfs_vget path.
O3X therefore caches the name in the znode so it can be returned cheaply in vnop_getattr, falling back to zap_value_search when necessary.
The value is also set on the vnode in vfs_vget() using vnode_update_identity(), which Spotlight expects.
Hardlinks add further complications.
In POSIX, all hardlinks share va_fileid, and z_links is incremented for each target. Finder on OS X also requires a separate va_linkid, and expects each hardlink target to have a unique va_linkid.
That va_linkid is then used in vfs_vget() to map back to the actual znode/fileid and update the name and parent ID.
To support this, O3X builds two AVL trees in vnop_getattr:
-
zfsvfs->z_hardlinks, indexed by(parent id, file id, name) -
zfsvfs->z_hardlinks_linkid, indexed by(linkid)
That allows O3X to create unique va_linkid values for hardlinks, starting at 0x80000000 to reduce collision risk.
vfs_vget() first checks the AVL tree for va_linkid; if found, it can zget() the correct va_fileid and set the hardlink name and parent ID. If not found, it falls back to regular va_fileid lookup.
Care is also needed in vnop_remove to clear hardlink AVL nodes, and in vnop_rename to update the source mapping. The AVL trees are unloaded at unmount.
Add the upstream OpenZFS repository as a remote:
[remote "upstream"]
url = git@github.com:openzfs/openzfs.git
fetch = +refs/heads/*:refs/remotes/upstream/*Set the rename limit very high so Git can find moved paths:
git config merge.renameLimit 999999or in config:
[merge]
renameLimit = 999999Make sure the remote is up to date:
git fetch upstreamCheck what is new:
git log --stat upstream/masterFor each new commit you want to bring in:
git cherry-pick <commit-hash>Fix any conflicts, then make sure it still compiles cleanly.
Always update the commit message afterward:
git commit --amendDelete lines such as:
Closes #324
since they do not match the downstream issue tracker.
Check out the PR branch, for example PR 124:
git fetch upstream pull/124/head:pr124
git checkout pr124Then inspect the commits you want to merge.