Add support for APFS and Sierra's HFS #1
Conversation
Sierra changes HFS' fs_type_num to 23, and APFS (which also seems to support compression using the same API) uses fs_type_num 29. This expands the check in fileIsCompressable to accept all three fs_type_num possibilities.
|
Thanks for this!
The change is almost suspiciously simple. Have you verified that fs_type_num values are all safe on all OS X versions? In other words, is it safe to accept all 3 of them on any OS X version because pre-10.12 versions don't use values 23 and 29 for anything and 10.12+ don't use 17? Or should we do a runtime check of the form
(macOSVersion < 10.12 && fs_type_num == 17) || (macOSVersion >= 10.12 && (fs_type_num == 23 || fs_type_num == 29))
Out of curiosity: so APFS still has no support for transparent compression?
R.
|
|
I just did a cursory check of bsd/vfs/vfs_conf.c XNU sources, and it seems those numbers were never used, and numbers are never re-used. APFS's value I found through adding a simple printf debug to afsctool. There's conflicting info found here, which states that FAT32 uses 29 in Sierra: https://groups.google.com/forum/#!msg/uk.comp.sys.mac/2BaK45HenzU/5VN8Ksb-CQAJ A version check seems reasonable, or perhaps there's a better way to detect filesystem type? As for APFS transparent compression, no mention was ever made about compression during presentations, and no currently-existing documentation mentions it. Final documentation has yet to be published, of course. In the meantime, APFS must support HFS' extended-attribute compression "hack" if it is to convert HFS volumes. |
|
Thanks. Reading the discussion on google groups I see there is also a string representation (f_fstypename). Maybe we should use that, with 3-character comparison to hfs and a 4-char. comparison to apfs, both case-insensitive.
|
|
The most recent edit now compares by string (case-insensitively). |
|
Great, thanks. I take it you tested it on 10.12, I'll test it on 10.9 on my end and then we can hopefully move on :)
R.
|
This should be more future-proof and less prone to side-effects than using the undocumented f_type statfs field. Contributed by Tyler Loch: #1
|
Thanks for the contribution. I've cherry-picked it. |
afsctool.c
Outdated
| @@ -92,7 +92,7 @@ bool fileIsCompressable(const char *inFile, struct stat *inFileInfo) | |||
| // ret >= 0 && fsInfo.f_type == 17 | |||
| // && S_ISREG(inFileInfo->st_mode) | |||
| // && (inFileInfo->st_flags & UF_COMPRESSED) == 0 ); | |||
| return (ret >= 0 && (fsInfo.f_type == 17 || fsInfo.f_type == 23 || fsInfo.f_type == 29) | |||
| return (ret >= 0 && (!strcasecmp("hfs", fsInfo.f_fstypename) || !strcasecmp("apfs", fsInfo.f_fstypename)) | |||
| && S_ISREG(inFileInfo->st_mode) | |||
There was a problem hiding this comment.
Hmm so would need to add "zfs" here too, since we finally figured out how that decmpfs stuff works (..enough to undo it!). What WOULD have been neat here is a test like
VOL_CAPABILITIES_FORMAT] & VOL_CAP_FMT_DECMPFS_COMPRESSION
So the FS can decide if it supports it or not. Since testing for "hfs" was the reason we had to add mimic-feature :) rsync and ditto both use capabilites test.
There was a problem hiding this comment.
|
Hmm so would need to add "zfs" here too, since we finally figured out how that decmpfs stuff works (..enough to undo it!).
Exactly what do you mean with that? You recognise the attribute when someone transfers a file without decompressing it first, and then do the decompression?
What I'm trying to understand is how this changes anything for an external compressor like afsctool which is only interested in knowing if the target filesystem has real support for HFS compression? (Which means support for resource forks for any file that's over roughly 3k once compressed.)
Even if you found a way to control the type of ZFS compression per file there would still be no point to let afsctool compress the data if the filesystem is going to do it all over again in a slightly different way (using gzip instead of zip).
I'll look into using capabilities because indeed, ZFS datasets mimicking hfs are a problem currently.
|
|
Well, this all started, because Apple's Mail does not check So, It would be pleasing if you guys would also use So that brings us to now, with ZFS I was forced to add support to handle decmpfs, so that Mail will work on a ZFS dataset. I actually do NOT set |
|
Apple Mail again... that's the culprit for me getting involved with KDE when I upgraded to 10.9 , because it no longer accepted NOT to cache all mail locally (inacceptable to me since I archive my email via a local IMAP server).
files where 0-length on ZFS.
That (what, where, how?! 8-)) actually happens with libarchive (bsdtar): unpacking an archive with --hfsCompression onto ZFS dataset gives 0-length files instead of errors or just non-compressed files.
we will then lie to `afsctool` just to get our way :)
Yeah, and what would your way be? As far as I can tell the whole problem is that you may already by lying to afsctool, telling it you're HFS while you're evidently not.
All this still doesn't really tell me what happens when you receive a hfs-compressed file, and above all not how I can avoid wasting resources doing the compression step when targetting a ZFS dataset in afsctool. It's not just the cycles, it's also an unneeded file rewrite which always comes with a risk and would also increase ZFS fragmentation for no reason.
Afsctool purpose is to save disk space and my version is designed to operate as efficiently as possible (I hope) on large directories. That's certainly how I use it, and that's also why it tries to determine if the kind of compression it applies is *really* supported.
IOW, I think I actually want a bit more from you than you want from me: a way to determine I'm dealing with a ZFS dataset so I can just skip anything that's on such a FS, just like when the FS type isn't hfs or apfs ;)
So, I intend to make us say `no` to `VOL_CAP_FMT_DECMPFS_COMPRESSION` for zfs, but `yes` if mimic is set. Which is how it should be really. Both `apfs` and `hfs` reply `yes`, as they should.
I *think* I don't agree with that, and not only because it would mean that the capabil. check is going to give the exact same result as just checking the FS type name. Hfs and APFS should indeed reply yes because they implement HFS compression, but from what I understand you don't intend to implement decmps in ZFS. That still allows you to recognise the xattr and unset it from and decompress incoming files.
|
Yes and no. I do not want to lie, I want to wave the ZFS flag proudly, and zfs will always report as zfs, and with the correct capabilities, by default. But, users required the mimic feature for the apps that fail to check capabilities. So, in for a penny, in for a pound. If (and only if) we are to mimic hfs, we will do so fully. This is actually also needed when booting OSX on zfs, on at the very least the "/" mount. After that, mimic can be off. Mimic "on" means we have about 40 HFS IOCTLs that we suddenly had to support (get next hard-link etc). Quite tedious. But, as it happens, you can detect zfs under mimic too, as it happens, the zfs userland needs to know. I set the statfs "subtype" to I do not want much from
Oh right, I forgot. Right now, the last thing they do when using decmpfs, is to set UF_COMPRESSED. So in vnop_setattr (chflags) when zfs writes the flag change, we call Yep. I big waste of cpu cycles, all because Mail blindly compresses. (Scarily, I tried to return error in makenamedstream, in setxattr, and chflags - Mail ignores all errors and just keeps going. hurrraah) So, in a perfect world, zfs reports as zfs - and nothing tries to compress. But, if mimic is on, or poor software is running, we can at least handle it by decompressing. Fragmentation is not an issue, as ZFS is COW. The decompress takes just as many cycles as HFS read would, but the net effect is a waste. So, what about But, I can just keep using |
|
On Tuesday December 19 2017 03:49:40 Jorgen Lundman wrote:
Yes and no. *I* do not want to lie, I want to wave the ZFS flag proudly, and zfs will always report as zfs
Good :)
But, as it happens, you can detect zfs under mimic too, as it happens, the zfs userland needs to know. I set the statfs "subtype" to `MNTTYPE_ZFS_SUBTYPE ('Z'<<24|'F'<<16|'S'<<8)`.
I was counting on something like that.
I do not want much from `Afsctool` really, but I want *you* to *want* to do the right thing. :) It is much better to use capabilities, than hardcoded string check. It is what creates this hassle to start with.
Of course I want to do the right thing, but I'm not yet convinced that using capabilities would be the way to go *in order to prevent inappropriate behaviour with ZFS* (as well as anywhere else). Using a dedicated API appeals to the geek in me, but if it ultimately gives me information that's unreliable like the current check I'll probably find more urgent matters to look into :)
I think I'll start by adding a check for MNTTYPE_ZFS_SUBTYPE.
> All this still doesn't really tell me what happens when you receive a hfs-compressed file
Oh right, I forgot. Right now, the last thing they do when using decmpfs, is to set UF_COMPRESSED. So in vnop_setattr (chflags) when zfs writes the flag change, we call `decmpfs_decompress_file()`, which converts the compressed namedstream back to plain text file, cleans up the xattrs and unsets UF_COMPRESSED.
That's more or less what rsync does when copying (potentially) compressed files to a FS which doesn't support decmpfs, right?
Yep. I big waste of cpu cycles, all because Mail blindly compresses. (Scarily, I tried to return error in makenamedstream, in setxattr, and chflags - Mail ignores all errors and just keeps going. hurrraah)
You filed a radar for that, I presume?
Fragmentation is not an issue, as ZFS is COW.
I meant the property reported by `zpool fragmentation`. I understand that this is only vaguely comparable to the symptom on traditional file systems but that it does behave in a comparable way - increasing over time up to a point where performance suffers or the pool even needs to be regenerated.
So, what about `afsctool` and zfs? Well, nothing really. But, I do need to test that we handle decmpfs correctly (in the tester environment we run all the time) so it would totally be neat if I could turn mimic on, (hence capabilities say `yes`) and confirm that a compressed write to zfs will not be a zero-length file using the `afsctool`.
And I'd like to avoid being as dumb as Apple Mail, compressing files if I know that they're going to be decompressed immediately behind my back. I could provide a build option for that though, if you see a use case for it.
Again, afsctool is for emulated in-place compression (or decompression) of existing files, it doesn't do anything else.
|
|
Second guessing the capability the file system reports doesn't make much sense to me. If the capability is asserted as supported, assume it is supported and proceed accordingly. If capability is asserted as unsupported, assume it is unsupported and proceed accordingly. I don't see what is gained by substituting your judgment for the judgment of the person setting the capability flag on or off. |
|
Don't worry, I'm not going to second-guess but will be making an informed choice instead. ZFS isn't HFS nor APFS, I know it will decompress HFS compressed files, so I'm going to work with that, period.
|
This is the preferred method to determine if the filesystem hosting a file supports HFS (decmpfs) compression. Also detect ZFS for Mac (o3x.org) which can mimic HFS and will then decompress files on the fly. afsctool will not waste cycles on compression and rewriting files unless requested with the HFSCOMPRESS_TO_ZFS CMake option (for testing). #1 (comment)
|
Done
`MNTTYPE_ZFS_SUBTYPE ('Z'<<24|'F'<<16|'S'<<8)`.
Doh, that's
```
#define MNTTYPE_ZFS_SUBTYPE 'ZFS'
```
in good Apple practice :)
|
Sierra changes HFS' fs_type_num to 23 (as mentioned here: jrk/afsctool#3), and APFS, which seems to support compression using the same API, uses fs_type_num 29.
This expands the check in fileIsCompressable to accept all three fs_type_num possibilities.
Tested using HFS and APFS disk images on macOS 10.12.4.
This is my first pull request...please be gentle :)