-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compress, decompress, compress results in lots of mismatch errors #62
Comments
This reproduces consistently for me with version of playonmac 4.4.3 which is the current most recent version for me. |
Set a breakpoint on the message output: before afsctool reverts the compression, the file ends up containing all zeros |
Focusing on one of the files which show this behavior: The file fits in one block. It seems that when compressed with zlib, the file is compressed into the resource fork (it isn't compressed enough to fit in the decmpfs xattr). However, when compressed with lzfse, it does compress enough to fit into the decmpfs xattr. It's not clear why this is a problem though. I've verified that the decmpfs xattr written is exactly the same when it's compressing a fresh version of the dll and when compressing after it's already been compressed and decompressed, it's not clear what could be carrying over to change things: with the same decmpfs xattr contents, enabling the compressed flag in one case works, and in the other case results in a file which reports to contain all zeros. It really feels like a macos bug, I don't see any differences I can observe in the file before/after a compress+decompress cycle, but we're getting different behavior. |
I believe this is a macos bug: if a file has been compressed into a resource fork, uncompressing the file, and recompressing directly into the decmpfs xattr, the file will appear as all zeros (of the correct len). I've created a minimal reproduction: https://gist.github.com/Dr-Emann/3d0f272d08c68a889fa368597a10cea7. If |
It's not clear why this is a problem though. I've verified that the decmpfs xattr written is exactly the same when it's compressing a fresh version of the dll and when compressing after it's already been compressed and decompressed
So you have verified that the compressed content is correct when you read it back back in? Theoretically that should mean that it decompresses to the expected original content but did you verify that?
Sadly I can't work on this myself because 10.9 doesn't support LZFSE compressed files (so it's also a bit hard to wrap my head about what's going on). We could implement the decompression inside afsctool but that won't arrange anything if we can still produce compressed files that appear to be filled with zeroes to anyone else.
Unless of course that implementation reproduces the issue...
What I don't get is that the verification step doesn't catch this problem and rewrites the file from the backup...
Anyway, if you do feel that this is an OS bug: how up-to-date is your OS? If you're on a still supported version you could file a radar about this, and see what Apple have to say. My hardware is limited to 10.13 even if I wanted to upgrade so there's no point for me to file a ticket when the 1st request is going to be that I install an up-to-date OS version...
Should we just disable compression into the decmpfs xattr?
I'm a bit surprised that LZFSE can compress better than zlib ... did you try the ultimate zlib compression (-9)?!
|
Results are compressing an app copied fresh out of the DMG.
I'm using a MacBook Pro (M1 Pro CPU, 16GB RAM). |
OK, my bad, I don't ever use that compression...
|
No worries! It's an eye-opener, for sure. |
No worries! It's an eye-opener, for sure.
Especially the part where a straight-C compressor outperforms (speed-wise) zlib which does have SIMD optimisations. I'll try to compare the timing difference on my system which has CloudFlare's highly optimised version.
|
Please do, I love a good benchmark. I wonder how much ARM optimisation Apple have done to these algos. I'm betting "a lot"! |
LZFSE is highly optimized for ARM so not big surprise |
LZFSE is highly optimized for ARM so not big surprise
You mean such that even plain C compiles to something highly efficient?
|
on Intel mac: ❯ time afsctool -c -T ZLIB -9 PlayOnMac.app ❯ time afsctool -c -T LZFSE PlayOnMac.app so looks like LZFSE is just well done:) |
I think there's some confusion. After a file has had compression turned on with a resource fork, compressing the same file into the decmpfs header and turning the compression flag on results in a file that appears to be filled with zeros. The verification step DOES catch this (the original content is not all zeros), and correctly reverts compression. I verified that the same decmpfs header content works if the file has not already been compressed to a resource fork in the past, but fails if the file has been compressed to a resource fork in the past. There's nothing compressor specific: it should be possible to find a file which compresses to the resource fork at zlib -1, and fits in the decmpfs header at zlib -9, and it would exhibit the same behavior, compressing with zlib -1, uncompressing, then compressing with zlib -9 would end up with a file containing all zeros (and validation would revert, if enabled). I'm running macos 13.3.1, the newest version. Feedback reported through the feedback assistant as FB12146617 |
I verified that the same decmpfs header content works if the file has not already been compressed to a resource fork in the past, but fails if the file has been compressed to a resource fork in the past.
So the issue is triggered by the following sequence?
1) compress a "new" (uncompressed) file such that its contents end up in the resource fork (e.g. with ZIP)
2) uncompress the file
3) recompress the resulting file with LZFSE so that its contents go into the xattr
(Please confirm that you can NOT skip step 2; afsctool should reject files that are already HFS-compressed.)
What happens if you replace step 2 with something like `cat $file1 > $file2`, i.e. an operation that should remove the entire resource fork? I have never checked if afsctool removes the resource fork if it doesn't contain anything else (and I'm not at my Mac right now).
Indeed, it should be possible to create a file that will just be too large when zipped with a lower compression level, to see if that triggers the same bug.
|
Yes, those steps reproduce the issue. I can confirm that skipping step two does not reproduce the problem. Replacing step 2 with |
This smells of APFS copy in place. I'll try to test on an HFS Extended partition. |
Yes, but it could also hint at my hunch that it is somehow tied to something to do with the resource fork, possibly something as stupid as not cleaning it up entirely. Except that |
It currently compresses on HFS with ZLIB compression <= 8 which make the compressed content too large for storing in the XATTR. With ZLIB 9 (afsctool -cfvv 9) it will fail because of a content mismatch during the verification step. See issue #62. As far as I can tell having had a file compressed into the resource fork AT A GIVEN INODE will make compression into the xattr fail at a later stage. It doesn't even have to be the same file. For instance: > cat /usr/lib/libSystem.B.dylib > kkk > afsctool -cfvv -S -L kkk $PWD/kkk: Compression type: ZLIB in resource fork (4) File size (uncompressed; reported size by Mac OS 10.6+ Finder): 59088 bytes / 59 KB (kilobytes, base-10) File size (compressed): 12288 bytes / 12 KiB Compression savings: 79.2% Number of extended attributes: 0 Total size of extended attribute data: 0 bytes Approximate overhead of extended attributes: 536 bytes Uncompressed file size reported in compressed header: 59088 bytes > date > kkk > afsctool -cfvv -S -L kkk size mismatch=0 read=30 failure=0 content mismatch=1 (Undefined error: 0) /Volumes/Debian/Users/bertin/work/src/new/Qt/kkk: Compressed file check failed, reverting file changes /Volumes/Debian/Users/bertin/work/src/new/Qt/kkk: Unable to compress file. File content type: public.data File data fork size (reported size by Mac OS X Finder): 30 bytes / 4 KB (kilobytes, base-10) Number of extended attributes: 0 Total size of extended attribute data: 0 bytes Approximate overhead of extended HFS+ attributes: 0 bytes Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 4344 bytes / 4 KiB > rm kkk > date > kkk > afsctool -cfvv -S -L kkk /Volumes/Debian/Users/bertin/work/src/new/Qt/kkk: Compression type: ZLIB in decmpfs xattr (3) File size (uncompressed; reported size by Mac OS 10.6+ Finder): 30 bytes / 0 KB (kilobytes, base-10) File size (compressed): 0 bytes Compression savings: 100.0% Number of extended attributes: 0 Total size of extended attribute data: 0 bytes Approximate overhead of extended attributes: 268 bytes Uncompressed file size reported in compressed header: 30 bytes
See the commit message of #8b89d8f20e8dbb3efddbbca1f9f7852318727d87 It looks like inodes have some kind of memory?!?! |
Repro
afsctool -c /Applications/PlayOnMac.app
(no errors)afsctool -d /Applications/PlayOnMac.app
(no errors)afsctool -c -T LZFSE /Applications/PlayOnMac.app
(errors!)Comparison
afsctool -c -T LZFSE /Applications/PlayOnMac.app
(no errors)Errors
The text was updated successfully, but these errors were encountered: