Add compatibility with Mac OS X and the BSDs#3
Conversation
The flag FNM_EXTMATCH for fnmatch() is a glibc extension. Make it a no-op when it's unavailable.
OS X has no l*xattr(), instead using *xattr() with a flag argument of XATTR_NOFOLLOW. Also, getxattr() and setxattr() take an extra argument for resource forks. Just use macros for l*xattr() which adapt to the different API.
Both for finding the number of processors, and for finding the amount of physical memory. Header #ifdef for sysinfo.h is mysteriously missing in unsquashfs.c . Use the exact same code as mksquashfs.c .
Replace with strdup() and free it manually.
Use sigwait() and alarm() for POSIX-compatible handling of SIGQUIT printing.
Clang compiles in C99 mode by default, where 'inline' means there should be another definition: http://clang.llvm.org/compatibility.html#inline Using 'static inline' instead works.
- Add endian macros to xattr.c - Move stdlib.h before squashfs_swap.h in read_xattrs.c
We need <sys/stat.h> for S_ISDIR and friends.
|
Just a heads up that this works beautifully on OS X 👍 |
|
👍 from me, too. Just tag me, if you need some testing/debugging |
Add two patches: First patch is essentially from plougher/squashfs-tools#3 and gives a fallback for missing FNM_EXTMATCH. The second patch is from plougher/squashfs-tools#9 and adds the missing include for the S_IF* constants. Fixes: #2922 Gentoo-Bug: https://bugs.gentoo.org/600934 (cherry picked from commit 5606540) Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Update unsquash-4.c
There seems to be a stack overflow in read_fragment_table_4 at via what seems to be an integer overflow. Still looking into this problem, it seems like two or three different problems combined.
The first problem overflows the bytes variable, so that the allocation is enormous.
```c
int bytes = SQUASHFS_FRAGMENT_BYTES(sBlk.s.fragments);
```
If we fix this by making the variable size_t, we run into an unrelated problem in which the stack VLA allocation of fragment_table_index can easily exceed RLIMIT_STACK.
```c
long long fragment_table_index[indexes];
```
In the case of my system, the RLIMIT_STACK is 8388608, and VLA is asking for 15728648. Plus the stack probably already has a bunch of other things. This is what I believe ultimately leads to the stack overflow.
Afterwards, the heap allocation seems to succeed, and the disastrous call to read_fs_bytes is made, which initiates transfer from the squashfs image to the stack. At this stage, a stack overflow appears to be in full effect.
```c
res = read_fs_bytes(fd, sBlk.s.fragment_table_start,
SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk.s.fragments),
fragment_table_index);
```
This problem is also present in other read_fragment_table_N functions, and in the original squashfs-tools.
```
Parallel unsquashfs: Using 8 processors
ASAN:SIGSEGV
=================================================================
==8221==ERROR: AddressSanitizer: stack-overflow on address 0x7ffef3ae9608 (pc 0x000000559011 bp 0x7ffef49e9670 sp 0x7ffef3ae9610 T0)
#0 0x559010 in read_fragment_table_4 /home/septimus/vr/squashfs-vr/squashfs-tools/unsquash-4.c:40:9
plougher#1 0x525073 in main /home/septimus/vr/squashfs-vr/squashfs-tools/unsquashfs.c:2763:5
plougher#2 0x7fb56c533a3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20a3f)
plougher#3 0x418468 in _start (/home/septimus/vr/squashfs-vr/squashfs-tools/unsquashfs+0x418468)
SUMMARY: AddressSanitizer: stack-overflow /home/septimus/vr/squashfs-vr/squashfs-tools/unsquash-4.c:40:9 in read_fragment_table_4
==8221==ABORTING
```
Perhaps we should avoid using VLA altogether, and allocate fragment_table_index to the heap?
This pull request is an example implementation of the fix for unsquash-4, but I don't have enough test vectors to verify it will not break anything.
Install squashfs-tools on FreeBSD - we cannot use ci/mksquashfs.sh as upstream squashfs-tools does not build out of the box on FreeBSD[1], but it is available in FreeBSD's ports collection and thus as a binary package. Add a test_script that runs "make check". [1] plougher/squashfs-tools#3
Install squashfs-tools on FreeBSD - we cannot use ci/mksquashfs.sh as upstream squashfs-tools does not build out of the box on FreeBSD[1], but it is available in FreeBSD's ports collection and thus as a binary package. Add a test_script that runs "make check". [1] plougher/squashfs-tools#3
|
I have a correspondence with Vasi regarding this which goes back quite a few years. I will not accept pull-requests for things which I personally cannot test and verify the correctness. If I accept code which adds compatibility with Mac OS X and the BSDs, then I assume responsibility for it's correctness now and ever-more. Which means on every release I will have to test against these systems. I cannot do that as I don't run any Mac OS X or BSD system. Vasi runs a separate project which adds these patches. As the whole raison d'etre for his project is that I won't accept these patches, I don't see why he wants me to include them. |
You don't have to run a macOS or BSD system yourself - tests can be run automatically across Linux, FreeBSD, macOS, etc. using hosted CI such as Cirrus-CI. |
|
@emaste Yeah, CI is totally an option, and what I do for squashfuse: https://github.com/vasi/squashfuse/blob/master/.cirrus.yml I'd welcome update PRs and CI PRs to https://github.com/vasi/squashfs-tools, or you could maintain a fork yourself! (It's totally legitimate for plougher to only accept the patches he wants. I've rejected a number of PRs on my projects when they weren't directionally aligned, it's part of open source.) |
|
@vasi absolutely. My point is just that if the lack of macOS or BSD systems is the limiting factor it''s easily overcome. |
6 participants
Makes unsquashfs and mksquashfs compile and run on Mac OS X 10.7 and higher, as well as the most recent versions of FreeBSD, OpenBSD, NetBSD and DragonflyBSD. They continue to work on Linux as well, of course.