Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compatibility with Mac OS X and the BSDs #3

Closed
wants to merge 9 commits into from
Closed

Conversation

@vasi
Copy link

@vasi vasi commented Jun 2, 2014

Makes unsquashfs and mksquashfs compile and run on Mac OS X 10.7 and higher, as well as the most recent versions of FreeBSD, OpenBSD, NetBSD and DragonflyBSD. They continue to work on Linux as well, of course.

vasi added 9 commits Mar 19, 2012
The flag FNM_EXTMATCH for fnmatch() is a glibc extension. Make it a no-op when it's unavailable.
OS X has no l*xattr(), instead using *xattr() with a flag argument of XATTR_NOFOLLOW. Also, getxattr() and setxattr() take an extra argument for resource forks.

Just use macros for l*xattr() which adapt to the different API.
Both for finding the number of processors, and for finding the
amount of physical memory.

Header #ifdef for sysinfo.h is mysteriously missing in unsquashfs.c . Use the
exact same code as mksquashfs.c .
Replace with strdup() and free it manually.
Use sigwait() and alarm() for POSIX-compatible handling of SIGQUIT
printing.
Clang compiles in C99 mode by default, where 'inline' means there should be
another definition: http://clang.llvm.org/compatibility.html#inline

Using 'static inline' instead works.
- Add endian macros to xattr.c
- Move stdlib.h before squashfs_swap.h in read_xattrs.c
We need <sys/stat.h> for S_ISDIR and friends.
@DomT4
Copy link

@DomT4 DomT4 commented Sep 30, 2014

Just a heads up that this works beautifully on OS X 👍

@timbru31
Copy link

@timbru31 timbru31 commented Dec 22, 2014

👍 from me, too.
Would love to see this in the official codebase.

Just tag me, if you need some testing/debugging

gentoo-bot pushed a commit to gentoo/gentoo that referenced this pull request Feb 3, 2017
Add two patches: First patch is essentially from

plougher/squashfs-tools#3

and gives a fallback for missing FNM_EXTMATCH.

The second patch is from

plougher/squashfs-tools#9

and adds the missing include for the S_IF* constants.

Fixes: #2922
Gentoo-Bug: https://bugs.gentoo.org/600934
(cherry picked from commit 5606540)
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
@develar
Copy link

@develar develar commented Mar 5, 2018

@vasi Could you please resolve merge conflict?
@plougher Why this PR is not yet merged? Could you please comment?

mingwandroid added a commit to mingwandroid/squashfs-tools that referenced this pull request Jul 11, 2019
Update unsquash-4.c

There seems to be a stack overflow in read_fragment_table_4 at via what seems to be an integer overflow. Still looking into this problem, it seems like two or three different problems combined.

The first problem overflows the bytes variable, so that the allocation is enormous.
```c
int bytes = SQUASHFS_FRAGMENT_BYTES(sBlk.s.fragments);
```

If we fix this by making the variable size_t, we run into an unrelated problem in which the stack VLA allocation of fragment_table_index can easily exceed RLIMIT_STACK.
```c
long long fragment_table_index[indexes];
```

In the case of my system, the RLIMIT_STACK is 8388608, and VLA is asking for 15728648. Plus the stack probably already has a bunch of other things. This is what I believe ultimately leads to the stack overflow.

Afterwards, the heap allocation seems to succeed, and the disastrous call to read_fs_bytes is made, which initiates transfer from the squashfs image to the stack. At this stage, a stack overflow appears to be in full effect.

```c
 res = read_fs_bytes(fd, sBlk.s.fragment_table_start,
         SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk.s.fragments),
         fragment_table_index);
```
This problem is also present in other read_fragment_table_N functions, and in the original squashfs-tools.

```
Parallel unsquashfs: Using 8 processors
ASAN:SIGSEGV
=================================================================
==8221==ERROR: AddressSanitizer: stack-overflow on address 0x7ffef3ae9608 (pc 0x000000559011 bp 0x7ffef49e9670 sp 0x7ffef3ae9610 T0)
    #0 0x559010 in read_fragment_table_4 /home/septimus/vr/squashfs-vr/squashfs-tools/unsquash-4.c:40:9
    plougher#1 0x525073 in main /home/septimus/vr/squashfs-vr/squashfs-tools/unsquashfs.c:2763:5
    plougher#2 0x7fb56c533a3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20a3f)
    plougher#3 0x418468 in _start (/home/septimus/vr/squashfs-vr/squashfs-tools/unsquashfs+0x418468)
SUMMARY: AddressSanitizer: stack-overflow /home/septimus/vr/squashfs-vr/squashfs-tools/unsquash-4.c:40:9 in read_fragment_table_4
==8221==ABORTING
```

Perhaps we should avoid using VLA altogether, and allocate fragment_table_index to the heap?
This pull request is an example implementation of the fix for unsquash-4, but I don't have enough test vectors to verify it will not break anything.
blake-riley added a commit to blake-riley/homebrew-core that referenced this pull request Sep 3, 2019
1. Source has moved from SourceForge to GitHub (latest release on SourceForge is 4 years old).
   The upstream author has released v4.4 on GitHub.
   Update homepage, url, head to reflect this.
2. Add patch to work with v4.4: the existing patch works with the SourceForge 4.3 release, but fails to match on the GitHub 4.4 release.
   The updated patch is a rebase of a long-standing PR (plougher/squashfs-tools#3) onto the 4.4 tag, solving conflicts.
3. Update doc.install targets (some of the old ones have disappeared).
blake-riley added a commit to blake-riley/homebrew-core that referenced this pull request Sep 3, 2019
1. Source has moved from SourceForge to GitHub (latest release on SourceForge is 4 years old).
   The upstream author has released v4.4 on GitHub.
   Update homepage, url, head to reflect this.
2. Add patch to work with v4.4: the existing patch works with the SourceForge 4.3 release, but fails to match on the GitHub 4.4 release.
   The updated patch is a rebase of a long-standing PR (plougher/squashfs-tools#3) onto the 4.4 tag, solving conflicts.
3. Update doc.install targets (some of the old ones have disappeared).
blake-riley added a commit to blake-riley/homebrew-core that referenced this pull request Sep 25, 2019
emaste added a commit to emaste/squashfuse that referenced this pull request Feb 28, 2020
Install squashfs-tools on FreeBSD - we cannot use ci/mksquashfs.sh as
upstream squashfs-tools does not build out of the box on FreeBSD[1],
but it is available in FreeBSD's ports collection and thus as a binary
package.

Add a test_script that runs "make check".

[1] plougher/squashfs-tools#3
emaste added a commit to emaste/squashfuse that referenced this pull request Feb 28, 2020
Install squashfs-tools on FreeBSD - we cannot use ci/mksquashfs.sh as
upstream squashfs-tools does not build out of the box on FreeBSD[1],
but it is available in FreeBSD's ports collection and thus as a binary
package.

Add a test_script that runs "make check".

[1] plougher/squashfs-tools#3
@plougher
Copy link
Owner

@plougher plougher commented Oct 29, 2020

I have a correspondence with Vasi regarding this which goes back quite a few years.

I will not accept pull-requests for things which I personally cannot test and verify the correctness.

If I accept code which adds compatibility with Mac OS X and the BSDs, then I assume responsibility for it's correctness now and ever-more. Which means on every release I will have to test against these systems. I cannot do that as I don't run any Mac OS X or BSD system.

Vasi runs a separate project which adds these patches. As the whole raison d'etre for his project is that I won't accept these patches, I don't see why he wants me to include them.

@plougher plougher closed this Oct 29, 2020
@emaste
Copy link

@emaste emaste commented Oct 29, 2020

I cannot do that as I don't run any Mac OS X or BSD system.

You don't have to run a macOS or BSD system yourself - tests can be run automatically across Linux, FreeBSD, macOS, etc. using hosted CI such as Cirrus-CI.

@vasi
Copy link
Author

@vasi vasi commented Oct 30, 2020

@emaste Yeah, CI is totally an option, and what I do for squashfuse: https://github.com/vasi/squashfuse/blob/master/.cirrus.yml I'd welcome update PRs and CI PRs to https://github.com/vasi/squashfs-tools, or you could maintain a fork yourself!

(It's totally legitimate for plougher to only accept the patches he wants. I've rejected a number of PRs on my projects when they weren't directionally aligned, it's part of open source.)

@emaste
Copy link

@emaste emaste commented Nov 3, 2020

@vasi absolutely. My point is just that if the lack of macOS or BSD systems is the limiting factor it''s easily overcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.