Rebase on top of squashfs-tools-4.5.1

[vlaci]

I've opened this PR so that we have a clear view what changes we have on upstream. We can decide to change active branches (as discussed in #11) later.

Done

• Rebased sasquatch onto upstream's 4.5.1 tag

  details

  $ git fetch https://github.com/plougher/squashfs-tools 4.5.1
  remote: Enumerating objects: 12036, done.
  remote: Counting objects: 100% (1/1), done.
  remote: Total 12036 (delta 0), reused 0 (delta 0), pack-reused 12035
  Receiving objects: 100% (12036/12036), 2.57 MiB | 3.99 MiB/s, done.
  Resolving deltas: 100% (8950/8950), done.
  From https://github.com/plougher/squashfs-tools
   * tag               4.5.1      -> FETCH_HEAD
  $ git push origin FETCH_HEAD:squashfs-tools-v4.5.1
  Enumerating objects: 1, done.
  Counting objects: 100% (1/1), done.
  Writing objects: 100% (1/1), 170 bytes | 170.00 KiB/s, done.
  Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
  To github.com:onekey-sec/sasquatch
   * [new tag]         FETCH_HEAD -> squashfs-tools-v4.5.1
  $ git rebase --rebase-merges=rebase-cousins squashfs-tools-v4.5.1

• Removed macOS conditional includes and let Nix include their own. This patch originates fron homebrew as it looks more complete

• Tested unblob on x86_64-linux and aarch64-linux

Todo

• Test on darwin
  https://github.com/onekey-sec/unblob/actions/runs/4298778428/jobs/7493287205
• Should we apply this patch only during nix build to limit the exposure of linux builds?

git range-diff squashfs-tools-v4.5..origin/sasquatch squashfs-tools-v4.5.1..origin/sasquatch-patches-rebase

  1:  2399660 <   -:  ------- README: Update after release
  2:  19b161c <   -:  ------- Mksquashfs: fix duplicate check when last file block is sparse
  3:  e12106c <   -:  ------- Makefile: we're now in the post release development phase
  4:  ebf5417 <   -:  ------- README: add info about release bug
  5:  f5c908e <   -:  ------- Avoid use of INSTALL_DIR for symlink targets
  6:  5f56f67 <   -:  ------- Don't sanity check fragment table location if there are no fragments
  7:  80b8441 <   -:  ------- unsquashfs: use squashfs_closedir() to delete directory
  8:  1993a4e <   -:  ------- unsquashfs: dynamically allocate name
  9:  9938154 <   -:  ------- unsquashfs: use linked list to store directory names
 10:  e048580 <   -:  ------- Unsquashfs: additional write outside destination directory exploit fix
 11:  19fcc93 <   -:  ------- Unsquashfs: Add makefile entry for unsquash-12.o
 12:  e011d1b <   -:  ------- Unsquash-3: Fix segmentation fault in read_inode()
 13:  f75390b <   -:  ------- Update Version when using git-archive to create zip (or tar) file
 14:  acd68ed <   -:  ------- Unsquashfs: release mutex in error situation in read_fs_bytes()
 15:  d9d8e08 <   -:  ------- Unsquashfs: release mutex in error situation in read_fs_bytes() V2
 16:  f151730 <   -:  ------- Unsquashfs: don't use max open file limit if larger than caches
 17:  7a09899 <   -:  ------- Mksquashfs: add new "expert" category to options text
 18:  5ae7238 <   -:  ------- Sqfstar: add new "expert" category to options text
 19:  9f62da6 <   -:  ------- xattrs: move check for xattrs outside read_xattrs_from_disk()
 20:  2556d39 <   -:  ------- xattrs: rename "flag" to "sanity_only"
 21:  ef3e0f0 <   -:  ------- Unsquashfs: don't abort if there's unsupported xattrs in the filesystem
 22:  2390387 <   -:  ------- Mksquashfs: improve xattr reading unsupported message
 23:  d5a583e <   -:  ------- READ: update to reflect current status
 24:  8e0ac4a <   -:  ------- mksquashfs: add missing full stop to help text
 25:  482b9d5 <   -:  ------- mksquashfs: remove full stop from help text
 26:  97543bc <   -:  ------- Mksquashfs: move pseudo defs and add more info in help text
 27:  11c9591 <   -:  ------- Mksquashfs: add "See also" subsection to help text
 28:  792b561 <   -:  ------- Mksquashfs: improve help text entry for -all-time
 29:  a5b5684 <   -:  ------- Mksquashfs: improve help text entry for -mkfs-time
 30:  8f468f3 <   -:  ------- Mksquashfs: add new Environment sub-section to help text
 31:  7fef071 <   -:  ------- Mksquashfs: update copyright year in -version output
 32:  5f66b8b <   -:  ------- Makefile: get rid of obsolete variable
 33:  254c910 <   -:  ------- mksquashfs-manpage.sh: add script to generate manpage
 34:  8a9d02e <   -:  ------- manpages: add manpage for Mksquashfs
 35:  49a5121 <   -:  ------- Makefile: add install section
 36:  25bce9a <   -:  ------- manpages: generate and install manpages with "make install"
 37:  d61eb68 <   -:  ------- manpages: don't use git to get toplevel of source (source root)
 38:  65f21f5 <   -:  ------- manpags: get rid of mandb update
 39:  7f9203e <   -:  ------- Revert some unintended updates
 40:  0cab25a <   -:  ------- lzma_wrapper: add usage function
 41:  479e488 <   -:  ------- lzma_xz_wrapper: add usage function
 42:  bd186a7 <   -:  ------- mksquashfs-manpage.sh: deal with the deprecated lzma compressor
 43:  98f504f <   -:  ------- Unsquashfs: tweak help text to make a better manpage
 44:  dad162d <   -:  ------- Unsquashfs: more improvements to the help text output
 45:  9e46a75 <   -:  ------- unsquashfs-manpage.sh: add script to generate manpage
 46:  589378a <   -:  ------- manpages: add manpage for Unsquashfs
 47:  c24cc61 <   -:  ------- Sqfstar: tweak help text to make a better manpage
 48:  f486330 <   -:  ------- mksquashfs: some more tweaks to the help text
 49:  67328d3 <   -:  ------- sqfstar: fix missing space
 50:  3a4f624 <   -:  ------- sqfstar-manpage.sh: add script to generate manpage
 51:  0425d3d <   -:  ------- manpages: add manpage for Sqfstar
 52:  dd4f15a <   -:  ------- sqfstar-manpage.sh: fix after replacing Synopsis section
 53:  709ba5c <   -:  ------- Update sqfstar.1 manpage
 54:  3c8fe4f <   -:  ------- Sqfscat: tweak help text to make a better manpage
 55:  037ebe7 <   -:  ------- sqfscat-manpage.sh: add script to generate manpage
 56:  e7e96fe <   -:  ------- manpages: add manpage for Sqfscat
 57:  9ceec04 <   -:  ------- Unsquashfs: make read_fs_bytes() match Mksquashfs
 58:  9c257c6 <   -:  ------- Mksquashfs: quieten static analyser
 59:  a8f61e2 <   -:  ------- Mksquashfs: check return result of ftruncate()
 60:  e21f11b <   -:  ------- Mksquashfs: quieten bogus static analyser use after free error
 61:  aca6c55 <   -:  ------- mksquashfs: limit the maximum cpiostyle filename to PATH_MAX
 62:  d22f4ce <   -:  ------- Mksquashfs: quieten static analyser use after free error
 63:  ab0784a <   -:  ------- sqfstar: initialise inode->tar_file to NULL if automatic directory
 64:  5f29c14 <   -:  ------- Sqfstar: fix leak of one file_buffer when EOF hit
 65:  de944c3 <   -:  ------- Sqfstar: fix leak of one tar_file when EOF hit
 66:  f0d228b <   -:  ------- sqfstar: Fix buffer overflow detected by ASAN build
 67:  1c2c2f4 <   -:  ------- Sqfstar: fix memory leak detected by ASAN build
 68:  f491ad8 <   -:  ------- Sqfstar: fix another leak detected by ASAN build
 69:  b978026 <   -:  ------- Sqfstar: Fix uninitialised value detected by MSAN build
 70:  b490a81 <   -:  ------- Sqfstar: Fix another uninitialised value detected by MSAN build
 71:  1fb5a95 <   -:  ------- Mksquashfs: check for read_bytes() error return
 72:  9662e94 <   -:  ------- Sqfstar: fix read_bytes() error return check
 73:  b89fffa <   -:  ------- Sqfstar: implement -no-hardlinks option
 74:  75951fb <   -:  ------- Sqfstar: Fix another uninitialised value detected by MSAN build
 75:  85c595b <   -:  ------- Unsquash-4: Fix out of bounds access detected by ASAN build
 76:  cae6152 <   -:  ------- Unsquash-4: Add check for corrupted inode gid value
 77:  12b4c04 <   -:  ------- Unsquash-3: Add check for corrupted inode uid and gid values
 78:  7d74db6 <   -:  ------- Unsquash-2: Add check for corrupted inode uid and gid values
 79:  98fed84 <   -:  ------- Unsquash-1: Add check for corrupted inode uid and gid values
 80:  fc7bd31 <   -:  ------- Unsquash-4: Fix out of bounds write detected by ASAN build
 81:  b99483f <   -:  ------- Unsquashfs: These are indexes not counts
 82:  e8b86fd <   -:  ------- Mksquashfs appending: Add check for corrupted inode uid and gid values
 83:  7a7ad9c <   -:  ------- Unsquash-4: Check for corrupted inode type
 84:  f3783bb <   -:  ------- Unsquash-4: Check for corrupted inode fragment index
 85:  24dfd28 <   -:  ------- Sqfstar: Fix potential unsigned char comparison with -128
 86:  9e1ad12 <   -:  ------- Unsquashfs: check for xattr_ids of 0 when reading xttrs
 87:  8cbadee <   -:  ------- Xattrs: read_xattrs_from_disk() return result should be unsigned
 88:  7c3632f <   -:  ------- Xattrs: Improve error messages in read_xattrs_from_disk()
 89:  8e7e85d <   -:  ------- Unsquash-4: Add check for corrupted inode xattr value
 90:  bc0c097 <   -:  ------- Xattr: Check for incorrect xattr value in xattr_id metadata
 91:  62e7cca <   -:  ------- Mksquashfs: Fix Mksquashfs silently ignoring pseudo files on appending
 92:  c9ea0a1 =   1:  cd2c89e add LZMA SDK 4.65
 93:  337d286 =   2:  ad2952c lzma465: modified COMPRESS_MF_MT mtMode calculation
 94:  4019be4 =   3:  5b53f01 add LZMA SDK 4.32 as lzmadaptive
 95:  b2a0524 =   4:  84f6038 add LZMA_lib to lzmadaptive
 96:  c53b465 =   5:  1a9a786 lzmadaptive fixes
 97:  d44c639 =   6:  f08694c add lzmalt implementation
 98:  f3c9a16 =   7:  27f41aa add GPL license file and README
 99:  0f339c1 =   8:  e0a4fa3 add .gitignore
100:  ec17fc5 =   9:  48957c0 uninlined compressor_uncompress
101:  2a2a3ac !  10:  9272593 Added additional LZMA decompressors
    @@ squashfs-tools/compressor.c: static struct compressor zstd_comp_ops = {
      static struct compressor unknown_comp_ops = {
      	0, "unknown"
      };
    -@@ squashfs-tools/compressor.c: static struct compressor unknown_comp_ops = {
    - struct compressor *compressor[] = {
    - 	&gzip_comp_ops,
    +@@ squashfs-tools/compressor.c: struct compressor *compressor[] = {
    + 	&xz_comp_ops,
    + 	&zstd_comp_ops,
      	&lzma_comp_ops,
     +    // CJH: Added additional LZMA decompressors. Order is intentional.
     +    &lzma_adaptive_comp_ops,
     +    &lzma_alt_comp_ops,
     +    &lzma_wrt_comp_ops,
    - 	&lzo_comp_ops,
    - 	&lz4_comp_ops,
    - 	&xz_comp_ops,
    + 	&unknown_comp_ops
    + };
    + 
     
      ## squashfs-tools/lzma_wrapper.c ##
     @@
102:  16189bc =  11:  e3cf86f Add LZO and XZ support
103:  3a584ac =  12:  06d8ade Do not consider error when LZMA size field is omitted from header
104:  175ddc2 =  13:  2ef7d83 Decompression errors are displayed in one place and try to continue with errors
105:  22ea6fc <   -:  ------- add missing include
106:  a778a1c =  14:  3f1bce2 Temporarily set the default processor count to 1 to prevent threading bug
107:  063d175 !  15:  1f72460 Perform swap auto-detection
    @@ squashfs-tools/unsquashfs.c: struct super_block sBlk;
      
     -int bytes = 0, swap, file_count = 0, dir_count = 0, sym_count = 0,
     +int bytes = 0, swap = -1, file_count = 0, dir_count = 0, sym_count = 0,
    - 	dev_count = 0, fifo_count = 0, socket_count = 0;
    + 	dev_count = 0, fifo_count = 0, socket_count = 0, hardlnk_count = 0;
      struct hash_table_entry *inode_table_hash[65536], *directory_table_hash[65536];
      int fd;
     @@ squashfs-tools/unsquashfs.c: int check_compression(struct compressor *comp)
108:  c84a082 !  16:  d04539b Add override option
    @@ squashfs-tools/unsquashfs.c: struct super_block sBlk;
      
     +struct override_table override = { 0 };
      int bytes = 0, swap = -1, file_count = 0, dir_count = 0, sym_count = 0,
    - 	dev_count = 0, fifo_count = 0, socket_count = 0;
    + 	dev_count = 0, fifo_count = 0, socket_count = 0, hardlnk_count = 0;
      struct hash_table_entry *inode_table_hash[65536], *directory_table_hash[65536];
     
      ## squashfs-tools/unsquashfs.h ##
     @@ squashfs-tools/unsquashfs.h: struct directory_stack {
    - #define MAX_FOLLOW_SYMLINKS 256
    + #define MAXIMUM_READ_SIZE 0x7ffff000
      
      /* globals */
     +extern struct override_table override;
109:  58c342e =  17:  81d3b2b Add command line options for overrides & compression
110:  9454909 =  18:  5e48f06 Add sasquatch target and no longer build default mksquashfs and unsquashfs
111:  6986672 =  19:  a51806b Changed command line options to avoid conflict with new upsteam options
112:  5ea8914 =  20:  0b6cd37 add .vscode to gitignore
113:  5af8463 =  21:  3c4aadd Makefile: fixed cleanup to remove object files
114:  b5939b3 =  22:  2810fac fixed a segfault in squashfs v3 endinan swap
115:  db9da1c =  23:  fb518cd enable LZ4 & ZSTD support
116:  67f2d1e =  24:  6ce56c0 always read generic superblock so we can check MAGIC even when endianess is fixed
117:  85643b7 =  25:  0c65f00 support for v4 bigendian filesystems (#2)
118:  b26c5c7 =  26:  7440d2a debian packaging
119:  8b21b8f =  27:  ae51d69 add build workflow
120:  3a83d70 =  28:  bbc3c98 add build dependencies
121:  a207273 =  29:  0dd6d43 build deb package on bullseye
122:  f3677c2 =  30:  7e9ef41 upload artifacts
123:  835a34e =  31:  b5971cf upload without name
124:  20cb3b7 =  32:  73d6058 add release workflow
125:  41a7342 =  33:  97abc53 build deb packages on every push & pull_requests
126:  04d8ee0 =  34:  0106ba9 fix workflow
127:  5806c80 =  35:  a53f710 build release on sasquatch-v* tags
128:  27fbdb4 =  36:  63efd57 Add big endian support for mksquashfs.
129:  349c088 =  37:  49f4d7d Build Debian package for AARCH64 and ARM32 in Github CI/CD.
130:  e638c14 =  38:  8f2a30c nix: add flake based packaging
131:  32d34e6 =  39:  35bfc04 ci: adding test for nix packaging
132:  0dd51a9 =  40:  0834911 ci: schedule job according to comment
133:  a1c52ff =  41:  1ce504e nix: add shell where dependencies are in scope
134:  1166210 !  42:  354b9e8 nix: add flake based packaging
    @@ flake.nix
      
            # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
            forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
    +@@
    +           inherit version;
    + 
    +           src = ./.;
    +-          patches = [ ]; # Patches are already applied to input
    +         });
    + 
    +         sasquatch-be = final.sasquatch-le.overrideAttrs (super: {
135:  1e55eee =  43:  45f960e chore(deps): improve LZMA dependencies resolution during pre-processing.
136:  851c447 <   -:  ------- fix: adapt the code to compile on OSX using conditional macros.
137:  47b8a2d <   -:  ------- fix: fix potential type casting issue.
138:  4c7cfe2 <   -:  ------- fix: fix use of undeclared sysctl identifiers on OSX.
139:  0a13ee6 =  44:  c3265ca fix: remove hardcoded gcc/g++ references.
140:  eb6caad =  45:  64e360b nix: build on OSX in ci
  -:  ------- >  46:  aa648c4 nix: pin version to work-around regression

---

Resolves #11