UPX compressed arm64/amd64/x86_64 executable crashes on macOS Ventura 13.0

[chchwy]

This issue tracker is ONLY used for reporting bugs.
Please use stackoverflow for supporting issues.

What's the problem (or question)?

The UPX compressed amd64 executable crashes on macOS 13.0.
I made a small CMake project which merely print out a few lines of text for reproducing the crash.

Here are the commands I used to compress the executable. Then it crashed immediately after I ran the compressed one.

% upx -vk ./HelloUPX4 -o./HelloUPX4-compressed
Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
     82304 ->     12304   14.95%   macho/amd64   HelloUPX4-compressed          

Packed 1 file.

% ./HelloUPX4-compressed 
zsh: segmentation fault  ./HelloUPX4-compressed
%

lldb

% lldb ./HelloUPX4-compressed 
(lldb) target create "./HelloUPX4-compressed"
Current executable set to '/Users/matt/HelloUPX4/build/Debug/HelloUPX4-compressed' (x86_64).
(lldb) run
Process 14499 launched: '/Users/matt/HelloUPX4/build/Debug/HelloUPX4-compressed' (x86_64)
Process 14499 stopped
* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x21090c000)
    frame #0: 0x000000021090c000
error: memory read failed for 0x21090c000
Target 0: (HelloUPX4-compressed) stopped.
(lldb) exit

What should have happened?

It should print the same text as before it was compressed.

This is the output when it is not compressed.

% lldb "HelloUPX4"
(lldb) target create "HelloUPX4"
Current executable set to '/Users/evatu/Work/HelloUPX4/build/Debug/HelloUPX4' (x86_64).
(lldb) run
Process 14516 launched: '/Users/evatu/Work/HelloUPX4/build/Debug/HelloUPX4' (x86_64)
Hello UPX4!
Done.
Process 14516 exited with status = 0 (0x00000000) 

Do you have an idea for a solution?

Actually no, but It was working on macOS 12.

How can we reproduce the issue?

I made a tiny CMake project with 16 lines of code for this bug report. It is configured to generate an amd64 executable. (Not arm64)

1. Clone the project at https://github.com/chchwy/upx-macos13-crash and go to the folder
2. mkdir build
3. cd build
4. cmake .. -G Xcode
5. cmake --build .
6. upx -vk ./Debug/HelloUPX4 -o./Debug/HelloUPX4-compressed
7. ./Debug/HelloUPX4-compressed <= crashed here

Please tell us details about your environment.

• UPX version used (upx --version): 3.96 (I tried the latest devel4 branch and it doesn't work either)
• Host Operating System and version: macOS 13.0 Ventura
• Host CPU architecture: arm64
• Target Operating System and version: same
• Target CPU architecture: amd64 (or called x86_64)

[markus-oberhumer]

I fear we do not have enough info to support macOS 13 at this time - has Apple released any current kernel/libc/ld/... source code yet?

[markus-oberhumer]

Info from #613 : macOS 13 source code has been released https://github.com/apple-oss-distributions/xnu

[markus-oberhumer]

Unsurprislingly arm64 macos13 also doesn't work.

[jreiser]

@markus-oberhumer The comment ... doesn't work does not help. Please specify what you tried, and what happened: how did it not work? It is much better to say, "I ran steps 1 through 7 of the Description using MacOS 13.0, and got SIGSEGV." And for SIGSEGV it is even better to run under lldb, then at the crash use lldb to report the pc, register contents, and traceback. And of course Apple's vaunted truss or dtrace is totally useless because it demands a complete run-time environment. "Anything acceptable to execve" is too simple.

[markus-oberhumer]

$ uname -a
Darwin macm1.oberhumer.com 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64

$ wget https://github.com/upx/upx/releases/download/v4.0.0/upx-4.0.0-src.tar.xz
$ tar -xJf upx-4.0.0-src.tar.xz
$ make -C upx-4.0.0-src build/release-clang
$ make -C upx-4.0.0-src/build/release-clang test

[jreiser]

@chchwy On Apple M1 hardware, then MacOS 13.0 (Ventura) with Xcode 14.1 (Nov.1, 2022) is not ready for prime time:

## Download from Apple and install  Command_Line_Tools_for_Xcode_14.1.dmg
## Then:
% cmake .. -G Xcode
zsh: command not found: cmake
## Download and install HomeBrew:
% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   [[snip]]
% brew --version
Homebrew 3.6.10
Homebrew/homebrew-core (git revision 3e28f99040e; last commit 2022-11-13)
% brew install cmake
% cmake .. -G Xcode
CMake Error:
  Xcode 1.5 not supported.
% cmake --version
cmake version 3.24.3

Suggestions?

[jreiser]

@markus-oberhumer Your report omitted the observed behavior, so is not reproducible.

Digging deeply, I see

% lldb /Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed
(lldb) target create "/Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed"
Current executable set to '/Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed' (arm64).
(lldb) process launch -s -- --version
error: Bad executable (or shared library)

So probably the error is ENOEXEC or EBADEXEC. What you see?

[flyingbird]

Did some research on my intel x86-64 model.
crashed in "call upx_main"
bugs in "amd64-darwin.macho-fold.h":
Two address calls sub_369. the second call crashed in sub_369:

That's all i can do.
Thanks.

[flyingbird]

Did some research on my intel x86-64 model. crashed in "call upx_main" bugs in "amd64-darwin.macho-fold.h": Two address calls sub_369. the second call crashed in sub_369:

That's all i can do. Thanks.

@markus-oberhumer @jreiser

[chchwy]

@chchwy On Apple M1 hardware, then MacOS 13.0 (Ventura) with Xcode 14.1 (Nov.1, 2022) is not ready for prime time:

## Download from Apple and install  Command_Line_Tools_for_Xcode_14.1.dmg
## Then:
% cmake .. -G Xcode
zsh: command not found: cmake
## Download and install HomeBrew:
% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   [[snip]]
% brew --version
Homebrew 3.6.10
Homebrew/homebrew-core (git revision 3e28f99040e; last commit 2022-11-13)
% brew install cmake
% cmake .. -G Xcode
CMake Error:
  Xcode 1.5 not supported.
% cmake --version
cmake version 3.24.3

Suggestions?

@jreiser

Run the following command before cmake. Please specify the Xcode.app path to the one you installed.

sudo /usr/bin/xcode-select --switch /Applications/Xcode-14.1.0.app

Probably you also need to open the Xcode app for the license accepting stuff for the first time.

[ChadKillingsworth]

We're seeing the same behavior on Google Closure Compiler: google/closure-compiler-npm#265

Oddly enough, an amd64 binary run on a M1 Mac via Rosetta does not exhibit the issue. Only when the binary is run on a non-M1 Mac does this come up.

[markus-oberhumer]

@chenrui333 Sounds reasonable. Hopefully we will be able to support macOS 13+ at some future point...

[markus-oberhumer]

Packing macOS binaries is disabled in Release builds for now. Commit 4870765

[tomnific]

For what it's worth, I did some poking around (unrelated to y'all's project) - tightbeamc appears to be a proprietary preprocessor for .tightbeam.c and .tightbeam.h files (possibly comparable to MIG, see below).

I have found references to something called Tightbeam kernel runtime which seems to be related to a new (or upcoming?) Mach feature called Exclaves. There is a related entity called Conclaves, but these might be derivative of Exclaves.

Presently all the syscalls for Exclaves are stubbed out to simply return KERN_NOT_SUPPORTED. What are Exclaves? Frankly I haven't the slightest clue. They seem related to IOKit/DriverKit, and appear to have a lot of calls related to IPC/RPC (mach ports are the underlying structure still) - I have also found references to XRT alongside this.

Whatever Exclaves are, they're internal to Apple, at least for now. I'm not sure if they're active on current versions of *OS or not.

Luckily, they can be disabled. To do so, you need to remove ExclaveKit and ExclaveCore from SUPPORTED_PLATFORMS in makedefs/MakeInc.cmd. Then, either add TIGHTBEAMC=nothx to your make invocation or, from the same file, remove the following lines:

ifeq ($(origin TIGHTBEAMC),undefined)
	export TIGHTBEAMC := $(shell $(XCRUN) -sdk $(SDKROOT) -find tightbeamc)
endif

Now everything should behave as normal (however due to other, unrelated, errors I haven't managed to actually build the kernel yet - but installhdrs worked fine).

[markus-oberhumer]

@tomnific

Could you please provide exact instructions how you are building the xnu kernel?

I'm stuck at *** Could not determine xnu version from SDK or KDK! Set RC_DARWIN_KERNEL_VERSION environment variable.. Stop.

My setup on a Mac Mini M2, Darwin 22.6.0:

$ git describe
xnu-10002.1.13

$ xcodebuild -version
Xcode 14.3.1
Build version 14E300c

[tomnific]

Ah my b - I forgot about something else I had to add to my make invocation. Here's the full command I'm running:

make SDKROOT=macosx ARCH_CONFIGS="X86_64 ARM64" KERNEL_CONFIGS=RELEASE TIGHTBEAMC==nothx LOGCOLORS=y RC_DARWIN_KERNEL_VERSION=23.0.0

I borked my macOS SDK when I ran the same invocation with the addition of the installhdrs target, so for me, running the above failed due to not being able to build certain modules or find certain headers, etc.

I stopped there and haven't had time to try continuing onwards. The guide I'm following is a mish mash of the *OS internals book's instructions and the ones found at https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html

I had to make some adjustments, attached below are the notes I took about what has changed about the process.

Best of luck, and if you figure out how to compile it, please do share! Regardless, I'll be back when I figure out how to compile it.

---

Compiling XNU for Apple Silicon

Just adding diffs from https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html for now.

Install Dependencies

DTrace/ctf*

1. Download DTrace

git clone https://github.com/apple-oss-distributions/dtrace.git

2. Open dtrace/dtrace.xcodeproj

3. Add codesigning identity to ctf* subprojects

4. Allow terminal.app to modify apps (security & privacy)

5. Corrected ditto command:

sudo ditto "$PWD/dst/" "$TOOLCHAIN"

AvailabilityVersions

1. Download AvailabilityVersions

git clone https://github.com/apple-oss-distributions/AvailabilityVersions.git

2. Install the Ninja build system

brew install ninja

3. Possible: run make before make install

4. make dst/usr/local/libexec/availability.pl executable

Install XNU Headers

1. Remove ExclaveKit and ExclaveCore from SUPPORTED_PLATFORMS in makedefs/MakeInc.cmd
2. Add TIGHTBEAMC=nothx to the make arguments

[markus-oberhumer]

Many thanks for your info!

I'm thinking about creating a GitHub Actions workflow that clones the apple-oss-distributions repositories, applies necessary patches and then builds the XNU kernel - running under GitHub macos-13.

But then I'm sure somebody else must have had this idea. Any hints?

[markus-oberhumer]

I've created https://github.com/upx/upx-fork-apple-oss-distributions-xnu , please continue XNU build discussions there.

[tomnific]

I've never used GitHub Actions before - so unfortunately I can't help you there. I'll add updates to the XNU fork as I figure things out though.

[markus-oberhumer]

@tomnific Well, GitHub Actions are basically shell scripts that are run in a container (more technically its a YAML file that is parsed by NodeJS which then generates and runs the shell scripts).

Please visit upx/upx-fork-apple-oss-distributions-xnu#1
and have a look at https://github.com/upx/upx-fork-apple-oss-distributions-xnu/actions/runs/7021092042
and https://github.com/upx/upx-fork-apple-oss-distributions-xnu/blob/master/.github/workflows/build-xnu-macos-13.yml

[61bcdefg]

I guess zsh: killed appears because the header of a non-x86_64 architecture executable must contains the MH_DYLDLINK flag and the mach-o must contains LC_LOAD_DYLINKER command (https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L1265, https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L1466).

I wrote a script to verify this

#!/usr/bin/env python3

import argparse
import lief

binary = None

# https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L731
def pie_required(exectype, execsubtype):
    if exectype == lief.MachO.CPU_TYPES.x86_64:
        return False
    elif exectype == lief.MachO.CPU_TYPES.ARM64:
        return True
    elif exectype == lief.MachO.CPU_TYPES.ARM and execsubtype == 12: # CPU_SUBTYPE_ARM_V7K
        return True
    else:
        return False

def add_dyldlink_flag(binary):
    if isinstance(binary, lief.MachO.FatBinary):
        for sub_binary in fat_binary:
            header = sub_binary.header
            if pie_required(header.cpu_type, header.cpu_subtype) and header.has(lief.MachO.HEADER_FLAGS.DYLDLINK) == False:
                header.add(lief.MachO.HEADER_FLAGS.DYLDLINK)
    else:
        header = binary.header
        if pie_required(header.cpu_type, header.cpu_subtype) and header.has(lief.MachO.HEADER_FLAGS.DYLDLINK) == False:
            header.add(lief.MachO.HEADER_FLAGS.DYLDLINK)

def add_LC_LOAD_DYLINKER(binary):
    load_command = lief.MachO.DylinkerCommand("/usr/lib/dyld")
    if isinstance(binary, lief.MachO.FatBinary):
        for sub_binary in fat_binary:
            header = sub_binary.header
            if pie_required(header.cpu_type, header.cpu_subtype) and sub_binary.has(lief.MachO.LOAD_COMMAND_TYPES.LOAD_DYLINKER) == False:
                sub_binary.add(load_command)
    else:
        header = binary.header
        if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.LOAD_DYLINKER) == False:
            binary.add(load_command)

parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input", help="input file", type=str, required=True)
parser.add_argument("-o", "--output", help="output file", type=str, required=True)

args = parser.parse_args()

input_file = args.input
output_file = args.output

binary = lief.parse(input_file)

add_dyldlink_flag(binary)
add_LC_LOAD_DYLINKER(binary)

binary.write(output_file)

This script add MH_DYLDLINK and LC_LOAD_DYLINKER to the executable, the zsh: killed problem was solved, but there was another problem

zsh: segmentation fault

lldb

Process 57821 stopped
* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x48)
    frame #0: 0x0000000183c037a0 dyld`dyld3::MachOAnalyzer::forEachRebase_Relocations(Diagnostics&, dyld3::MachOLoaded::LinkEditInfo const&, dyld3::MachOFile::SegmentInfo const*, void (char const*, dyld3::MachOLoaded::LinkEditInfo const&, dyld3::MachOFile::SegmentInfo const*, bool, unsigned int, unsigned char, unsigned long long, dyld3::MachOAnalyzer::Rebase, bool&) block_pointer) const + 116
dyld`dyld3::MachOAnalyzer::forEachRebase_Relocations:
->  0x183c037a0 <+116>: ldr    w2, [x8, #0x48]
    0x183c037a4 <+120>: mov    x0, x20
    0x183c037a8 <+124>: mov    x1, x24
    0x183c037ac <+128>: bl     0x183bf75e4               ; dyld3::MachOLoaded::getLinkEditContent(dyld3::MachOLoaded::LayoutInfo const&, unsigned int) const
Target 0: (test_upx) stopped.

[jreiser]

@61bcdefg Thank you for the URLs and script! That is progress in understanding the requirements of MacOS.

At first glance, it looks like the site of the SIGSEGV, namely dyld:dyld3::MachOAnalyzer::forEachRebase_Relocations, is trying to rebase a table of relocations, but there is no such table in the specific executable. This is typical of Apple code, which assumes that any executable is a full-blown result of an XCode project with all the bells and whistles: symbol table, relocations, libraries, two-level relocations, linker info, etc. Even the vaunted DTrace is sh*t because it assumes much too much, and as a result cannot process the simplest executable (a two-instruction program that performs a "bare" system call to exit(0); with no symbols, no libraries, no dylinker, etc.)

UPX would rather not construct a compressed executable with all that baggage: it occupies space and could easily interfere with the de-compressed program. What if the dylinker used by UPX is not the same as the one used by the de-compressed program, what if the system library does not understand being "initialized" twice (once by the UPX de-compresion stub, once by the de-compressed program), etc.

Because of the current murkiness involving instantiation and initialization that is performed by execve, now I am considering "just" intercepting the initialization of the executable's static constructors, after all the "system" machinations with dylinker, symbol tables, etc. Perhaps it will be possible to use only {mmap, munmap, fetch and store to memory} to de-compress the LC_SEGMENT_64 appropriately.

[Jeremy5909]

Is there a good alternative until this is fixed?

[jreiser]

There is no alternative known to us. The Apple documentation known to us is not specific enough to implement against (what are the exact requirements for execve so succeed?) The best we can do is trial-and-error, which is slow and frustrating. The Apple open-source kernel does not build, so cannot be used as a hermeneutic for discovery.

[jreiser]

Well, if the only thing that matters is small download size: compress the executable using your favorite file compressor, and make a shell archive (such as shar) containing the result. The shell de-compresses info a temporary file, and exec that file with the original parameters to the script. The drawbacks are: need a place in the filesystem for the file, need a name for the file, need storage space for the contents of the file, the file must have eXecute permission and must be executable (mounting a filesystem with option noexec defeats this, and other MacOS properties and permissions probablly apply), must handle deleting the file eventually, the de-compressor must be available, probably requires more execution time.

[61bcdefg]

The Apple open-source kernel does not build, so cannot be used as a hermeneutic for discovery.

Have you ever try https://github.com/blacktop/darwin-xnu-build