upx output of binary 'go' file can not run on macOS #222

hr3lxphr6j · 2018-09-18T07:26:38Z

What's the problem (or question)?

upx output binary file can not run on macOS

What should have happened?

package main

import "fmt"

func main() {
	fmt.Println("Hello World")
}

$ go build go-test.go

$ ./go-test
Hello World

$ upx go-test
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   2003464 ->   1007632   50.29%   macho/amd64   go-test

Packed 1 file.

$ ./go-test
[1]    9461 killed     ./go-test

Do you have an idea for a solution?

How can we reproduce the issue?

Please tell us details about your environment.

UPX version used (upx --version):

upx 3.95
UCL data compression library 1.03
zlib data compression library 1.2.11
LZMA SDK version 4.43

Host Operating System and version: macOS High Sierra 10.13.6
Host CPU architecture: x86_64
Target Operating System and version: macOS High Sierra 10.13.6
Target CPU architecture: x86_64

The text was updated successfully, but these errors were encountered:

joshdk · 2018-09-19T01:39:44Z

I can confirm that I am also seeing this issue. It appears to be an issue with the official 3.95 release, as I do not see the failure using the official 3.94 release.

To reproduce I built a simple Docker image (using the following Dockerfile with macos$ docker build -t upx-macos-repro .) that installs the 3.94 and 3.95 UPX releases:

FROM golang:1.10.4

RUN apt update \
 && apt install xz-utils \
 && cd /tmp \
 && wget --quiet https://github.com/upx/upx/releases/download/v3.94/upx-3.94-amd64_linux.tar.xz \
 && wget --quiet https://github.com/upx/upx/releases/download/v3.95/upx-3.95-amd64_linux.tar.xz \
 && tar -xf upx-3.94-amd64_linux.tar.xz \
 && tar -xf upx-3.95-amd64_linux.tar.xz

I then ran the image with:

macos$ docker run --rm -it \
  -v $PWD:$PWD -w $PWD \
  upx-macos-repro:latest

Inside the image, I compiled and compressed test binaries using both releases:

root# ls -lh
total 8.0K
-rw-r--r-- 1 root root 356 Sep 19 01:18 Dockerfile
-rw-r--r-- 1 root root  72 Sep 19 01:03 main.go

root# GOOS=darwin GOARCH=amd64 go build -o hello .

root# ls -lh
total 2.0M
-rw-r--r-- 1 root root  356 Sep 19 01:18 Dockerfile
-rwxr-xr-x 1 root root 2.0M Sep 19 01:25 hello
-rw-r--r-- 1 root root   72 Sep 19 01:03 main.go

root# cp hello hello-3.94

root# cp hello hello-3.95

root# ls -lh
total 6.0M
-rw-r--r-- 1 root root  356 Sep 19 01:18 Dockerfile
-rwxr-xr-x 1 root root 2.0M Sep 19 01:25 hello
-rwxr-xr-x 1 root root 2.0M Sep 19 01:25 hello-3.94
-rwxr-xr-x 1 root root 2.0M Sep 19 01:25 hello-3.95
-rw-r--r-- 1 root root   72 Sep 19 01:03 main.go

root# /tmp/upx-3.94-amd64_linux/upx hello-3.94
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94        Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   2085008 ->    696976   33.43%   macho/amd64   hello-3.94

Packed 1 file.

root# /tmp/upx-3.95-amd64_linux/upx hello-3.95
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   2085008 ->    696336   33.40%   macho/amd64   hello-3.95

Packed 1 file.

root# ls -lh
total 3.5M
-rw-r--r-- 1 root root  356 Sep 19 01:18 Dockerfile
-rwxr-xr-x 1 root root 2.0M Sep 19 01:25 hello
-rwxr-xr-x 1 root root 681K Sep 19 01:25 hello-3.94
-rwxr-xr-x 1 root root 681K Sep 19 01:25 hello-3.95
-rw-r--r-- 1 root root   72 Sep 19 01:03 main.go

root# sha256sum --binary --tag hello*
SHA256 (hello) = 4a15193351665d4acf528b01c8edd6e486e4ae8867c890e2c0514b679351159f
SHA256 (hello-3.94) = 0cbaf0a47d94ffb7f1680b8a4a5525ce8bc645842cf31fa1cc57a0b40ad88b38
SHA256 (hello-3.95) = 9c8dc585c0ac84af66b19561111cd4a81cd64e4a59eb83adcc2b3382910e0b15

After exiting the image, I then ran the original and compressed binaries, resulting in the execution failure:

macos$ ./hello
Hello World
macos$ ./hello-3.94
Hello World
macos$ ./hello-3.95
Killed: 9

Additionally, I have bundled all of these files together and uploaded them in repro.tar.gz.

Thank you all for the hard work on this amazing project!
Cheers!

#222 modified: p_mach.cpp modified: p_mach.h modified: stub/src/amd64-darwin.macho-entry.S also *.macho-*.h, *.macho-fold.map, amd64-darwin.macho-entry.bin.dump

jreiser · 2018-09-30T20:31:02Z

The tip of devel branch, 37d6a34, works for me; also tested on Mojave. A compiled version for Linux is available beginning at the upx-automatic-builds page as: https://github.com/upx/upx-automatic-builds/blob/devel-20180924-37d6a34-travis/amd64-linux-gcc-8-release/upx-git-37d6a34e5867.out .

#222 modified: p_mach.cpp modified: stub/src/amd64-darwin.macho-main.c modified: stub/src/amd64-darwin.macho-upxmain.c also *-darwin.macho-fold.{h,map}

jreiser · 2018-10-14T04:12:22Z

The tip of devel branch fixes upx -d for such 'rust' executables.

hairyhenderson · 2018-10-20T23:57:55Z

@jreiser is there any timeline on a UPX release with this fix?

jreiser · 2018-11-02T23:54:55Z

@tammersaleh See the comment #222 (comment) of Sep.30. I'll bet you can figure out how to get the version which runs on amd64-darwin.

hmage · 2018-11-03T16:27:42Z

@jreiser which is not what package managers will do -- they will update the package only when new release is out.

hmage · 2018-11-03T16:28:17Z

See Homebrew/homebrew-core#32422

hairyhenderson · 2018-11-12T14:04:44Z

@jreiser is there anything blocking a 3.96 release with this fix? Is there anything we can do to help this along?

jreiser · 2018-11-12T15:44:02Z

@hairyhenderson

Post a Comment that "I tried commit #... [paste the 40-character commit ID, which github will turn into a link] from ... [give the download link, or say how you created the executable of upx itself]. The output from upx compression worked for me in this case, and upx -d decompression restored the original file." The experience of UPX Team is that not enough users actually try the development versions. Posting reproducible issues is a big help that is made even more valuable when real users verify developer fixes before a release.
See the comment upx output of binary 'go' file can not run on macOS #222 (comment) for an example of pre-built executable of upx. Or use homebrew, or borrow a Linux box.
If part 1 works, then why aren't you happy? Oh, you say that some other entire project wants to use upx? Then have 1 or 2 others from that project chime in, "I tried commit #..., it works for me."
Making a release is not an automated process. There are 6 or 7 different hardware+OS combinations which must be tested to verify hand-written assembly language decompressors. Virtualization is not available for some of them, so real hardware must be employed.
Markus is the only one who can make an official release. The NRV match finder+selector is not open source. The UCL implementation is compatible but with slightly less compression. Almost any worthwhile real executable will be smaller if compressed with LZMA, but LZMA decompression takes around 10 times long [still reasonably fast] and about 10 times the energy (a multiply operation for every bit).
$DAYJOB often has priority.

hairyhenderson · 2018-11-12T16:23:23Z

Thanks for the quick response @jreiser. I'll try to find some time to test the fix.

I can imagine that it's a complex project to deal with, given all the different platforms you support! I'm not trying to complain or rush things, and I totally understand priority pressures. It just wasn't clear to me where this stood. Now it is, and I thank you for the clarity!

hairyhenderson · 2018-11-13T01:02:49Z

Ok, I've used brew install --HEAD upx to build 473b904, and I've compressed a Go program (gomplate) with it, then decompressed (with upx -d), and it looks good to me ✅:

$ sha256sum bin/*
5b7a8474260b3ed09884afb0b798aa0e710cfa42b17b1dcff92e77bb38207c21  bin/gomplate_darwin-amd64 # original
dd30fde418e918dca38764d72821bb32cbe5a8836adac8412c68a61b6e9bfb19  bin/gomplate_darwin-amd64-slim # compressed
5b7a8474260b3ed09884afb0b798aa0e710cfa42b17b1dcff92e77bb38207c21  bin/gomplate_darwin-amd64-slim-decompressed # decompressed

and the compressed executable works:

$ bin/gomplate_darwin-amd64-slim -i '{{ print "it works!" }}'
it works!

Perhaps some others can verify as well?

grtfou · 2018-12-13T08:24:45Z

It's work. Thanks for @hairyhenderson 's answer.

$ go version
go version go1.11.1 darwin/amd64

$ brew install --HEAD upx

$ upx -V
upx 3.96-git-473b90419f68
UCL data compression library 1.03
zlib data compression library 1.2.11
LZMA SDK version 4.43

$ upx -o besthello hello
$ ./besthello
Hello World!

hairyhenderson · 2018-12-13T12:43:39Z

Thanks @grtfou!

hairyhenderson · 2018-12-13T12:48:19Z

@jreiser @markus-oberhumer - it looks like at least 2 more (myself and @grtfou) have confirmed the fix. I'll try to see if I can find more if you need more broad testing.

Also @hackmac89 - I see you 👍d my comment - does that mean you can also verify the fix? Can you do so in writing? 🙂

geocine · 2018-12-19T13:16:53Z

@hairyhenderson I can confirm , your comment is working. I uninstalled 3.95 and installed HEAD

hackmac89 · 2019-01-07T14:47:48Z

@hairyhenderson yep, can confirm it´s working.

upx currently breaks macOS binaries (see upx/upx#222)

fxcoudert · 2019-02-08T07:26:55Z

@markus-oberhumer could a 3.96 release be cut? While shipping upx in Homebrew we are seeing the number of necessary patches accumulate (4 patches now) to make the software work.

markus-oberhumer · 2019-02-15T10:37:50Z

Yeah, a new release is probably a good idea. Should happen soon.

steeve · 2020-05-27T13:31:23Z

Thanks @jreiser! I'll do that later today

steeve · 2020-05-27T19:56:09Z

@jreiser here you go! Here is the uncompressed executable https://we.tl/t-OFgjlWIeMe
As I said earlier, that executable works with upx 3.94.

Thanks!

steeve · 2020-05-28T12:51:29Z

Just ran a bisect, and it found this commit: 4f69799.
(on a side note, the build was painless, congrats!)

gonejack · 2020-05-29T03:01:10Z

@hairyhenderson it works for me too now, after go upgraded to 1.14.3 while upx is still at 3.96

MenJeremy · 2020-05-29T13:07:55Z

I have the same issue but with Lazarus IDE 2.0.8 x86_64.
Any suggestions how to downgrade upx on Mac to 3.94 using homebrew?

steeve · 2020-05-29T23:43:21Z

Any suggestions how to downgrade upx on Mac to 3.94 using homebrew?

You can do that:

brew uninstall upx
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/b1f92701367028062abb3518426aa5d8dacc224b/Formula/upx.rb

MenJeremy · 2020-06-01T19:26:39Z

Thanks @steeve, I can now compress again with version 3.94 :)

#222 go-1.14 (but not go-1.14.3 ?) modified: ../p_mach.cpp modified: ../p_mach.h modified: ../p_unix.cpp modified: ../p_unix.h modified: ../stub/src/amd64-darwin.macho-main.c modified: ../stub/amd64-darwin.macho-fold.h modified: ../stub/arm.v5a-darwin.macho-fold.h modified: ../stub/arm64-darwin.macho-fold.h modified: ../stub/tmp/amd64-darwin.macho-fold.map modified: ../stub/tmp/arm.v5a-darwin.macho-fold.map modified: ../stub/tmp/arm64-darwin.macho-fold.map

#222 modified: p_unix.cpp

MenJeremy · 2020-06-16T16:06:21Z

Any suggestions how to downgrade upx on Mac to 3.94 using homebrew?

You can do that:
brew uninstall upx
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/b1f92701367028062abb3518426aa5d8dacc224b/Formula/upx.rb

@steeve Is there a way to test different versions, ex: 3.9.5, I can't seem to find a list of formulas for upx ?

steeve · 2020-06-16T16:35:21Z

You could find the history at https://github.com/Homebrew/homebrew-core/commits/master/Formula/upx.rb

You can look for the affect commit: #222 (comment)

KempWatson · 2020-08-31T21:48:17Z

Where is v3.96 for Mac OS? I don't see it in the downloads in GitHub? Thanks!

jreiser · 2020-08-31T22:12:35Z

@KempWatson That's right, there are no pre-built executables of UPX for MacOS listed on https://github.com/upx/upx/releases because there are no free (as in beer) public build environments for MacOS like there are for Linux and MSWindows. This is due to Apple's licensing policies. Various users have reported success building UPX using brew; search the internet for examples. (Or, because UPX is universally cross-platform [any UPX on any platform can compress for every supported executable file format] then use any Linux or WIndows machine to do the compression.)

jph00 · 2020-11-11T21:16:10Z

@jreiser GitHub Actions provides a free public build environment for MacOS, Linux, and MSWindows.

dimitarvp · 2020-12-22T01:07:01Z

On macOS Big Sur 11.1:

Tried using an older version via the methods specified above and figured I'll try the last version (3.95_1) but...

/usr/local/Homebrew/Library/Homebrew/formulary.rb:227:in `load_file': Invalid usage: Installation of upx from a GitHub commit URL is unsupported! 'brew extract upx' to stable tap on GitHub instead. (UsageError)

Current version (3.96) produces binaries that display Killed when started still.

dimitarvp · 2020-12-22T01:18:25Z

Haven't tried compiling upx from source btw; just did brew install upx. Attached at the bottom is a zipped executable file that you can generate yourself (if you have Rust installed) and test like so:

$ upx --version
upx 3.96
# <extra text snipped>
$ cargo new --bin hello
$ cd hello
$ cargo build --release
$ ./target/release/hello
"Hello, world!"
$ upx --all-methods ./target/release/hello
$ ./target/release/hello
[1]    40004 killed     ./target/release/hello

hello.zip

Issue seems to be with v3.94 of upx that comes installed by default in ubuntu-latest images of Github actions. A number of very similar issues: upx/upx#450 upx/upx#222 Signed-off-by: Harikrishnan Balagopal <harikrishmenon@gmail.com>

…mediately Issue seems to be with v3.94 of upx that comes installed by default in ubuntu-latest images of Github actions. A number of very similar issues: upx/upx#450 upx/upx#222 Signed-off-by: Harikrishnan Balagopal <harikrishmenon@gmail.com>

…mediately (#441) Issue seems to be with v3.94 of upx that comes installed by default in ubuntu-latest images of Github actions. A number of very similar issues: upx/upx#450 upx/upx#222 Signed-off-by: Harikrishnan Balagopal <harikrishmenon@gmail.com>

upx 3.96 currently fails there. See upx/upx#222

#222 go-1.14 (but not go-1.14.3 ?) modified: ../p_mach.cpp modified: ../p_mach.h modified: ../p_unix.cpp modified: ../p_unix.h modified: ../stub/src/amd64-darwin.macho-main.c modified: ../stub/amd64-darwin.macho-fold.h modified: ../stub/arm.v5a-darwin.macho-fold.h modified: ../stub/arm64-darwin.macho-fold.h modified: ../stub/tmp/amd64-darwin.macho-fold.map modified: ../stub/tmp/arm.v5a-darwin.macho-fold.map modified: ../stub/tmp/arm64-darwin.macho-fold.map

#222 modified: p_unix.cpp

jreiser changed the title ~~upx output binary file can not run on macOS~~ upx output of binary 'go' file can not run on macOS Sep 19, 2018

joshdk mentioned this issue Sep 19, 2018

Downgrade UPX from 3.95 back to 3.94 stackrox/ossls#23

Merged

hmage mentioned this issue Sep 25, 2018

upx 3.95 output of binary 'go' file can not run on macOS Homebrew/homebrew-core#32422

Closed

6 tasks

hairyhenderson mentioned this issue Sep 25, 2018

Slim build broken (darwin-amd64) hairyhenderson/gomplate#407

Closed

hairyhenderson mentioned this issue Oct 3, 2018

gomplate 3.0.0 Homebrew/homebrew-core#32405

Closed

tammersaleh mentioned this issue Nov 2, 2018

upx is incompatible with golang generated Mac64 binary #187

Closed

zhanhb mentioned this issue Dec 19, 2018

Unable to process macho/i386 binary #237

Closed

estahn mentioned this issue Jan 13, 2019

upx breaks binaries estahn/cloudping#13

Closed

estahn added a commit to estahn/cloudping that referenced this issue Jan 13, 2019

fix: remove upx (#13)

a466d30

upx currently breaks macOS binaries (see upx/upx#222)

hairyhenderson mentioned this issue Feb 8, 2019

upx: Patch to fix Go bug with 3.95 release Homebrew/homebrew-core#36809

Closed

5 tasks

hairyhenderson mentioned this issue Jun 3, 2020

Intermittent failure using -slim binaries, exit code 127 hairyhenderson/gomplate#861

Closed

jreiser added a commit that referenced this issue Jun 7, 2020

Set .b_extra for hdr, especially for Mach-o for 'go'.

211bbc0

#222 modified: p_unix.cpp

ahmetb mentioned this issue Jun 7, 2020

Consider UPX to reduce size of krew binaries kubernetes-sigs/krew#610

Closed

hairyhenderson mentioned this issue Dec 15, 2020

Updates upx to 3.96 hairyhenderson/gomplate#1003

Closed

corneliusweig mentioned this issue Jan 24, 2021

build is 'passing' but binary fails: exec format error corneliusweig/ketall#122

Closed

HarikrishnanBalagopal mentioned this issue Mar 3, 2021

chore: fix an issue with upx and mac where the process gets killed immediately konveyor/move2kube#441

Merged

rmarchei added a commit to rmarchei/p2cli that referenced this issue Apr 9, 2021

fix darwin/arm64 build

4e9089a

upx 3.96 currently fails there. See upx/upx#222

rmarchei mentioned this issue Apr 9, 2021

fix darwin/arm64 build rmarchei/p2cli#3

Merged

svishwanath-tw mentioned this issue Jul 12, 2021

Talisman global install support for ARM64 thoughtworks/talisman#303

Closed

clavinjune mentioned this issue Apr 15, 2022

darwin/arm64 Binary Not Working clavinjune/serve#12

Closed

markus-oberhumer pushed a commit that referenced this issue Aug 17, 2022

Set .b_extra for hdr, especially for Mach-o for 'go'.

972b670

#222 modified: p_unix.cpp

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upx output of binary 'go' file can not run on macOS #222

upx output of binary 'go' file can not run on macOS #222

hr3lxphr6j commented Sep 18, 2018

joshdk commented Sep 19, 2018

jreiser commented Sep 30, 2018

jreiser commented Oct 14, 2018

hairyhenderson commented Oct 20, 2018

jreiser commented Nov 2, 2018

hmage commented Nov 3, 2018

hmage commented Nov 3, 2018

hairyhenderson commented Nov 12, 2018

jreiser commented Nov 12, 2018

hairyhenderson commented Nov 12, 2018

hairyhenderson commented Nov 13, 2018

grtfou commented Dec 13, 2018 •

edited

Loading

hairyhenderson commented Dec 13, 2018

hairyhenderson commented Dec 13, 2018

geocine commented Dec 19, 2018

hackmac89 commented Jan 7, 2019

fxcoudert commented Feb 8, 2019

markus-oberhumer commented Feb 15, 2019

steeve commented May 27, 2020 •

edited

Loading

steeve commented May 27, 2020 •

edited

Loading

steeve commented May 28, 2020 •

edited

Loading

gonejack commented May 29, 2020

MenJeremy commented May 29, 2020

steeve commented May 29, 2020

MenJeremy commented Jun 1, 2020

MenJeremy commented Jun 16, 2020

steeve commented Jun 16, 2020

KempWatson commented Aug 31, 2020

jreiser commented Aug 31, 2020

jph00 commented Nov 11, 2020

dimitarvp commented Dec 22, 2020

dimitarvp commented Dec 22, 2020

upx output of binary 'go' file can not run on macOS #222

upx output of binary 'go' file can not run on macOS #222

Comments

hr3lxphr6j commented Sep 18, 2018

What's the problem (or question)?

What should have happened?

Do you have an idea for a solution?

How can we reproduce the issue?

Please tell us details about your environment.

joshdk commented Sep 19, 2018

jreiser commented Sep 30, 2018

jreiser commented Oct 14, 2018

hairyhenderson commented Oct 20, 2018

jreiser commented Nov 2, 2018

hmage commented Nov 3, 2018

hmage commented Nov 3, 2018

hairyhenderson commented Nov 12, 2018

jreiser commented Nov 12, 2018

hairyhenderson commented Nov 12, 2018

hairyhenderson commented Nov 13, 2018

grtfou commented Dec 13, 2018 • edited Loading

hairyhenderson commented Dec 13, 2018

hairyhenderson commented Dec 13, 2018

geocine commented Dec 19, 2018

hackmac89 commented Jan 7, 2019

fxcoudert commented Feb 8, 2019

markus-oberhumer commented Feb 15, 2019

steeve commented May 27, 2020 • edited Loading

steeve commented May 27, 2020 • edited Loading

steeve commented May 28, 2020 • edited Loading

gonejack commented May 29, 2020

MenJeremy commented May 29, 2020

steeve commented May 29, 2020

MenJeremy commented Jun 1, 2020

MenJeremy commented Jun 16, 2020

steeve commented Jun 16, 2020

KempWatson commented Aug 31, 2020

jreiser commented Aug 31, 2020

jph00 commented Nov 11, 2020

dimitarvp commented Dec 22, 2020

dimitarvp commented Dec 22, 2020

grtfou commented Dec 13, 2018 •

edited

Loading

steeve commented May 27, 2020 •

edited

Loading

steeve commented May 27, 2020 •

edited

Loading

steeve commented May 28, 2020 •

edited

Loading