Windows 64-bit executable crashes with "Access denied" - Windows 10, UPX 3.91 and 3.94

[apankrat]

What's the problem (or question)?

Got a report from a customer, can't reproduce, but the symptoms are interesting.

A 64-bit executable - C++, built with Visual C++ 2013, v120_xp.
Been compressing it with UPX 3.91 since October 2013.
The installation base is at least in 10s of thousands.
Never had a single UPX-related issue reported until now.

A couple of days ago a customer complained that the program no longer starts. The OS is Windows 10. Ruled out the program's own crash as a cause, also ruled out any antivirus interference (there are no 3rd party AV installed). The event log shows:

Faulting application name: ***, version: ***, time stamp: 0x5a1c2d50
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000028ae40
Faulting process id: 0x3d85c
Faulting application start time: 0x01d36c3fe2462f40
Faulting application path: ***
Faulting module path: unknown
Report Id: ***
Faulting package full name:
Faulting package-relative application ID:

Switched from UPX 3.91 to 3.94 - the issue remained.
Removed UPX compression - the issue went away.

Also of relevance - this gentleman has been running an older version of the program (not dramatically different and also UPX-3.91 compressed) on the same machine and OS for over a year.

I've asked for additional details, including if he perhaps was "tightening up" his OS setup in some form. Once I hear back, I'll update the ticket. I can ask for any details required and, probably, can ask to run diagnostics, collect logs, etc. Just need some guidance from you guys as to what to ask for.

What should have happened?

Program should've launched

Do you have an idea for a solution?

No

How can we reproduce the issue?

Can't tell

Please tell us details about your environment.

• UPX version used - 3.91 and 3.94
• Host Operating System and version: Windows 8.1
• Host CPU architecture: 64-bit
• Target Operating System and version: Windows 10
• Target CPU architecture: 64-bit

[jreiser]

If there are thousands of users but only one user reports a problem running "the same" software, then the leading candidate is the individual box. Which Windows 10 update (1703, 1709, etc.) and build number (16299.64, etc.)? Was the Windows update successful (did not run out of disk space, etc.)? Did the problem start before the most recent Windows update? How old is the hardware, how much RAM, what are the system environment variables, is more than one harddrive involved? Run the RAM diagnostic program overnight. Verify the sha256 checksum of all files that should be the same.

[apankrat]

I agree it's likely to be a box-specific issue, but I don't think it's of a hardware nature. In my 20+ years I've seen a fair amount of errors caused by faulty RAM sticks and this one doesn't fit the profile. The executable size is measly 2 megs, no dependencies outside of basic Win32 API. The issue is readily reproducible irrespective of the machine state and it disappears once the UPX is taken out of the picture.

Windows 10 build number

I will ask, however why are you mentioning this? Are there known issues with specific W10 builds? I can also try and ask for a minidump if it'd be of any use.

What are the system environment variables?

Are there vars that may trigger "Access denied" failures on launch?

[jreiser]

My questions were general to try to get context and understand the environment.

Access denied failure on launch

Which object name (at least, the length of the name, number of directories in the path, size, ...) and what are the access permissions of every file named by the command line (or other launch mechanism)?

The issue is readily reproducible irrespective of the machine state and it disappears once the UPX is taken out of the picture.

Can this one particular user reproduce the problem on more than one box? In more than one working directory? For more than one user-level input file?

executable size is measly 2 megs

Plesae send it, or at least a text listing of the PE file layout: sections, addresses, properties, etc.

I can ask for a minidump

Sure. The identity of the program which was denied access, the literal pathname whose access was denied (and the access permissions at every step along the path), the machine register values, instruction stream around the program counter, values in memory near the stack pointer, etc. It's unclear to me which particular access was denied, and what was running at that instant.

[apankrat]

My questions were general to try to get context and understand the environment.

Aye. Here are some details:

1. Binary in question - bvckup2-1.78.13.0-x64.exe - this is 3.91-compressed version, ~900KB
   MD5 is 9914d80522c5371c5f4665b718aef9f9
2. Unredacted entry from the Event Log:

Faulting application name: bvckup2.exe, version: 1.78.13.0, time stamp: 0x5a1c2d50
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000028ae40
Faulting process id: 0x3d85c
Faulting application start time: 0x01d36c3fe2462f40
Faulting application path: C:\Program Files\Bvckup 2\bvckup2.exe
Faulting module path: unknown
Report Id: 388669ca-94aa-4c3c-a9aa-092aed36ce36
Faulting package full name:
Faulting package-relative application ID:

Launched without any command line parameters.

As I mentioned above, compressing with 3.94 yields the same symptoms, but I don't have a corresponding Event Log entry.

Re: file permissions - will try and confirm, but there shouldn't be anything exotic.
Re: minidump / stack trace - ditto, will ask, but it's a production server, so they may or may not have it.

Can this one particular user reproduce the problem on more than one box?

I don't know, but it's a good question. This is happening on a server machine at this gentleman's work. He also has the program installed on one (or two?) machines at home and it works fine there.

In more than one working directory?

Will try and ask.

For more than one user-level input file?

Not sure what you mean by this.

[jreiser]

I get different md5sum than expected. I downloaded twice, once via Firefox and once via curl. My two downloads compare equal.

$ md5sum *
98b7a3e7f14bb6a3b4d8d870e57c60e1  bvckup2-1.78.13.0-x64.exe
98b7a3e7f14bb6a3b4d8d870e57c60e1  bvckup2-1.78.13.0-x64.exe-2
5c082eb4d37fe7434d38d1b60fe999ba  bvckup2-1.78.13.0-x64.upx-d
$ ls -l
total 4284
-rw-rw-r--. 1 user group  950368 Dec  5 13:28 bvckup2-1.78.13.0-x64.exe
-rw-rw-r--. 1 user group  950368 Dec  5 13:34 bvckup2-1.78.13.0-x64.exe-2
-rw-rw-r--. 1 user group 2476128 Dec  5 13:28 bvckup2-1.78.13.0-x64.upx-d

[apankrat]

My bad. Correct hash is 98b7a3e7f14bb6a3b4d8d870e57c60e1.

[jreiser]

Using objdump -x -D (GNU objdump version 2.26.1-1 on Fedora 25) on the results of upx -d, I see

bvckup2-1.78.13.0-x64.upx-d:     file format pei-x86-64
bvckup2-1.78.13.0-x64.upx-d
architecture: i386:x86-64, flags 0x0000012f:
HAS_RELOC, EXEC_P, HAS_LINENO, HAS_DEBUG, HAS_LOCALS, D_PAGED
start address 0x00000001400eddd8

Characteristics 0x22
        executable
        large address aware

Time/Date               Mon Nov 27 07:20:48 2017
Magic                   020b    (PE32+)
MajorLinkerVersion      12
MinorLinkerVersion      0
SizeOfCode              00175e00
   <<snip>>
ImageBase               0000000140000000

so this file is only one week old. Thousands of users updating in only one week is a wonder. Did the previous version have the same problem?
Also the reported Fault offset: 0x000000000028ae40 does not help much because none of the
31 lines in the disassembly that contain the substring "28ae" are relevant; the last three are

   1402441df:   05 ae 28 d0 59          add    $0x59d028ae,%eax
   1402528ac:   74 00                   je     0x1402528ae
   1402528ae:   6f                      outsl  %ds:(%rsi),(%dx)

which are far below offset 0x28ae40.
Can we get the identity of the object to which access was denied?

[apankrat]

so this file is only one week old. Thousands of users updating in only one week is a wonder.

My remark about the user count was in the context of never seeing a single UPX issue in four years prior.

Did the previous version have the same problem?

It does. I asked to try a year old version and it had the same issue.

Can we get the identity of the object to which access was denied?

I don't know how to do this. Can you give me some pointers? Preferably not involving launching the program under a debugger, because I don't think it'll be an option.

[jreiser]

Hmmm. [Casting about for other possibilities ...] Does a compressed copy of notepad.exe work? What about a compressed copy of date in a Command window? Does varying the compression algorithm matter: upx --lzma versus not using lzma?

[apankrat]

All fair questions. The good news is that I heard back from the person who reported this and he is going to join our exchange here and help with the diagnostics. This may take a day or two though.

[paulwarwicker]

Apologies for the tardy response.

A botched cleanup after an apparently successful upgrade to version 1709 left me with an unusable machine which had to be re-imaged from scratch. The upside of that was that I had a much cleaner environment to re-test in. Now 1709 (build 16299.125). Previously was 1703.

The same problem was immediately reproducible in the re-imaged environment with no extras beyond our coreload image.

Hardware is reasonable new, no older than a year. It is a HP Z240 workstation, 32GB memory. It has 2x 2TB disks and has always had space.

I can confirm that the latest version of bvckup2 (using upx) also has the same problem in this environment, starts and immediately stops. The upx-less version works as expected.

Issue can occur in multiple directories.

I have tried to compress and run notepad.exe. As you can see below it does not run. I'm unsure what I am doing wrong here as this seems to be a very simple test and the verify step appears to be okay.

I could not see that there was a --lzma option.

Note: I had the same output below in the previous environment at 1703 before upgrading.

PS C:\> mkdir tmp
    Directory: C:\
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       15/12/2017     21:47                tmp
PS C:\> cd tmp
PS C:\tmp> cp C:\windows\system32\notepad.exe .
PS C:\tmp> ls
    Directory: C:\tmp
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       29/09/2017     14:41         246784 notepad.exe
PS C:\tmp> C:\opt\tools\upx394w\upx.exe -o notepad-upx.exe notepad.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    246784 ->    180736   73.24%    win64/pe     notepad-upx.exe
Packed 1 file.
PS C:\tmp> C:\opt\tools\upx394w\upx.exe -t notepad-upx.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017
testing notepad-upx.exe [OK]
Tested 1 file.
PS C:\tmp> ls
    Directory: C:\tmp
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       29/09/2017     14:41         180736 notepad-upx.exe
-a----       29/09/2017     14:41         246784 notepad.exe
PS C:\tmp> .\notepad-upx.exe
Program 'notepad-upx.exe' failed to run: The parameter is incorrectAt line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
PS C:\tmp>



PS C:\windows\syswow64> .\notepad.exe                 # runs okay
PS C:\windows\syswow64> C:\opt\tools\upx394w\upx.exe -o notepad-upx.exe .\notepad.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    236544 ->    168448   71.21%    win32/pe     notepad-upx.exe

Packed 1 file.
PS C:\windows\syswow64> C:\opt\tools\upx394w\upx.exe -t notepad-upx.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

testing notepad-upx.exe [OK]

Tested 1 file.
PS C:\windows\syswow64> .\notepad-upx.exe
Program 'notepad-upx.exe' failed to run: The parameter is incorrectAt line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

PS C:\windows\syswow64>

I can run the program under a debugger if necessary. I will have to rebuild this workstation but Visual Studio and/or the Windows Debugging Tools will be part of that.

[apankrat]

John, any plans to look at this one?

I'm asking because we now maintain two 64bit builds of a product (one upx'd and one not), two separate update channels, etc. Obviously, it's not that much of an inconvenience, but it'd still be nice to have this resolved properly.

[jreiser]

Fixing this issue soon is unlikely. The priorities are Android shared libraries and iOS. UPX Team also has less experience and knowledge on MS Windows.
What could hasten a fix: a crisp identification of the problem(s). What is the identity of the object to which "Access denied" [the first report], and why "The parameter is incorrectAt line:1 char:1" [a following report]? If UPX generates an invalid PE file, then which bytes are incorrect and why [and what is a good reference for documentation]? Upon execution, does the problem happen before, during, or after de-compression? What is the value of the program counter (instruction pointer)?
If the UPX strategy is essentially correct, then "mere" coding errors usually can be fixed quickly. If the nature of the bug is an incorrect understanding of the environment or conventions of 64-bit PE, then that probably takes longer.

[Mattiwatti]

I tried this on a Windows 10 VM to see what would happen, and I couldn't reproduce it (meaning the access violation in the exe linked above - notepad not starting is another issue) until I tried unpacking the exe and repacking it again. It then crashed somewhere in the application logic dereferencing some pointer with an address close to the default image base address, which due to high entropy ASLR is about 128GB off the mark on Windows 10. Looking at the PE file revealed that the relocations were indeed missing. But to actually crash due to not having a reloc dir you need to complete the following three steps:

• Lack a relocation dir (obviously);
• Don't declare IMAGE_FILE_RELOCS_STRIPPED in the image file header;
• Do claim to support ASLR by enabling IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE in the DllCharacteristics field of the optional header.
  (All three are required - in other words, if you simply flip one of the flags above the exe will run.)

The exe managed to complete this trifecta of death. Again though, this was only after first unpacking and then repacking it. The original exe never crashed for me. Are you sure the file has only been packed once?

After finding this I managed to reproduce it on Windows 7 as well, on pretty much any 64 bit exe including UPX itself. So I don't think it is related to Windows 10, although Windows 10 did introduce a much more aggressive mandatory ASLR policy:

(reference)

So summing up:

• UPX generates an incorrect PE when extracting a previously packed exe. As far as I can tell there are no runtime issues, but OP might contradict me on that.
• The PE is incorrect in that (1) the relocation directory bytes are present (accounted for) but contain zero bytes, and (2) the header does not give an RVA and size for the relocation directory even though it is physically in the file.
• (Minor) the SizeOfHeaders field in the optional headers is incorrect (given as rva(section[0]) == 0x1000, should be 0x400 for 99.99% of files). I doubt anyone ever uses this field though.