Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows 64-bit executable crashes with "Access denied" - Windows 10, UPX 3.91 and 3.94 #154

Open
apankrat opened this issue Dec 5, 2017 · 29 comments

Comments

Projects
None yet
5 participants
@apankrat
Copy link

commented Dec 5, 2017

What's the problem (or question)?

Got a report from a customer, can't reproduce, but the symptoms are interesting.

A 64-bit executable - C++, built with Visual C++ 2013, v120_xp.
Been compressing it with UPX 3.91 since October 2013.
The installation base is at least in 10s of thousands.
Never had a single UPX-related issue reported until now.

A couple of days ago a customer complained that the program no longer starts. The OS is Windows 10. Ruled out the program's own crash as a cause, also ruled out any antivirus interference (there are no 3rd party AV installed). The event log shows:

Faulting application name: ***, version: ***, time stamp: 0x5a1c2d50
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000028ae40
Faulting process id: 0x3d85c
Faulting application start time: 0x01d36c3fe2462f40
Faulting application path: ***
Faulting module path: unknown
Report Id: ***
Faulting package full name:
Faulting package-relative application ID:

Switched from UPX 3.91 to 3.94 - the issue remained.
Removed UPX compression - the issue went away.

Also of relevance - this gentleman has been running an older version of the program (not dramatically different and also UPX-3.91 compressed) on the same machine and OS for over a year.

I've asked for additional details, including if he perhaps was "tightening up" his OS setup in some form. Once I hear back, I'll update the ticket. I can ask for any details required and, probably, can ask to run diagnostics, collect logs, etc. Just need some guidance from you guys as to what to ask for.

What should have happened?

Program should've launched

Do you have an idea for a solution?

No

How can we reproduce the issue?

Can't tell

Please tell us details about your environment.

  • UPX version used - 3.91 and 3.94
  • Host Operating System and version: Windows 8.1
  • Host CPU architecture: 64-bit
  • Target Operating System and version: Windows 10
  • Target CPU architecture: 64-bit
@jreiser

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

If there are thousands of users but only one user reports a problem running "the same" software, then the leading candidate is the individual box. Which Windows 10 update (1703, 1709, etc.) and build number (16299.64, etc.)? Was the Windows update successful (did not run out of disk space, etc.)? Did the problem start before the most recent Windows update? How old is the hardware, how much RAM, what are the system environment variables, is more than one harddrive involved? Run the RAM diagnostic program overnight. Verify the sha256 checksum of all files that should be the same.

@apankrat

This comment has been minimized.

Copy link
Author

commented Dec 5, 2017

I agree it's likely to be a box-specific issue, but I don't think it's of a hardware nature. In my 20+ years I've seen a fair amount of errors caused by faulty RAM sticks and this one doesn't fit the profile. The executable size is measly 2 megs, no dependencies outside of basic Win32 API. The issue is readily reproducible irrespective of the machine state and it disappears once the UPX is taken out of the picture.

Windows 10 build number

I will ask, however why are you mentioning this? Are there known issues with specific W10 builds? I can also try and ask for a minidump if it'd be of any use.

What are the system environment variables?

Are there vars that may trigger "Access denied" failures on launch?

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

My questions were general to try to get context and understand the environment.

Access denied failure on launch

Which object name (at least, the length of the name, number of directories in the path, size, ...) and what are the access permissions of every file named by the command line (or other launch mechanism)?

The issue is readily reproducible irrespective of the machine state and it disappears once the UPX is taken out of the picture.

Can this one particular user reproduce the problem on more than one box? In more than one working directory? For more than one user-level input file?

executable size is measly 2 megs

Plesae send it, or at least a text listing of the PE file layout: sections, addresses, properties, etc.

I can ask for a minidump

Sure. The identity of the program which was denied access, the literal pathname whose access was denied (and the access permissions at every step along the path), the machine register values, instruction stream around the program counter, values in memory near the stack pointer, etc. It's unclear to me which particular access was denied, and what was running at that instant.

@apankrat

This comment has been minimized.

Copy link
Author

commented Dec 5, 2017

My questions were general to try to get context and understand the environment.

Aye. Here are some details:

  1. Binary in question - bvckup2-1.78.13.0-x64.exe - this is 3.91-compressed version, ~900KB
    MD5 is 9914d80522c5371c5f4665b718aef9f9
  2. Unredacted entry from the Event Log:

Faulting application name: bvckup2.exe, version: 1.78.13.0, time stamp: 0x5a1c2d50
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000028ae40
Faulting process id: 0x3d85c
Faulting application start time: 0x01d36c3fe2462f40
Faulting application path: C:\Program Files\Bvckup 2\bvckup2.exe
Faulting module path: unknown
Report Id: 388669ca-94aa-4c3c-a9aa-092aed36ce36
Faulting package full name:
Faulting package-relative application ID:

Launched without any command line parameters.

As I mentioned above, compressing with 3.94 yields the same symptoms, but I don't have a corresponding Event Log entry.

Re: file permissions - will try and confirm, but there shouldn't be anything exotic.
Re: minidump / stack trace - ditto, will ask, but it's a production server, so they may or may not have it.

Can this one particular user reproduce the problem on more than one box?

I don't know, but it's a good question. This is happening on a server machine at this gentleman's work. He also has the program installed on one (or two?) machines at home and it works fine there.

In more than one working directory?

Will try and ask.

For more than one user-level input file?

Not sure what you mean by this.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

I get different md5sum than expected. I downloaded twice, once via Firefox and once via curl. My two downloads compare equal.

$ md5sum *
98b7a3e7f14bb6a3b4d8d870e57c60e1  bvckup2-1.78.13.0-x64.exe
98b7a3e7f14bb6a3b4d8d870e57c60e1  bvckup2-1.78.13.0-x64.exe-2
5c082eb4d37fe7434d38d1b60fe999ba  bvckup2-1.78.13.0-x64.upx-d
$ ls -l
total 4284
-rw-rw-r--. 1 user group  950368 Dec  5 13:28 bvckup2-1.78.13.0-x64.exe
-rw-rw-r--. 1 user group  950368 Dec  5 13:34 bvckup2-1.78.13.0-x64.exe-2
-rw-rw-r--. 1 user group 2476128 Dec  5 13:28 bvckup2-1.78.13.0-x64.upx-d
@apankrat

This comment has been minimized.

Copy link
Author

commented Dec 5, 2017

My bad. Correct hash is 98b7a3e7f14bb6a3b4d8d870e57c60e1.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

Using objdump -x -D (GNU objdump version 2.26.1-1 on Fedora 25) on the results of upx -d, I see

bvckup2-1.78.13.0-x64.upx-d:     file format pei-x86-64
bvckup2-1.78.13.0-x64.upx-d
architecture: i386:x86-64, flags 0x0000012f:
HAS_RELOC, EXEC_P, HAS_LINENO, HAS_DEBUG, HAS_LOCALS, D_PAGED
start address 0x00000001400eddd8

Characteristics 0x22
        executable
        large address aware

Time/Date               Mon Nov 27 07:20:48 2017
Magic                   020b    (PE32+)
MajorLinkerVersion      12
MinorLinkerVersion      0
SizeOfCode              00175e00
   <<snip>>
ImageBase               0000000140000000

so this file is only one week old. Thousands of users updating in only one week is a wonder. Did the previous version have the same problem?
Also the reported Fault offset: 0x000000000028ae40 does not help much because none of the
31 lines in the disassembly that contain the substring "28ae" are relevant; the last three are

   1402441df:   05 ae 28 d0 59          add    $0x59d028ae,%eax
   1402528ac:   74 00                   je     0x1402528ae
   1402528ae:   6f                      outsl  %ds:(%rsi),(%dx)

which are far below offset 0x28ae40.
Can we get the identity of the object to which access was denied?

@apankrat

This comment has been minimized.

Copy link
Author

commented Dec 6, 2017

so this file is only one week old. Thousands of users updating in only one week is a wonder.

My remark about the user count was in the context of never seeing a single UPX issue in four years prior.

Did the previous version have the same problem?

It does. I asked to try a year old version and it had the same issue.

Can we get the identity of the object to which access was denied?

I don't know how to do this. Can you give me some pointers? Preferably not involving launching the program under a debugger, because I don't think it'll be an option.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

Hmmm. [Casting about for other possibilities ...] Does a compressed copy of notepad.exe work? What about a compressed copy of date in a Command window? Does varying the compression algorithm matter: upx --lzma versus not using lzma?

@apankrat

This comment has been minimized.

Copy link
Author

commented Dec 6, 2017

All fair questions. The good news is that I heard back from the person who reported this and he is going to join our exchange here and help with the diagnostics. This may take a day or two though.

@paulwarwicker

This comment has been minimized.

Copy link

commented Dec 15, 2017

Apologies for the tardy response.

A botched cleanup after an apparently successful upgrade to version 1709 left me with an unusable machine which had to be re-imaged from scratch. The upside of that was that I had a much cleaner environment to re-test in. Now 1709 (build 16299.125). Previously was 1703.

The same problem was immediately reproducible in the re-imaged environment with no extras beyond our coreload image.

Hardware is reasonable new, no older than a year. It is a HP Z240 workstation, 32GB memory. It has 2x 2TB disks and has always had space.

I can confirm that the latest version of bvckup2 (using upx) also has the same problem in this environment, starts and immediately stops. The upx-less version works as expected.

Issue can occur in multiple directories.

I have tried to compress and run notepad.exe. As you can see below it does not run. I'm unsure what I am doing wrong here as this seems to be a very simple test and the verify step appears to be okay.

I could not see that there was a --lzma option.

Note: I had the same output below in the previous environment at 1703 before upgrading.

PS C:\> mkdir tmp
    Directory: C:\
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       15/12/2017     21:47                tmp
PS C:\> cd tmp
PS C:\tmp> cp C:\windows\system32\notepad.exe .
PS C:\tmp> ls
    Directory: C:\tmp
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       29/09/2017     14:41         246784 notepad.exe
PS C:\tmp> C:\opt\tools\upx394w\upx.exe -o notepad-upx.exe notepad.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    246784 ->    180736   73.24%    win64/pe     notepad-upx.exe
Packed 1 file.
PS C:\tmp> C:\opt\tools\upx394w\upx.exe -t notepad-upx.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017
testing notepad-upx.exe [OK]
Tested 1 file.
PS C:\tmp> ls
    Directory: C:\tmp
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       29/09/2017     14:41         180736 notepad-upx.exe
-a----       29/09/2017     14:41         246784 notepad.exe
PS C:\tmp> .\notepad-upx.exe
Program 'notepad-upx.exe' failed to run: The parameter is incorrectAt line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
PS C:\tmp>



PS C:\windows\syswow64> .\notepad.exe                 # runs okay
PS C:\windows\syswow64> C:\opt\tools\upx394w\upx.exe -o notepad-upx.exe .\notepad.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    236544 ->    168448   71.21%    win32/pe     notepad-upx.exe

Packed 1 file.
PS C:\windows\syswow64> C:\opt\tools\upx394w\upx.exe -t notepad-upx.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94w       Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

testing notepad-upx.exe [OK]

Tested 1 file.
PS C:\windows\syswow64> .\notepad-upx.exe
Program 'notepad-upx.exe' failed to run: The parameter is incorrectAt line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\notepad-upx.exe
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

PS C:\windows\syswow64>

I can run the program under a debugger if necessary. I will have to rebuild this workstation but Visual Studio and/or the Windows Debugging Tools will be part of that.

@apankrat

This comment has been minimized.

Copy link
Author

commented Jan 8, 2018

John, any plans to look at this one?

I'm asking because we now maintain two 64bit builds of a product (one upx'd and one not), two separate update channels, etc. Obviously, it's not that much of an inconvenience, but it'd still be nice to have this resolved properly.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2018

Fixing this issue soon is unlikely. The priorities are Android shared libraries and iOS. UPX Team also has less experience and knowledge on MS Windows.
What could hasten a fix: a crisp identification of the problem(s). What is the identity of the object to which "Access denied" [the first report], and why "The parameter is incorrectAt line:1 char:1" [a following report]? If UPX generates an invalid PE file, then which bytes are incorrect and why [and what is a good reference for documentation]? Upon execution, does the problem happen before, during, or after de-compression? What is the value of the program counter (instruction pointer)?
If the UPX strategy is essentially correct, then "mere" coding errors usually can be fixed quickly. If the nature of the bug is an incorrect understanding of the environment or conventions of 64-bit PE, then that probably takes longer.

tamalsaha added a commit to appscodelabs/libbuild that referenced this issue Feb 19, 2018

@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2018

I tried this on a Windows 10 VM to see what would happen, and I couldn't reproduce it (meaning the access violation in the exe linked above - notepad not starting is another issue) until I tried unpacking the exe and repacking it again. It then crashed somewhere in the application logic dereferencing some pointer with an address close to the default image base address, which due to high entropy ASLR is about 128GB off the mark on Windows 10. Looking at the PE file revealed that the relocations were indeed missing. But to actually crash due to not having a reloc dir you need to complete the following three steps:

  • Lack a relocation dir (obviously);
  • Don't declare IMAGE_FILE_RELOCS_STRIPPED in the image file header;
  • Do claim to support ASLR by enabling IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE in the DllCharacteristics field of the optional header.
    (All three are required - in other words, if you simply flip one of the flags above the exe will run.)

The exe managed to complete this trifecta of death. Again though, this was only after first unpacking and then repacking it. The original exe never crashed for me. Are you sure the file has only been packed once?

After finding this I managed to reproduce it on Windows 7 as well, on pretty much any 64 bit exe including UPX itself. So I don't think it is related to Windows 10, although Windows 10 did introduce a much more aggressive mandatory ASLR policy:
10
(reference)

So summing up:

  • UPX generates an incorrect PE when extracting a previously packed exe. As far as I can tell there are no runtime issues, but OP might contradict me on that.
  • The PE is incorrect in that (1) the relocation directory bytes are present (accounted for) but contain zero bytes, and (2) the header does not give an RVA and size for the relocation directory even though it is physically in the file.
  • (Minor) the SizeOfHeaders field in the optional headers is incorrect (given as rva(section[0]) == 0x1000, should be 0x400 for 99.99% of files). I doubt anyone ever uses this field though.
@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2018

An update: it is in fact possible to a get a UPX-packed notepad.exe to run under Windows 10, but this is a bit more complicated than one might expect 😄 This is due to some minor bugs in the current PE64 packer combined with the new crap malware security features that are continuously being added to Windows 10.

it verks

The most problematic addition by far is CFG (that's Control Flow Guard for you Linux people). This is actually not a Windows 10 feature; it was added in Windows 8.1, so most of below applies to that OS too. The reason why CFG is problematic for UPX in particular is because it requires storing a static list of all allowed call targets in the load configuration directory of the PE. This is the GuardCFFunctionTable in the IMAGE_LOAD_CONFIG_DIRECTORY (see winnt.h). And unlike with the import directory you cannot cheat by calling a GetProcAddress equivalent to fill the entries at runtime: the ntdll loader does this for you early in the process lifetime based on the guard table, and is not happy if you try to mess with the call target bitmap afterwards. So a proper implementation of CFG support would require storing this table completely uncompressed, because it has to be accessible by the loader before any user code is even allowed to run.

CFG also requires specifying two function pointers in the load config table (GuardCFCheckFunctionPointer/GuardCFDispatchFunctionPointer) that validate and dispatch calls respectively. They are normally added as dummy placeholders by the linker. The loader replaces them with the real functions after the bitmap of valid call targets has been initialized. This shouldn't be difficult; it only requires updating the VAs to take the UPX stub into account.

The last important part of the load config directory is not related to CFG but to the cookie used by the /GS compiler switch (which is ancient), but it is similar to the CFG pointers in that it resides in the load config dir and must be a valid VA 'within the file' (i.e. using raw offsets). Therefore SecurityCookie may not point to UPX0 as it does now, because that section only has a virtual size and a raw size of 0. Lastly, the value of the cookie must be the magic constant 0x2B992DDFA232. This value is hardcoded in ntdll in 8.1 and later, and it seems to be very attached to it. It broke into my kernel debugger with an assert when I changed the value by one byte.

What if the PE does not support CFG? Then it must still have a load config directory with at least the cookie VA iff ((ih.MajorSubsystemVersion >= 6 && ih.MinorSubSystemVersion >= 3) || ih.MajorSubSystemVersion > 6). PE images targeting older OSes are given a pass, but if you have a minimum requirement of Windows 8.1, the loader enforces that you have the cookie. So, for images that target 8.1 or 10 (like notepad.exe), the 'minimum requirements to run' load config directory looks like this:

minload

NB: there are two copies of the size of the load configuration directory, and they must always match! (OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_LOAD_CONFIG].Size and the Size field of a PIMAGE_LOAD_CONFIG_DIRECTORY.) If you change one size and forget about the other, get ready for some extremely unhelpful error messages.

OK, so far the load configuration directory part. What's wrong or missing on the UPX side currently?

  • I encountered quite a few misaligned load configuration directory RVAs in the UPX output (so far only that directory, no others). This will cause NtCreateSection to fail in the kernel before process creation even begins and results in extremely vague/obscure error messages. See #182. The directory address must be at least dword aligned on x86, and at least quadword aligned on x64. At the moment they are sometimes only aligned to half of the required alignment (on both x86 and x64).
  • As said in my previous post, SizeOfHeaders is too large. Also said in my previous post: "I doubt anyone ever uses this field". Guess who was wrong? Turns out the kernel uses it when creating a file map for a section object, as part of a sum of fields which is then compared to the file size on disk to check if you weren't lying about the physical size (claiming the file is bigger than it actually is). I'm not really sure why it's so important to check this, but the kernel does it, so the easiest solution is probably to just set it to the correct value. Reference for this:

SizeOfHeaders

The combined size of the following items, rounded to a multiple of the value specified in the FileAlignment member.

  • e_lfanew member of IMAGE_DOS_HEADER
  • 4 byte signature
  • size of IMAGE_FILE_HEADER
  • size of optional header
  • size of all section headers
  • The VA of the stack cookie should be a location in either the UPX1 or the UPX2/.rsrc sections, not UPX0 which does not exist on disk. Other fields can be zeroed out for now if necessary.
  • CFG images are not supported. I can't say I'm losing sleep over this, but others might disagree. Other than fully implementing support like for TLS, there are two possible quick'n dirty fixes:
    • Set OptionalHeader.DllCharacteristics &= ~IMAGE_DLLCHARACTERISTICS_GUARD_CF. This does exactly what you'd expect.
    • Currently you can also 'enable' CFG on any executable by simply setting the size of the load config directory to 0x100, zeroing all fields except the cookie which is required, and setting IMAGE_DLLCHARACTERISTICS_GUARD_CF on the optional header.

procexp

Microsoft security strikes again.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2018

@Mattiwatti: Thank you for the explanations, diagrams, references, and screen shots! We really appreciate your help. Now to the code ...

@apankrat

This comment has been minimized.

Copy link
Author

commented Feb 27, 2018

@Mattiwatti

Woah, that's one impressive write-up. To comment on this bit:

Are you sure the file has only been packed once?

Yes, positive.

@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2018

OK, time to do some triage then!

I took another look at bvckup2-1.78.13.0-x64.exe after identifying the problems I had with notepad. The RVA of the relocation directory is misaligned, so that could be a reason for the issues you're having. In that case I don't have an explanation for the differences in behaviour between our systems though. It could be that my CPU (i7 6700K) is subtly different from yours and recovers from the unaligned memory access somehow. Or even the hypervisor, since I don't have Windows 10 on any physical machines. I think the alignment check by the kernel is literally addr % sizeof(void*) == 0 though, so it seems unlikely.

See attachment:
bvckup2.zip

C:\Users\Matti\Desktop\bvckup2>sha1sum *.exe
69231bc103d2460a888db68575eea64e35be4999  bvckup2-1.78.13.0-x64.fixes-no-aslr.exe
34f8fd63d0a15df85cca3c4c56af5a679f3b3c06  bvckup2-1.78.13.0-x64.fixes.exe
f3df3557d88aba73bf86c5a43b7d0a04f61ac480  bvckup2-1.78.13.0-x64.no-aslr.exe

We can use these to check which of the following is true:

  1. The relocation info is corrupted in some way that causes the executable to crash on your system but not on mine, even though I get full high entropy bottom-up ASLR;
  2. One or more of the issues listed in my second post is at fault;
  3. Both (1) and (2) are the case;
  4. None of these is the case, and we should look for some other cause.

The 'fixes' variants were modified as follows:

  • Corrected the misaligned relocation directory RVA and moved the directory contents by +4 bytes;
  • Changed the VA of SecurityCookie in the load configuration directory to point to some spare bytes in the UPX1 section where I put The Magic Value (tm).
  • Changed SizeOfHeaders from 0x1000 to 0x400.

You should try the files in the order listed above. If the first one still crashes, you don't need to bother with the rest. In that case you should still attach a debugger to bvckup2-1.78.13.0-x64.no-aslr.exe though and verify that it being loaded at 0x140000000 to make sure there is no fascist enterprise policy at work that is force-relocating executables that do not support ASLR.

If none of these work, I need some way to replicate the crash or we won't get far. Best would be a minimal VM image (clean install, don't activate Windows, don't put sensitive data on it) - preferably VirtualBox .vdi format and the .vbox file as well for the machine settings. I know that's a super inconvenient request, but most issues loading PE images can't be diagnosed with a user mode debugger since they occur in the kernel.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2018

@Mattiwatti [unless] my CPU (i7 6700K) is subtly different from yours and recovers from the unaligned memory access somehow    On x86* all memory transfers to or from an integer or floating-point register work just fine regardless of alignment, unless in User mode (ring 3) the AC bit (Alignment Check: bit 18, the 0x40000 bit) is a '1' in EFLAGS. Furthermore, the speed penalty for misalignment is at most 1 cycle, and often is 0 cycles. The data cache "just works" that way. This is one of the good things that Intel did (and AMD got it in the licensing deals.) The only exceptions are some SSEx instructions that transfer 16 bytes and demand 16-byte alignment.
The AC bit can be set by user mode, and the OS can pass the trap to user mode. So in theory user mode could log all the misalignments. In practice there are too many of them in low-level code and language run-time libraries.

@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2018

Yes, and Windows even goes as far as to have a decoder in the kernel to change e.g. movdqa GP faults into movdqu for SSE! (though this is only enabled for legacy applications by default - if you really want to you can enable it like this: BOOLEAN yes = TRUE; NtSetInformationThread(NtCurrentThread, ThreadEnableAlignmentFaultFixup, &yes, NULL)).

What the kernel does is a bit different: whenever it needs to transfer data across the ring 3 <-> ring 0 boundary, it probes the user address:

void ProbeForWrite(void* address, unsigned size, unsigned alignment)
{
  if (address > MM_USER_PROBE_ADDRESS)
    ExRaiseAccessViolation();
  if (address + size < address)
    ExRaiseAccessViolation();
  if ((address & (alignment - 1)) != 0)
    ExRaiseDatatypeMisalignment();
  // ...more checks here if the range spans multiple pages...
}

Thus the exception (meaning SEH exception, not CPU exception) is entirely forced, which is why I found it unlikely that there is any difference in behaviour between our systems re: this. I think the idea behind the alignment part of the probe is mostly an 'is this not bogus' check, because pointers and structs are normally already correctly aligned by the compiler and/or your brain.

@apankrat

This comment has been minimized.

Copy link
Author

commented Feb 28, 2018

OK, time to do some triage then!

Pinging @paulwarwicker. I merely build the exe, it's Paul who has the magic box that breaks it.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2018

@Mattiwatti Please help me setup Win64 debugging environment. Pretend that my only exposure to Win64 has been by using general applications, FileExplorer, and CMD utilities. What I think I want is to gain control in a machine-language debugger as soon as possible after Launch of an application that has been compressed by upx, together with a means to print the current layout of the address-space mappings, and a list of what kernel32 already has done to (or with) the Sections. I have thorough knowledge and extensive experience with the analogous situation on many *nix systems.

@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2018

Sure. I'll start with the bare minimum needed to get started:

  • Download VirtualBox, install it, and set up a new VM. The default settings are mostly fine, but just enable anything that improves performance if it's not already checked. Finally, add a serial port and set it up exactly as follows:
    vbox
    You don't actually need a virtual serial port to debug Windows 8.1 and later because they allow debugging over ethernet (which I actually recommend because it's much faster). But this method works for any version of Windows, and once you've set up a machine for serial debugging it takes about 5 minutes to switch to ethernet debugging if you follow the MSDN docs.
  • Download the Windows SDK. The only thing you need to check are the Debugging Tools for Windows. If this is your dev machine you may want the SDK headers and .libs as well; just be aware that the UWP crap is non-optional in that case.
  • You now have windbg.exe available in C:\Program Files (x86)\Windows Kits\10\Debuggers\x64. Make a shortcut to it and set the command line as follows:
    windbg
  • Right click My Computer and go to Properties -> Advanced system settings -> Environment variables. Under System variables, press New and add _NT_SYMBOL_PATH with value srv*C:\Symbols*https://msdl.microsoft.com/download/symbols. Also actually make the C:\Symbols directory. You need this to get proper debug symbols from the MS servers in order to make sense of the disassembly output.
  • On the VM guest side, after installing the OS and the VirtualBox guest additions for shared folders, you need to enable kernel debugging mode. In theory, and according to MSDN, all that's necessary is
bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200

However, Windows 10 is extremely hostile to debuggers and people with any amount of technical skill in general. It will delete driver files if they cause too many crashes, go into a 'forced recovery' state in which you must run the recovery tool from the installation CD to 'help' with your boot issues, and even do a full rollback to a previous version of the OS (wiping your settings in the process) in the worst case. I don't have a comprehensive list of settings, so the best I can do is give you my bcdedit output from a Windows 10 VM that is semi-workable:

C:\Users\Matti>bcdedit /enum {current}

Windows Boot Loader
-------------------
identifier              {current}
device                  partition=C:
path                    \Windows\system32\osloader.efi
description             Windows 10
locale                  en-US
inherit                 {bootloadersettings}
testsigning             Yes
recoveryenabled         No
isolatedcontext         Yes
allowedinmemorysettings 0x15000075
osdevice                partition=C:
systemroot              \Windows
kernel                  smokeweed.exe
resumeobject            {af348d94-4c40-11e7-8f61-9a35e19d54aa}
nx                      OptIn
bootmenupolicy          Legacy
bootstatuspolicy        IgnoreAllFailures
debug                   Yes

Of these, testsigning = Yes, recoveryenabled = No, bootmenupolicy = Legacy, and bootstatuspolicy = IgnoreAllFailures are absolutely mandatory for a usable debugging environment. smokeweed.exe is the name of the kernel image to boot, the default if not specified is ntoskrnl.exe. If you see anything in your output that is missing in mine, that's not an accident. It is most likely a setting I deleted very purposefully and very angrily. Do the same now to save yourself trouble later.

After doing the above, you should be able to attach! Start WinDbg first, then the VM. You should see output in WinDbg during boot if everything is working, and you can break with CTRL+BREAK (g for go again).

WinDbg is actually a lot like gdb in that it is both extremely powerful and extremely arcane. Its syntax is bizarre, and I won't even comment on the GUI. Since the official reference may be daunting due to how extensive it is, I recommend simply starting with Common WinDbg commands instead. That will probably cover all standard debugger stuff you're looking for.

Since UPX is a user mode program, the assumption is that at some point you should (hopefully) be able to attach a user mode debugger to your target process. You can use WinDbg for this, and in fact I recommend installing it on the guest as well as the host (both with _NT_SYMBOL_PATH set) because many tools make use of the symbol directory if it's present. I don't recommend using WinDbg for user mode debugging however. x64dbg is much better.

Other than x64dbg, there are some other tools that I would say are essential (not going to give an explanation for everything, you can click the links yourself):

Re: virtual memory layout of a process: this is something that really only comes into play after kernel has finished creating the kernel section object (NB: this is an unfortunate name as PE images also have 'sections' - these refer to different things), the process object and initial thread with a minimal PEB and TEB (Process/Thread Environment Block). The kernel does have EPROCESS and ETHREAD structs respectively that are occasionally interesting to inspect, but during kernel mode debugging there will usually be no process address space to inspect as there is no process. If there is a process with an address space, don't expect too much from WinDbg; it will often simply fail to display user mode memory unless you specifically force it to work as a ring 3-in ring 0-debugger. It's just not really built for it. You can get extremely detailed info on system memory related things like PTEs and driver pool allocations, but not much in the way of anything that would be of interest for UPX. Almost all of the kernel debugging I needed to do was related to NtCreateSection, and I was inspecting local variables and the validation logic. Never memory.

So if someone is messing with the process address space before user code gets to execute, and it is not the kernel, who is it? Ntdll. I think I sense some confusion re: this from your post, as you mentioned kernel32 😉 Because of its impressive name (kernel32) it's easy to fall into the trap of thinking that kernel32 matters in user mode, or even that it is the kernel itself. But kernel32 is a nobody in Windows land. Nowadays it only forwards to functions from KernelBase.dll. And KernelBase is also a nobody. It only contains MSDN-compliant wrapper functions for the real implementations which are in... ntdll. Ntdll is the boss in user mode, ntoskrnl is the boss in kernel mode.

Almost all Windows internals related stuff is purposely undocumented and to some extent voodoo, but if you're interested in a more 'official' source than my ramblings then I highly suggest Windows Internals. The first part of the 7th edition is freshly out and up to date for Windows 10, but I'd recommend the 6th edition instead at the moment. I have the 7th edition and it contains many frustrating forward references to part 2 which is not out yet. The 6th edition deals with Windows 7 so most or all of it is still relevant.

@jreiser

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

@Mattiwatti Thank you for another very detailed tour. I noticed these things:

  • On Windows10 My Computer is known as This PC
  • bcdedit requires Administrator privileges, so WinKey > Windows System > right click Command Prompt > More > Run as Administrator
  • On a non-EFI machine use winload.exeinstead of osloader.efi
  • VirtualBox guest additions for shared folders is Devices > Insert Guest Additions CD image ...
  • Using smokeweed.exe causes the VM to fail to start. I could not figure out how to change back to ntoskrnl.exe (FileExplorer copy+paste(with changed name) does not work in that directory, the directory is protected, the ownerID is protected, ...), so I had to delete the VM and install again from scratch. This disappointed me. The re-install is still going, so I wlll have more to report later.
@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

Ahh, I've led you astray there, sorry. There is no need to have a smokeweed.exe kernel entry in bcdedit (or any other name, unless you are using a modified kernel image). winload.{efi|exe} is also the correct default name for the bootloader. For future reference, the way to set this would have been
bcdedit /set {current} kernel filename.ext
where filename.ext is a file in C:\Windows\System32. To reset to the default ntoskrnl.exe:
bcdedit /deletevalue {current} kernel

The reason I have a modified kernel with such a blatantly non-official name is because I use a driver that employs SSDT hooking to defeat anti-debugger detections. Because this is a lazy form of rootkit writing, this will set off Microsoft's own rootkit named PatchGuard [technical analyses part 1, 2, 3] and crash the system. Because I am interested in being able to debug and not in rootkit development, I simply disable Patchguard on-disk. The modified bootloader (winload.{exe|efi}) is needed to load a kernel image that has been tampered with, and as a precaution I have patched bootmgfw.efi (bootmgr on non-EFI systems) as well. Finally, the stand-out filename is because some software attempts to detect exactly this in an attempt to be clever. Unfortunately for them, this also goes through a syscall which I can intercept to log this to the ring 0 debugger. If there is any difference in behaviour between the standard and non-standard filenames, I simply return the clean filename.

@paulwarwicker

This comment has been minimized.

Copy link

commented Mar 5, 2018

Apologies for the slow response again. Busy week last week and this is my backup program that continuously backs up in real time so had to wait for a quieter period. I have been following the thread with interest.

Steps taken:

  1. Exported config and uninstalled working system (now v78.20 automatically installed on 2018-02-09 from the upxless channel)
  2. Verified baseline that the error still occurs as originally reported by running bvckup2-1.78.13.0-x64.exe (obtained from https://bvckup2.com/files/bvckup2-1.78.13.0-x64.exe). See below for Event Viewer entry.
  3. Downloaded bvckup2.zip from git
  4. Verifed sha1 checksums
  5. Ran bvckup2-1.78.13.0-x64.fixes-no-aslr.exe - failure
  6. Ran bvckup2-1.78.13.0-x64.fixes.exe - failure
  7. Ran bvckup2-1.78.13.0-x64.no-aslr.exe - failure
  8. Reinstalled from bvckup2-setup-1.78.16.0-upxless.exe
  9. Ran bvckup2.exe - success

As an aside I also tried running these same 3 executables at home where I also have W10 and I do not have the upx problem as far as I know. Running all 3 caused AVG to sniff the executable and declare that "You've discovered a very rare file". All 3 have been submitted to AVG Virus Lab for deep analysis. Will let you know the outcome.

The original continues to run perfectly.

bvckup2-1.78.13.0-x64.exe

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0
Problem signature:
P1: bvckup2-1.78.13.0-x64.exe
P2: 1.78.13.0
P3: 5a1c2d50
P4: StackHash_4906
P5: 0.0.0.0
P6: 00000000
P7: PCH_3D_FROM_bvckup2_1_78_13_0_x64+0x000000000026AAD5
P8: c0000005
P9: 0000000000000008
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER5578.tmp.WERInternalMetadata.xml

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bvckup2-1.78.13._30b875bcf4bb665d8e74c5db6c91193ed476548a_de97c015_2bd14e54
Analysis symbol:
Rechecking for solution: 0
Report Id: ef26040c-7a95-4352-924a-7a6102abf664
Report Status: 524388

bvckup2-1.78.13.0-x64.fixes-no-aslr.exe

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0
Problem signature:
P1: bvckup2-1.78.13.0-x64.fixes-no-aslr.exe
P2: 1.78.13.0
P3: 5a1c2d50
P4: StackHash_9f0f
P5: 0.0.0.0
P6: 00000000
P7: PCH_3D_FROM_bvckup2_1_78_13_0_x64_fixes_no_aslr+0x000000000026AAD5
P8: c0000005
P9: 0000000000000008
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6E9B.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bvckup2-1.78.13._ebe8a424cfad5ec70c5b487bef2bf57c319151_71dce926_43f465e1
Analysis symbol:
Rechecking for solution: 0
Report Id: 3b8041fc-271f-4ec1-9f84-2f71b6fba4f3
Report Status: 524388

bvckup2-1.78.13.0-x64.fixes.exe

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0
Problem signature:
P1: bvckup2-1.78.13.0-x64.fixes.exe
P2: 1.78.13.0
P3: 5a1c2d50
P4: StackHash_056b
P5: 0.0.0.0
P6: 00000000
P7: PCH_3D_FROM_bvckup2_1_78_13_0_x64_fixes+0x000000000026AAD5
P8: c0000005
P9: 0000000000000008
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER419A.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bvckup2-1.78.13._c6ff55865824895a18183ab9c8eeab32eb34eb5_da14bb00_1c8d3759
Analysis symbol:
Rechecking for solution: 0
Report Id: 8a217ec2-fe07-4082-a115-5ee87ac68ccd
Report Status: 524388

bvckup2-1.78.13.0-x64.no-aslr.exe

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0
Problem signature:
P1: bvckup2-1.78.13.0-x64.no-aslr.exe
P2: 1.78.13.0
P3: 5a1c2d50
P4: StackHash_eb4f
P5: 0.0.0.0
P6: 00000000
P7: PCH_3D_FROM_bvckup2_1_78_13_0_x64_no_aslr+0x000000000026AAD5
P8: c0000005
P9: 0000000000000008
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2C49.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bvckup2-1.78.13._8112461ecf5da9391c41171af0da2d88f55cbb6_98166b12_4b3e2236
Analysis symbol:
Rechecking for solution: 0
Report Id: 899cea30-f56e-42d9-be97-01f4e95ef177
Report Status: 524388

Checksums

f3df3557d88aba73bf86c5a43b7d0a04f61ac480 *bvckup2-1.78.13.0-x64.no-aslr.exe
34f8fd63d0a15df85cca3c4c56af5a679f3b3c06 *bvckup2-1.78.13.0-x64.fixes.exe
69231bc103d2460a888db68575eea64e35be4999 *bvckup2-1.78.13.0-x64.fixes-no-aslr.exe
@Mattiwatti

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2018

Thanks @paulwarwicker.

The event viewer log entries use a bit of an erm... 'special' (some might say: borderline useless) format, but the most important part is this:
P7: PCH_3D_FROM_bvckup2_1_78_13_0_x64_fixes_no_aslr+0x000000000026AAD5

'P7' will be 'parameter 7', and I don't know what 'PCH_3D_FROM' stands for. But the last part can be read as '<filename> at RVA 0x26AAD5', or, since the file has ASLR disabled, simply: <imagebase> (= 0x140000000) + 0x26AAD5 = 0x14026AAD5. This is the address of the program counter (RIP) at the time of the crash. P8 = c0000005 is the fault status: access violation/segfault.

RVA 0x26AAD5 lies within the UPX0 section of the exe, so we can conclude a few things from this:

  • The kernel section and process are created successfully and the section view is mapped into the process. Otherwise there could be no access violation.
  • The ntdll loader has initialized successfully, otherwise the faulting address would be ntdll+XXXX, and the status some other one indicating initialization failure.
  • The AV occurs during unpacking, not in the program itself.

This is 'good news' since it means the crash can be debugged using a normal user mode debugger and doesn't involve reverse engineering ntdll loader stuff.

For reference, this is what the state of the program should look like at this point:
3.91
So this should be an (indirect, via a jmp) call to LoadLibraryA. The dump view in the bottom left view shows the memory RCX is pointing at, that's the function argument. (NB: the reported fault address 0x14026AAD5 is actually one instruction below this one; that's because RIP points to the next instruction.)

Unfortunately this presents a bit of an issue as the original exe was packed using UPX 3.91, which has some fairly major differences in this part of the assembly stub compared to the current version. Furthermore the screenshot above is meaningless to me (but maybe not to @jreiser, which is why I included it.)

So as far as I can see, there are still two variables/questions left to eliminate:

  1. Whether this is a bug in a previous version of UPX that has already been fixed (either by accident or on purpose, and possibly in combination with one of the above mentioned issues)
  2. Whether there is some fascist enterprise policy at work on your machine that is force relocating images that do not support ASLR, since the fault report doesn't mention the address the executable was loaded at.

To answer (1), here is another ZIP (sorry...) containing the same file, recompressed using UPX from the current devel branch @ d698eb6:
bvckup2-1.78.13.0-x64.fixes-no-aslr-devel.zip

C:\Users\Matti\Desktop\bvckup2>sha1sum *.exe
1a3755b4c7cc4547d1fabbaf5de37ca957d722eb  bvckup2-1.78.13.0-x64.fixes-no-aslr-devel.exe

The relocation directory got KIA in the process of doing this as per my first post, but that shouldn't be an issue as we've already abandoned ASLR for the moment anyway.
Here is what the same point (now RVA 0x268876) should look like:
3.95

If this file still crashes, I honestly can't think of another reason for this code causing an AV other than (2), despite how overwhelmingly stupid such a policy would be (since it is almost guaranteed to crash the exe, it would be much better to simply show a dialog saying 'this file has been blocked from loading' or somesuch.) So in that case, please confirm the image base is correct 😉

Running all 3 caused AVG to sniff the executable and declare that "You've discovered a very rare file".

This is not surprising, as all three executables are obviously UPX-compressed but have a modified (non-standard) header. This will set off any AV as it is typical malware behaviour. The only way you could make an antivirus angrier is if you literally named the file virus.exe.

@paulwarwicker

This comment has been minimized.

Copy link

commented Mar 7, 2018

Latest executable also failed, so I think this is looking more like a situation that affects me and me alone.

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: bvckup2-1.78.13.0-x64.fixes-no-aslr-devel.exe
P2: 1.78.13.0
P3: 5a1c2d50
P4: StackHash_6eb4
P5: 0.0.0.0
P6: 00000000
P7: PCH_40_FROM_bvckup2_1_78_13_0_x64_fixes_no_aslr_devel+0x000000000026887C
P8: c0000005
P9: 0000000000000008
P10: 
@GR-C

This comment has been minimized.

Copy link

commented Apr 4, 2018

Hello everybody,

I've encountered the same problem with my application. The error 0xc0000005 (means "Access violation") happens when you enable the Windows Defender Exploit Guard Setting Force randomization for images (Mandatory ASLR) (Start Windows Defender Security Center > App & browser control (on the sidebar) > Exploit protection > Exploit protection settings). This setting is available since Windows 10 1709.

Starting my app compressed with upx version 3.94 (2017-05-12) causes the app to crash on start. Starting the app without upx compression all works well.

Update:
Compressing the app with upx version 3.91 does not lead to the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.