Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault with aarch64 (arm64) binaries and 64k page size runtime #254

Open
danrue opened this Issue Feb 26, 2019 · 9 comments

Comments

Projects
None yet
3 participants
@danrue
Copy link

danrue commented Feb 26, 2019

What's the problem (or question)?

When running skipgen, which distributes upx'd binaries, a segfault occurs at skipgen runtime when the linux kernel is built with 64k page size on aarch64. skipgen binaries that are not run through upx work on both 4k page size and 64k page size, and upx binaries do work on 4k page size.

What should have happened?

skipgen should run without segfaulting on 64k page size.

Do you have an idea for a solution?

No - not familiar with upx internals.

How can we reproduce the issue?

Accessing a 64k page size aarch64 system may be difficult. I can provide a qemu userspace if necessary that can be used to reproduce locally.

Please tell us details about your environment.

  • UPX version used (upx --version):
    $ upx --version
    upx 3.95
    UCL data compression library 1.03
    zlib data compression library 1.2.11
  • Host Operating System and version:
    Linux 4.20.10-200.fc29.x86_64 #1 SMP Fri Feb 15 18:33:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Host CPU architecture:
    x86_64
  • Target Operating System and version:
    Linux 4.20.4 #1 SMP PREEMPT Fri Feb 22 18:14:42 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
  • Target CPU architecture:
    aarch64
@jreiser

This comment has been minimized.

Copy link
Contributor

jreiser commented Feb 27, 2019

Please upload ("Attach files by"...) an actual (small) pre-UPX executable that suffers SIGSEGV when compressed by UPX and run on a 64k page size machine. Apply zip or gzip to reduce upload size, and to satisfy github.
If uploading is not possible, then please run readelf --segments --dynamic my_app, then copy+paste the results into a Comment here. Put a line with three backticks (decimal 96, octal 0140, hex 0x60; often to the left of the '1' digit in the upper row of a keyboard) both before and after the paste, so that columns line up nicely. Thank you.
[Edit: added --dynamic to show if DF_1_PIE when ET_DYN.]

@danrue

This comment has been minimized.

Copy link
Author

danrue commented Feb 27, 2019

Hi @jreiser, thanks for helping.

The upx-compressed binary can be downloaded at https://github.com/Linaro/skipgen/releases/download/v0.2.3/skipgen_0.2.3_linux_arm64.tar.gz.

Below is the output of the readelf command requested, first with the upx compressed binary, and second with the uncompressed binary.

compressed with upx:

drue@xps:~/go/src/github.com/linaro/skipgen$ ls -l dist/linux_arm64/skipgen 
-rwxrwxr-x. 1 drue drue 768364 Feb 26 21:11 dist/linux_arm64/skipgen
drue@xps:~/go/src/github.com/linaro/skipgen$ file dist/linux_arm64/skipgen
dist/linux_arm64/skipgen: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
drue@xps:~/go/src/github.com/linaro/skipgen$ readelf --segments --dynamic dist/linux_arm64/skipgen 

Elf file type is EXEC (Executable file)
Entry point 0xcadfc
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
                 0x00000000000bb864 0x00000000000bb864  R E    0x10000
  LOAD           0x0000000000000000 0x00000000000cc000 0x00000000000cc000
                 0x0000000000000000 0x0000000000169138  RW     0x1000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x8

There is no dynamic section in this file.

Built but not compressed with upx:

drue@xps:~/go/src/github.com/linaro/skipgen$ ls -l dist/linux_arm64/skipgen 
-rwxrwxr-x. 1 drue drue 2122272 Feb 26 21:14 dist/linux_arm64/skipgen
drue@xps:~/go/src/github.com/linaro/skipgen$ file dist/linux_arm64/skipgen
dist/linux_arm64/skipgen: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=M8_ev1WGidG78sPbaDn3/8NCpZHuGYsyNxFfrobtm/6tgZPkQxOYlO4AVr7OIg/sQpkkqvjvxLT5ogDv_DT, stripped
drue@xps:~/go/src/github.com/linaro/skipgen$ readelf --segments --dynamic dist/linux_arm64/skipgen 

Elf file type is EXEC (Executable file)
Entry point 0x5d460
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000010040 0x0000000000010040
                 0x0000000000000188 0x0000000000000188  R      0x10000
  NOTE           0x0000000000000f9c 0x0000000000010f9c 0x0000000000010f9c
                 0x0000000000000064 0x0000000000000064  R      0x4
  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
                 0x00000000000d0120 0x00000000000d0120  R E    0x10000
  LOAD           0x00000000000e0000 0x00000000000f0000 0x00000000000f0000
                 0x0000000000104cd6 0x0000000000104cd6  R      0x10000
  LOAD           0x00000000001f0000 0x0000000000200000 0x0000000000200000
                 0x0000000000016220 0x0000000000035138  RW     0x10000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x8
  LOOS+0x5041580 0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000         0x8

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .note.go.buildid 
   02     .text .note.go.buildid 
   03     .rodata .typelink .itablink .gosymtab .gopclntab 
   04     .noptrdata .data .bss .noptrbss 
   05     
   06     

There is no dynamic section in this file.
@danrue

This comment has been minimized.

Copy link
Author

danrue commented Feb 27, 2019

Here is the original skipgen binary.

skipgen.gz

@jreiser jreiser changed the title Segfault with amd64 binaries and 64k page size runtime Segfault with aarch64 (arm64) binaries and 64k page size runtime Feb 27, 2019

@jreiser

This comment has been minimized.

Copy link
Contributor

jreiser commented Feb 27, 2019

Thank you for Attach-ing the skipgen.gz and skipgen_0.2.3_linux_arm64.tar.gz files.
I have identified one 64k issue, but want to check for more. Please run strace -e trace=open,openat,mmap,mprotect,munmap ./skipgen (using the one compressed by upx) then copy+paste the output here. strace should report about 27 lines.

@jreiser

This comment has been minimized.

Copy link
Contributor

jreiser commented Feb 28, 2019

Or, please provide access to a qemu-aarch64 which runs on x86_64 (I notice that you use Fedora 29, which is great) and enforces 64kB pages.

@danrue

This comment has been minimized.

Copy link
Author

danrue commented Feb 28, 2019

I'm running Fedora but the target system is running a custom openembedded build.

Here's the strace. I don't think it's what you expected to see..

root@juno:~# strace -e trace=open,openat,mmap,mprotect,munmap ./skipgen
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)

The other usecase that I haven't tested but you should know about is 16k pages, which is also a supported configuration.

@jreiser

This comment has been minimized.

Copy link
Contributor

jreiser commented Feb 28, 2019

In this specific case the problem is that the first [PT_]LOAD ends at (0x10000 + 0xbb864) = 0xcb864 with protection "R E", while the second [PT_]LOAD begins at 0xcc000 with protection "RW", and the differing protections cannot be assigned to the same 64KB page which contains 0xcc000. So SIGSEGV from execve() is a reasonable error indication. UPX can "fix" the problem, but probably there will be a complication. The file may be too short: there may not be enough 64KB chunks to "back" the entire mapping of the address range [0x10000, +0xbb864), namely 0xc0000 bytes. It's unclear what the kernel is required to do in this case. The nicest thing would be for the kernel to zero the bytes in the address range [0xcb864, 0xd0000), or at least in [0xcc000, 0xd0000) as implied by a filesystem blocksize of 4KB, but that is not guaranteed. The kernel could say, "I map 64KB pages, the file does not contain enough to map the last whole 64KB page of address space; you lose." So UPX probably will have to pad the compressed output to a 64KB boundary. What a waste.

jreiser added a commit that referenced this issue Mar 2, 2019

arm64 (aarch64) and PAGE_SIZE > 4KiB
Future: use prctl(PR_SET_MM, PR_SET_MM_START_BRK, addr)
#254
	modified:   p_lx_elf.cpp
	modified:   stub/src/amd64-linux.elf-main.c
@zatrazz

This comment has been minimized.

Copy link

zatrazz commented Mar 3, 2019

Hi John,

Dan has asked to check this out and I think the issue is UPX is not laying out the PT_LOAD considering
the possible max page size for the architecture. Binutils, for instance, take this in consideration while generating files to avoid this same issue. For instance:

$ cat test.c
#include <stdio.h>
#include <unistd.h>

int main ()
{
long sz = sysconf(_SC_PAGESIZE);
printf ("pagesize=%zu\n", sz);
return 0;
}
$ gcc test.c && readelf --segments ./a.out | head -n 28

Elf file type is DYN (Shared object file)
Entry point 0x670
There are 9 program headers, starting at offset 64

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x00000000000001f8 0x00000000000001f8 R 0x8
INTERP 0x0000000000000238 0x0000000000000238 0x0000000000000238
0x000000000000001b 0x000000000000001b R 0x1
[Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000864 0x0000000000000864 R E 0x10000
LOAD 0x000000000000fd68 0x000000000001fd68 0x000000000001fd68
0x00000000000002a8 0x00000000000002a9 RW 0x10000
DYNAMIC 0x000000000000fd78 0x000000000001fd78 0x000000000001fd78
0x0000000000000200 0x0000000000000200 RW 0x8
NOTE 0x0000000000000254 0x0000000000000254 0x0000000000000254
0x0000000000000044 0x0000000000000044 R 0x4
GNU_EH_FRAME 0x000000000000085c 0x000000000000085c 0x000000000000085c
0x0000000000000008 0x0000000000000008 R 0x4
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
GNU_RELRO 0x000000000000fd68 0x000000000001fd68 0x000000000001fd68
0x0000000000000298 0x0000000000000298 RW 0x8

You can see that for LOAD segments, both the VirtAddr and Align are set according with maximum architecture pagesize (for aarch64 64k in this case). You can change to assume the underlying 4K page:

$ gcc test.c -Wl,-z,max-page-size=0x1000 && readelf --segments ./a.out | head -n 28

Elf file type is DYN (Shared object file)
Entry point 0x670
There are 9 program headers, starting at offset 64

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x00000000000001f8 0x00000000000001f8 R 0x8
INTERP 0x0000000000000238 0x0000000000000238 0x0000000000000238
0x000000000000001b 0x000000000000001b R 0x1
[Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000864 0x0000000000000864 R E 0x1000
LOAD 0x0000000000000d68 0x0000000000001d68 0x0000000000001d68
0x00000000000002a8 0x00000000000002a9 RW 0x1000
DYNAMIC 0x0000000000000d78 0x0000000000001d78 0x0000000000001d78
0x0000000000000200 0x0000000000000200 RW 0x8
NOTE 0x0000000000000254 0x0000000000000254 0x0000000000000254
0x0000000000000044 0x0000000000000044 R 0x4
GNU_EH_FRAME 0x000000000000085c 0x000000000000085c 0x000000000000085c
0x0000000000000008 0x0000000000000008 R 0x4
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
GNU_RELRO 0x0000000000000d68 0x0000000000001d68 0x0000000000001d68
0x0000000000000298 0x0000000000000298 RW 0x8

So I think UPX should do the same strategy instead of trying to change kernel behavior regarding ELF specification.

A possible option is also to add an option to assume underlying pagesize, so it would be possible to optimize the VM usage if the idea is not to deploy on system with different page sizes.

Also keep in mind other architectures also may fail, for instance powerpc which also defines multiple pages sizes.

@jreiser

This comment has been minimized.

Copy link
Contributor

jreiser commented Mar 4, 2019

Please try the tip of devel branch. The [PT_}LOADs in the compressed executable output generated by UPX should retain the alignment that appears in the input to UPX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.