Fast unwinding doesn't work with thumb.

[chefmax]

• Compile ASan library with -mthumb option.
• cat test.c

#include <stdlib.h>

__attribute__((noinline))
int baz (int N) {
  int *p = (int *) malloc(10 * sizeof(int));
  return p[N];
}

__attribute__((noinline))
int bar (int N) {
  return baz(N);
}

__attribute__((noinline))
int foo (int N) {
  return bar(N);
}

int main () {
  return foo(10);
}

• compile test.c with -mthumb:

$ clang -target armv7l-tizen-linux-gnueabi  -march=armv7-a   -mfloat-abi=softfp -mfpu=neon --sysroot=$TOOLCHAIN_ROOT/armv7l-tizen-linux-gnueabi/sys-root/ -B$TOOLCHAIN_ROOT/ -fsanitize=address 3.c -S -mthumb -fno-omit-frame-pointer

• Run on target:

-sh-4.1# ASAN_OPTIONS=allow_addr2line=true  ./a.out 
=================================================================
==809==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xb4d00ff8 at pc 0x000ac12d bp 0xbdceebf8 sp 0xbdceebf4
READ of size 4 at 0xb4d00ff8 thread T0
    #0 0xac12b in baz /tmp/3.c:6
    #1 0xac143 in bar /tmp/3.c:12
    #2 0xac157 in foo /tmp/3.c:18
    #3 0xac16b in main /tmp/3.c:23
    #4 0xb6094867 in __libc_start_main /home/max/build/v6/sources/glibc/csu/libc-start.c:289

0xb4d00ff8 is located 0 bytes to the right of 40-byte region [0xb4d00fd0,0xb4d00ff8)
allocated by thread T0 here:
    #0 0x96cfd in __interceptor_malloc /home/max/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:52

SUMMARY: AddressSanitizer: heap-buffer-overflow /tmp/3.c:6 in baz
Shadow bytes around the buggy address:
  0x369a01a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a01b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a01c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a01d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a01e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x369a01f0: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00[fa]
  0x369a0200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a0210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a0220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a0230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x369a0240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==809==ABORTING

After some investigation, I found out, that in Thumb mode we use r7 register as fp (that corresponds to ARMv6 architecture) and r14 as lr. Now, looking to __interceptor_malloc prologue:

00096ca4 <__interceptor_malloc>:
   96ca4:       e92d 47f0       stmdb   sp!, {r4, r5, r6, r7, r8, r9, sl, lr}
   96ca8:       af03            add     r7, sp, #12
   96caa:       f5ad 6d84       sub.w   sp, sp, #1056   ; 0x420
   96cae:       4680            mov     r8, r0
   96cb0:       a804            add     r0, sp, #16

Here, we have three extra words between saved fp and lr (r8, r9, sl), thus corresponding code from BufferedStackTrace::FastUnwindStack extracts wrong value for pc in unwinding loop:

uhwptr pc1 = frame[1];

Moreover, it's not guaranteed that we would have exactly three words between fp and lr, consider foo prologue:

000ac148 <foo>:
   ac148:       b580            push    {r7, lr}
   ac14a:       466f            mov     r7, sp
   ac14c:       b088            sub     sp, #32

So, for Thumb, we don't actually know from which slot we can extract the saved lr: from fp[1], fp[2], fp[3] or from fp[4].

Right now, I don't see a general solution for the issue, but IMHO it would be nice to extract at least first lr value to find out malloc's location in user code.

[yugr]

Have you tried with GCC?

[chefmax]

Yeah, GCC isn't better here (it generates slightly different code, but symptoms are the same). The only working configuration I saw is non-optimized compiler-rt + non-optimized user code (both in Thumb).

[yugr]

Some relevant reading: https://llvm.org/bugs/show_bug.cgi?id=18505 . I'm not sure anything can be done here (except maybe documenting this in a FAQ?).

[chefmax]

Perhaps we can also define CAN_FAST_UNWIND to 0 for thumb?

[yugr]

This would rule out the case of Thumb libasan and ARM user code (which should be working now?). And we'll still have the case of Thumb user code calling ARM libasan (which can not be detected until runtime).

[chefmax]

This would rule out the case of Thumb libasan and ARM user code (which should be working now?).

No, this doesn't work because we can't leave out zero frame (__interceptor_malloc) with meaningful LR value. Moreover, if we could, ARM uses r11 as a frame pointer, while Thumb uses r7, so the second extracted fp value is likely to be invalid if we mix ARM with Thumb.

And we'll still have the case of Thumb user code calling ARM libasan (which can not be detected until runtime).

I tend to think that fast unwinding can work only by accident here, but we don't want to change ARM libasan behavior.

[kcc]

I think this is exactly https://llvm.org/bugs/show_bug.cgi?id=18505.
Do you folks mind if I close this bug as a dup of the above?

• @eugenis

[yugr]

Sure.

[chefmax]

Let me close it by myself, sorry for the noise.