Possible stack overflow issue

[fgeek]

Reproducer: yara-re-stack-overflow.txt (SHA1: 58002f9ad135c3840d2e7ad07a100ed9b911f016)

./bin/yara yara-re-stack-overflow.txt /usr/bin/strings
==24458==ERROR: AddressSanitizer: stack-overflow on address 0x7ffd55429918 (pc 0x561ee987cdf0 bp 0x7ffd5542ad10 sp 0x7ffd554298f0 T0)
    #0 0x561ee987cdef in _yr_re_emit /home/hsalo/src/yara/libyara/re.c:757
<snip>
    #251 0x561ee987df14 in _yr_re_emit /home/hsalo/src/yara/libyara/re.c:934

SUMMARY: AddressSanitizer: stack-overflow /home/hsalo/src/yara/libyara/re.c:757 in _yr_re_emit
==24458==ABORTING

00000000  72 75 6c 65 20 41 30 30  30 30 30 30 30 30 30 30  |rule A0000000000|
00000010  30 20 7b 0a 73 74 72 69  6e 67 73 3a 0a 09 24 30  |0 {.strings:..$0|
00000020  30 30 30 30 30 30 30 30  30 30 30 20 3d 2f 30 5b  |00000000000 =/0[|
00000030  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  |0000000000000000|
*
00000290  30 30 30 30 30 30 30 30  30 30 30 30 30 5d 30 30  |0000000000000]00|
000002a0  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  |0000000000000000|
*
00001060  30 30 30 30 30 30 3f 30  30 30 30 30 30 30 30 30  |000000?000000000|
00001070  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  |0000000000000000|
*
000016f0  30 30 30 30 30 2f 3d 3d  30                       |00000/==0|
000016f9

[plusvic]

This is an interesting case. The regular expression is not long enough to reach the limit of 6000
characters (RE_MAX_AST_LEVELS), but it's long enough to exhaust the stack when AddressSanitizer is enabled. This is because AddressSanitizer consumes a lot of stack space, and therefore the number of nested calls the program can do safely is drastically reduced. The same input doesn't cause an stack overflow if AddressSanitizer is disabled as the program has more stack available.

There has been some back and forth for RE_MAX_AST_LEVELS in the past (see #674) but 5000 some to be the sweet spot.