Debugging segfault on aarch64 (ARM)

[Zenexer]

I'm encountering an occasional segfault in PHP 8.1 on aarch64. The coredumps I've collected point to pcre2, and disabling JIT seems mitigate the segfaults. However, I haven't been able to get a reliable repro because it's quite a rare issue.

I posted details at oerdnj/deb.sury.org#1721, but there probably isn't much there that will help other than a sparse backtrace from the coredump. Do you have any advice for debugging the issue further or developing a reliable repro?

Here's the backtrace copied from the other issue, for convenience--not too helpful, I'm afraid:

#0  vld1q_u8 (__a=0xffff9e000000 <error: Cannot access memory at address 0xffff9e000000>) at /usr/lib/gcc/aarch64-linux-gnu/10/include/arm_neon.h:17550
#1  ffcps_1_utf (str_end=0xffff9dfffffa "", str_ptr=0xffff9e000000 <error: Cannot access memory at address 0xffff9e000000>, offs1=8, offs2=<optimized out>,
    chars=<optimized out>) at src/pcre2_jit_neon_inc.h:190
#2  0x0000ffffa6662118 in ?? ()
#3  0x0000ffff9dfffff8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

There isn't much else that the segfaults have in common. They're happening during arbitrary PHP requests handled by XenForo, relatively popular software. There doesn't appear to be anything that the requests have in common. Based on the timing I'm seeing, it's likely happening toward the end of each request; XenForo's PCRE usage happens during the beginning of each request, which makes developing a repro more difficult.

Installed package version is 10.39-2+0~20211122.14+debian11~1.gbp0d570b

[zherczeg]

Based on this trace the issue is in the simd accelerated character search. The string ends at 0xffff9dfffffa, but the code tries to load the next aligned word from 0xffff9e000000, which is probably an invalid address, and an error occurs. Maybe stopping at ffcps_1_utf when str_ptr>str_end condition happens could help.

[Seldaek]

I got another one on PHP 8.1 / aarch64, not sure if related but also fairly cryptic (to me anyway..)

Core was generated by `php -dmemory_limit=512M bin/console cache:warmup --env=prod --no-debug'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000ffffad30a518 in ?? () from /lib/aarch64-linux-gnu/libpcre2-8.so.0

It does seem to be related to PCRE JIT as disabling that in PHP fixes it. I wasn't able to get more info than that despite installing libpcre2-dev, not sure what else I am missing.

[zherczeg]

Hard to tell. The original report blamed a C function, not jit code. To do something with it, I would need a pattern/input pair. Does it happen with the latest master?

[Seldaek]

Ah sorry I was misled by the pcre2_jit_neon_inc.h containing jit. I will see if I can get master compiled and run that. Getting the pattern/input is probably very tricky as it's a huge piece of code which ends up crashing, but I'll see what I can do.

[Seldaek]

Running master I now get this:

#0  0x0000ffff9fd55d98 in ffcps_default () from /lib/aarch64-linux-gnu/libpcre2-8.so.0
#1  0x0000ffff98623404 in ?? ()

I had no luck figuring out what code exactly causes it sorry :/

[zherczeg]

pcre2_jit_neon_inc.h generates these ffcps_something functions with macros. Then it might be related to the original issue.

[Seldaek]

Ok, well I don't have an easy way for you to repro unfortunately but at least I can reliably reproduce it so I can offer to try out a patch in case you have any "stabbing in the dark" fixes in mind :)

[Zenexer]

You have a reliable repro? I gave up because it was happening for maybe one in a thousand requests, which made debugging quite tedious. I may be able to patch it--or at least narrow it down--if I have a better repro.

[zherczeg]

The code is an external contribution so I don't know much about it, but it seems a buffer overrun happens, probably a while loop with a ptr<=end check instead of a ptr<end.

[svenauhagen]

Hi,

I am running into the same problem on our server with php8.1 and libpcre2.
I compiled libpcre2 without optimizations to get a better output.
It happens every now and then so it is not really consistent.

Here is the latest crash report, I will try to find the problem but help is appreciated :)

(gdb) bt full
#0 0x0000ffffbac124d0 in vld1q_u8 (__a=0xffff98600000 <error: Cannot access memory at address 0xffff98600000>) at /usr/lib/gcc/aarch64-linux-gnu/10/include/arm_neon.h:17551
No locals.
oerdnj/deb.sury.org#1 ffcps_1 (str_end=0xffff985fffff "", str_ptr=0xffff98600000 <error: Cannot access memory at address 0xffff98600000>, offs1=11, offs2=10, chars=1164258605)
at src/pcre2_jit_neon_inc.h:190
qw = {mem = "\000\000\000\377", '\000' <repeats 11 times>, dw = {4278190080, 0}}
ic = {x = 1164258605, c = {c1 = 45 '-', c2 = 45 '-', c3 = 101 'e', c4 = 69 'E'}}
compare1_type = compare_match1
compare2_type = compare_match1i
cmp1a = {45 <repeats 16 times>}
cmp1b = {0 <repeats 16 times>}
cmp2a = {101 <repeats 16 times>}
cmp2b = {32 <repeats 16 times>}
diff = 1
char1a = 45 '-'
char2a = 101 'e'
char1b = 45 '-'
char2b = 69 'E'
p1 = 0xffff98600009 <error: Cannot access memory at address 0xffff98600009>
align_offset = 10
data = {0, 0, 0, 255, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0, 0}
prev_data = {112, 108, 117, 103, 105, 110, 115, 47, 119, 111, 111, 99, 111, 109, 109, 101}
data2 = {255, 0, 0, 255, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
eq = {0, 0, 0, 255, 0 <repeats 12 times>}
oerdnj/deb.sury.org#2 0x0000ffff9d512090 in ?? ()
No symbol table info available.
oerdnj/deb.sury.org#3 0x00ff0000ff000000 in ?? ()
No symbol table info available.

[zherczeg]

it looks a similar problem, str_end=0xffff985fffff and Cannot access memory at address 0xffff98600000.

[svenauhagen]

yes, looks like a one off in PHP at some point during processing

[Zenexer]

In my case, the lack of consistency remains the primary impediment to narrowing down the cause. I still can't repro it outside of production. arm_neon.h is quite dense, and I'm unfamiliar with NEON instructions, so I'm not able to easily spot any logic errors by reviewing the code. And, of course, by the time it segfaults, a lot of context leading up to the crash has been lost.

I'm willing to help in any way I can, but I don't really have the necessary expertise to handle this efficiently.

[svenauhagen]

This piece of code crashes my server in the end:

const PATH_PATTERN_SEARCH_JSON = '#(DOMAIN_PLACEHOLDER)([a-z]+)([_A-Z])(-[-_a-z0-9]+).json$#i';
$domain_replace = 'default';
$search_pattern = str_replace( 'DOMAIN_PLACEHOLDER', $domain_replace, PATH_PATTERN_SEARCH_JSON );
preg_replace("#(woocommerce-)([a-z]+)([_A-Z])(-[-_a-z0-9]+).json$#i", $search_pattern, "/var/www/wordpress/wp-content/languages/plugins/woocommerce-en_US-d5ba42
31878fa2e7d350f73c3da91138.json");

But it takes a lot of requests to do so.
I am also not very familiar with the NEON instructions and it does not look like the neon code is the problem to begin with. ffcps_1 gets wrong arguments if from php I think.