MacOS: Segfault during an on-trace exception handling

[mkokryashkin]

LuaJIT crashes with a segmentation fault while handling and on-trace exception.

• OS: MacOS
• OS Version: 12.6.5 (older versions of MacOS 12 too, but MacOS 13 is ok)
• Architecture: arm64

Reproducer:

local function memory_payload()
    local t = {}
    for i = 1, 1e10 do
        t[ffi.new('uint64_t')] = i
    end
end

local res = pcall(memory_payload)
assert(res == false)

How to run:

Just type luajit test.lua.

Actual behavior

Segmentation fault with the following backtrace:

* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x8)
  * frame #0: 0x00000001a093acb8 libunwind.dylib`unw_set_reg + 264
    frame #1: 0x000000010256ea64 luajit`err_unwind_jit(version=1, actions=6, uexclass=5500374307216568834, uex=0x0000600002c48000, ctx=0x000000016d8aa228) at l
j_err.c:530:7
    frame #2: 0x00000001a093b5f4 libunwind.dylib`unwind_phase2 + 468
    frame #3: 0x00000001a093b408 libunwind.dylib`_Unwind_RaiseException + 712
    frame #4: 0x000000010256eb98 luajit`err_raise_ext(g=0x00000001028003d8, errcode=2) at lj_err.c:752:3
    frame #5: 0x000000010256eb04 luajit`lj_err_throw(L=0x0000000102800378, errcode=2) at lj_err.c:768:3
    frame #6: 0x000000010256f1fc luajit`lj_err_run(L=0x0000000102800378) at lj_err.c:880:3
    frame #7: 0x000000010256f640 luajit`err_msgv(L=0x0000000102800378, em=LJ_ERR_TABOV) at lj_err.c:903:3
    frame #8: 0x000000010256f544 luajit`lj_err_msg(L=0x0000000102800378, em=LJ_ERR_TABOV) at lj_err.c:909:3
    frame #9: 0x0000000102587900 luajit`lj_tab_resize [inlined] newhpart(L=0x0000000102800378, t=0x0000000102810cd8, hbits=27) at lj_tab.c:62:5
    frame #10: 0x00000001025878d8 luajit`lj_tab_resize(L=0x0000000102800378, t=0x0000000102810cd8, asize=0, hbits=27) at lj_tab.c:278:5
    frame #11: 0x00000001025884f8 luajit`rehashtab(L=0x0000000102800378, t=0x0000000102810cd8, ek=0x0000000102800540) at lj_tab.c:387:3
    frame #12: 0x0000000102589058 luajit`lj_tab_newkey(L=0x0000000102800378, t=0x0000000102810cd8, key=0x0000000102800540) at lj_tab.c:471:2

Expected behavior

Successful error handling with no segfault.