Segmentation fault during on-trace exception handling

[mkokryashkin]

On the LuaJIT 2.1 branch (8635cba at the moment), if one makes a build like this:

make -j CCDEBUG=" -g -ggdb3"

And then runs the following script:

function on_gc(t)
end;
function test_finalizers()
    local result = {}
    local i = 1
    local ffi = require('ffi')
    while true do
        result[i] = ffi.gc(ffi.cast('void *', 0), on_gc)
        i = i + 1
    end
    return "done"
end;
local ok, err = pcall(test_finalizers)
print(ok, err)
local ok, err = pcall(test_finalizers)
print(ok, err)

It results in a successfully handled pcall for the first case, and a segmentation fault for the second case:

false   test.lua:8: table overflow
[1]    416059 segmentation fault (core dumped)  ./src/luajit test.lua

Backtrace:

#0  0x000055555555e39d in err_msgv (L=L@entry=0x7ffff7c86380, em=em@entry=LJ_ERR_TABOV)
    at lj_err.c:878
#1  0x000055555555e40b in lj_err_msg (L=L@entry=0x7ffff7c86380, em=em@entry=LJ_ERR_TABOV)
    at lj_err.c:888
#2  0x0000555555581fdf in lj_tab_resize (L=0x7ffff7c86380, t=0x7ffff7c92e70, asize=0,
    hbits=<optimized out>) at lj_tab.c:239
#3  0x000055555558219c in rehashtab (L=<optimized out>, t=<optimized out>, ek=<optimized out>)
    at lj_tab.c:367
#4  0x0000555555581b1e in lj_tab_newkey (L=0x7ffff7c86380, t=0x7ffff7c92e70,
    key=0x7fffffffe228) at lj_tab.c:452
#5  0x0000555555581a65 in lj_tab_set (L=<optimized out>, t=<optimized out>,
    key=<optimized out>) at lj_tab.c:566
#6  0x00005555555a275c in lj_cdata_setfin (L=<optimized out>, cd=0x7ffe77c81928,
    obj=0x7ffff7c90020, it=4294967287) at lj_cdata.c:95
#7  0x0000555540abfed2 in ?? ()
#8  0x00007ffff7c86380 in ?? ()
#9  0x00007ffff7c86550 in ?? ()
#10 0x00007ffe77c81928 in ?? ()
#11 0x00007ffff7c90020 in ?? ()
#12 0x00007ffff7c94240 in ?? ()
#13 0x00007ffff7c86380 in ?? ()
#14 0x0000000000000000 in ?? ()

The issue seems to be the same as in #1004 -- a frame fix is required, but now for err_msgv. Proposed fix:

diff --git a/src/lj_err.c b/src/lj_err.c
index 3ee70b86..9652ef35 100644
--- a/src/lj_err.c
+++ b/src/lj_err.c
@@ -875,6 +875,10 @@ LJ_NORET LJ_NOINLINE static void err_msgv(lua_State *L, ErrMsg em, ...)
   const char *msg;
   va_list argp;
   va_start(argp, em);
+  if (LJ_HASJIT) {
+    TValue *base = tvref(G(L)->jit_base);
+    if (base) L->base = base;
+  }
   if (curr_funcisL(L)) L->top = curr_topL(L);
   msg = lj_strfmt_pushvf(L, err2msg(em), argp);
   va_end(argp);

After its application the result is correct:

false   test.lua:8: table overflow
false   test.lua:4: table overflow

However, there are other places in src/lj_err.c where the same issue is possible -- lj_err_optype and lj_err_optype_call at least, and maybe I've missed some others. So, it comes to mind that a better fix would possibly be a more general frame fix up, so we won't encounter this kind of issue again.

[MikePall]

Fixed. Thanks!