Assertion active hook flag removed hit

[XmiliaH]

I tried to look into SEGV in in lua_resume after lua_yield from hook. Therefore, the following c library was used:

#include <lua.h>
#include <lauxlib.h>
#include <pthread.h>
#include <sched.h>

static volatile int reinstalled = 0;

static void test_hook(lua_State *L, lua_Debug *ar) {
    reinstalled = 0;
    while (!reinstalled);
    if (lua_isyieldable(L)) {
        lua_sethook(L, NULL, 0, 0);
        lua_yield(L, 0);
        return;
    }
}

static void *thread_proc(void* data) {
    lua_State* L = (lua_State*)data;
    while(1) {
        lua_sethook(L, test_hook, LUA_MASKCOUNT, 1);
        reinstalled = 1;
        sched_yield();
    }
}

static int start(lua_State* L) {
    pthread_t t;
    pthread_create(&t, NULL, thread_proc, L);
    return 0;
}

static const luaL_Reg funcs[] = {
  { "start", start },
  { NULL, NULL }
};

LUALIB_API int luaopen_testhook(lua_State *L) {
    lua_createtable(L, 0, sizeof(funcs)/sizeof(funcs[0])-1);
    luaL_register(L, NULL, funcs);
    return 1;
}

compiled with gcc -o testhook.so -lpthread -pthread -shared -I . test.c
and on the Lua side:

local test = require("testhook")

jit.off()

local function y() end

local function x(...)
    y()
end

local c = coroutine.wrap(function()
    test.start()
    while true do
        x(1, 2)
    end
end)

while true do
    c()
end

When running LuaJIT with debug info, LUA_USE_APICHECK, and LUA_USE_ASSERT enabled this will hit quite reliable the assertion

LuaJIT/src/lj_dispatch.c

Line 376 in e2c312e

lj_assertG(hook_active(g), "active hook flag removed");

I do not see a Lua API abuse/misuse here. Furthermore, to me the hit of the assertion seems not related to the original issue.

When assertions are disabled this sometimes result in a segmentation fault:

(gdb) r
Starting program: /home/xmilia/projects/luajit/src/luajit test.lua
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7bfc700 (LWP 11109)]

Thread 1 "luajit" received signal SIGSEGV, Segmentation fault.
lj_dispatch_ins (L=0x7ffff7c34400, pc=0x7ffff7c3c468) at lj_dispatch.c:409
409       const BCIns *oldpc = cframe_pc(cf);
(gdb) bt f
#0  lj_dispatch_ins (L=0x7ffff7c34400, pc=0x7ffff7c3c468) at lj_dispatch.c:409
        olderr = 2
        fn = 0x7ffff7c349c0
        pt = 0x7ffff7c3c3f8
        __func__ = "lj_dispatch_ins"
        cf = 0x0
        oldpc = <optimized out>
        g = <optimized out>
        slots = <optimized out>
#1  0x00005555555855ae in lj_vm_inshook ()
No symbol table info available.
#2  0x00005555555847d6 in lj_ff_coroutine_wrap_aux ()
No symbol table info available.
#3  0x000055555556c5ad in lua_pcall (L=0x7ffff7c32380, nargs=0, nresults=-1, errfunc=<optimized out>) at lj_api.c:1145
        g = 0x7ffff7c323e0
        oldh = 0 '\000'
        ef = 40
        status = <optimized out>
        __func__ = "lua_pcall"
#4  0x000055555555cbfb in docall (L=0x7ffff7c32380, narg=0, clear=0) at luajit.c:122
        status = <optimized out>
        base = 2
#5  0x000055555555db21 in handle_script (argx=<optimized out>, L=0x7ffff7c32380) at luajit.c:292
        narg = 0
        status = <optimized out>
        fname = <optimized out>
        status = <optimized out>
        fname = <optimized out>
        narg = <optimized out>
#6  pmain (L=0x7ffff7c32380) at luajit.c:550
        s = 0x555555601050 <smain>
        argv = <optimized out>
        argn = <optimized out>
        flags = <optimized out>
#7  0x000055555558382e in lj_BC_FUNCC ()
No symbol table info available.
#8  0x000055555556c671 in lua_cpcall (L=<optimized out>, func=<optimized out>, ud=<optimized out>) at lj_api.c:1173
        g = 0x7ffff7c323e0
        oldh = 0 '\000'
        status = <optimized out>
        __func__ = "lua_cpcall"
#9  0x000055555555c6fe in main (argc=2, argv=0x7fffffffde38) at luajit.c:581
        status = <optimized out>
        L = 0x7ffff7c32380

[MikePall]

lua_sethook() (only when setting a count hook) is signal-safe, but not thread-safe.

This makes a difference here, since the code doesn't use locks¹, just careful ordering of memory operations which ought to work on all supported architectures. That is, provided there's no parallel execution. Signal handlers run asynchronously, but suspend the regular thread execution while running.

OK, so rewrite it to send a signal from the other thread (but be careful with one-shot/reentrance semantics) and do the lua_sethook() in the signal handler. If you can still get it to crash, please report here.

¹ Use of locks would require dragging in problematic OS-specific dependencies just for that sole purpose.

[XmiliaH]

It seems that

LuaJIT/src/lj_dispatch.c

Line 336 in e2c312e

g->hookmask = (uint8_t)((g->hookmask & ~HOOK_EVENTMASK) | mask);

was eating up the change done in

LuaJIT/src/lj_dispatch.c

Line 373 in e2c312e

hook_enter(g);

.
This cannot happen in the signal case, so this should be marked as invalid due to API misuse (I was not aware of the signal vs thread safe fact).

[MikePall]

OK. The remaining question is whether signal-safety vs. thread-safety is what the original report is about. I'll ask.

Anyway, it crashes in a different place, so there might still be a different issue there.

[XmiliaH]

Anyway, it crashes in a different place, so there might still be a different issue there.

Maybe, but so far I was not able to get that crash without the assertion being triggered.