jit: introduce new JIT state for XS calls

Current JIT recording semantics consider FFI calls as an atomic action
regarding the LuaJIT VM: it doesn't change the guest stack at the
result, and the only way the flow yields to the VM after leaving it --
return through the corresponding FFI machinery to the place it enters.

However, there are ways to break this atomicity via FFI: the lua_State
can be replaced with another one, or one can make another Lua call via
LuaC API within C function called via FFI on the current lua_State
(@mraleph called the latter approach an "FFI sandwich", and I'm strictly
against eating it).

E.g. Tarantool fibers do not respect the native Lua coroutine switching
mechanism, thus current lua_State can be substituted via FFI while
recording, with no notification to compiler, tracing another lua_State.

Furthermore there is triggers machinery in Tarantool creating the "FFI
sandwiches" mentioned above to execute a Lua function, with the same
lua_State the action triggering its execution via FFI does.

Thereby I propose to mask the JIT state with the introduced LJ_TRACE_XS
bit, while calling C function for already recorded FFI semantics. If C
call is "atomic" as described above, the JIT state is unmasked and the
recording is resumed. Otherwise the recording is asyncroniously
aborted.

Fixes: tarantool gh-4427
Fixes: tarantool gh-4491

src/lj_ccall.c

@@ -1147,7 +1147,9 @@ int lj_ccall_func(lua_State *L, GCcdata *cd)
     gcsteps = ccall_set_args(L, cts, ct, &cc);
     ct = (CType *)((intptr_t)ct-(intptr_t)cts->tab);
     cts->cb.slot = ~0u;
+    lj_trace_maskxs(L);
     lj_vm_ffi_call(&cc);
+    lj_trace_unmaskxs(L);
     if (cts->cb.slot != ~0u) {  /* Blacklist function that called a callback. */
       TValue tv;
       setlightudV(&tv, (void *)cc.func);

src/lj_jit.h

@@ -137,7 +137,8 @@ typedef enum {
   LJ_TRACE_START,	/* New trace started. */
   LJ_TRACE_END,		/* End of trace. */
   LJ_TRACE_ASM,		/* Assemble trace. */
-  LJ_TRACE_ERR		/* Trace aborted with error. */
+  LJ_TRACE_ERR,		/* Trace aborted with error. */
+  LJ_TRACE_XS = 0x20	/* External subroutine is called via FFI */
 } TraceState;
 
 /* Post-processing action. */

src/lj_trace.h

@@ -39,6 +39,12 @@ LJ_FUNCA void LJ_FASTCALL lj_trace_stitch(jit_State *J, const BCIns *pc);
 LJ_FUNCA int LJ_FASTCALL lj_trace_exit(jit_State *J, void *exptr);
 
 /* Signal asynchronous abort of trace or end of trace. */
+#define lj_trace_maskxs(L) \
+  (G2J(G(L))->state |= G2J(G(L))->state & LJ_TRACE_ACTIVE ? LJ_TRACE_XS : 0)
+
+#define lj_trace_unmaskxs(L) \
+  (G2J(G(L))->state &= G2J(G(L))->L == (L) ? ~LJ_TRACE_XS : 0)
+
 #define lj_trace_abort(g)	(G2J(g)->state &= ~LJ_TRACE_ACTIVE)
 #define lj_trace_end(J)		(J->state = LJ_TRACE_END)