Fix maxslots when recording BC_VARG.

Analyzed by Sergey Kaplun.

(cherry-picked from commit 94ada59)

While recording BC_VARG `J->maxslot` isn't shrunk to the effective stack
top. This leads to dead values stored in the JIT slots and the following
assertion failure for these slots check in `rec_check_slots()`. Note,
that `rec_varg()` modifies `maxslot` only under the condition that
`maxslot` should be increased, but the dead values are left for the
opposite case.

This patch removes the condition inside `rec_varg()` only for the case
when varargs are not defined on trace (`framedepth` is 0), but the
similar issue still occurs for the case when varargs are defined on the
trace.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#8825

src/lj_record.c

@@ -1812,8 +1812,7 @@ static void rec_varg(jit_State *J, BCReg dst, ptrdiff_t nresults)
       }
       for (i = nvararg; i < nresults; i++)
 	J->base[dst+i] = TREF_NIL;
-      if (dst + (BCReg)nresults > J->maxslot)
-	J->maxslot = dst + (BCReg)nresults;
+      J->maxslot = dst + (BCReg)nresults;
     } else if (select_detect(J)) {  /* y = select(x, ...) */
       TRef tridx = J->base[dst-1];
       TRef tr = TREF_NIL;

test/tarantool-tests/lj-1024-varg-maxslot.test.lua

@@ -0,0 +1,23 @@
+local tap = require('tap')
+local test = tap.test('lj-noticket-varg-usedef'):skipcond({
+  ['Test requires JIT enabled'] = not jit.status(),
+})
+
+test:plan(1)
+
+jit.opt.start('hotloop=1')
+
+local counter = 0
+-- luacheck: ignore
+local anchor
+while counter < 3 do
+  counter = counter + 1
+  -- BC_VARG 5 1 0. `...` is nil (argument for the script).
+  -- luacheck: ignore
+  -- XXX: some condition to use several slots on the Lua stack.
+  anchor = 1 >= 1, ...
+end
+
+test:ok(true, 'BC_VARG recording 0th frame depth')
+
+os.exit(test:check() and 0 or 1)