Integer underflow during the `string.byte` recording

[Buristan]

When calculating the amount of the results, it is possible that the len will underflow and become positive:

LUA_PATH="src/?.lua;;" src/luajit -jdump=i -Ohotloop=1 -e "
local y
local str = 'xxx'
for _ = 1, 4 do
  y = (str):byte(0X7FFFFFFF, -0X7FFFFFFF)
end

assert(y == nil)
"
LuaJIT ASSERT lj_record.c:165: rec_check_slots: slot 8 type mismatch: stack type 0 vs IR type 16

The len becomes 7, since end == 0x80000005 start == 0x7ffffffe.

The following patch fixes the issue:

diff --git a/src/lj_ffrecord.c b/src/lj_ffrecord.c
index 3c4d4eb5..ce24ba90 100644
--- a/src/lj_ffrecord.c
+++ b/src/lj_ffrecord.c
@@ -865,8 +865,8 @@ static void LJ_FASTCALL recff_string_range(jit_State *J, RecordFFData *rd)
       J->base[0] = lj_ir_kstr(J, &J2G(J)->strempty);
     }
   } else {  /* Return string.byte result(s). */
-    ptrdiff_t i, len = end - start;
-    if (len > 0) {
+    if (end > start) {
+      ptrdiff_t i, len = end - start;
       TRef trslen = emitir(IRTGI(IR_SUBOV), trend, trstart);
       emitir(IRTGI(IR_EQ), trslen, lj_ir_kint(J, (int32_t)len));
       if (J->baseslot + len > LJ_MAX_JSLOTS)

UPD: string.sub() has the same issue, but underflow for it doesn't affect the correctness.

[MikePall]

Fixed. Thanks!