Stack overflow cannot be properly reported by xpcall

[nlebedenco]

The stack overflow check in lj_state_growstack does not account for a potential error handler invoked by xpcall. Changing it to if (L->stacksize >= LJ_STACK_MAXEX) allows the error handler to be called without causing a second error. The way it is now when the error handler requests a stack slot for the error object it bumps into the check at the beginning of lj_state_growstack and raises a LUA_ERRERR.

Current behaviour:

$> luajit.exe -e "f = function() f() end; f()"
error in error handling

Expected behaviour:

$> luajit.exe -e "f = function() f() end; f()"
luajit.exe: (command line):1: stack overflow
stack traceback:
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        ...
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in function 'f'
        (command line):1: in main chunk
        [C]: at 0x7ff6418a5630

[MikePall]

Fixed. Thanks!

[corsix]

This change (8135de2) has the unfortunate side effect of causing stack overflows to be reported long before they need to be; suppose L->stacksize is 33000 and lj_state_growstack is called with need=30, then lj_state_growstack will choose to double the stack to LJ_STACK_MAX, immediately causing L->stacksize == LJ_STACK_MAXEX and thus lj_err_msg(L, LJ_ERR_STKOV). This despite the fact that there is ample space to grow the stack by 30 slots.

[MikePall]

I rewrote the stack overflow handling to resolve your concerns.