Memory leak when custom allocator fails to conform to requirements #1393
Description
LuaJIT version: latest master 871db2c
OS: Linux
Arch: aarch64
When custom allocator is used and allocated pointer address is outside of the acceptable range (47bit?), LuaJIT rejects the address and call to lua_newstate returns NULL without deallocating previously allocated memory.
In particular, in scenarios when lua_newstate is used with fallback to luaL_newstate, this causes memory leak.
#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>
#include <stdio.h>
#include <stdlib.h>
static void *custom_alloc(void *ud, void *ptr, size_t osize, size_t nsize) {
if (nsize == 0) {
fprintf(stderr, "Freed address: %p\n", ptr);
free(ptr);
return NULL;
}
ptr = realloc(ptr, nsize);
fprintf(stderr, "Allocated address: %p\n", ptr);
return ptr;
}
int main(void) {
lua_State *L = lua_newstate(custom_alloc, NULL);
if (L == NULL) {
fprintf(stderr, "Failed to create Lua state with custom allocator\n");
L = luaL_newstate();
}
lua_close(L);
return 0;
}This leak is hard to detect, as tools like -fsanitize=address or valgrind changes allocator behaviour and returns addresses that are acceptable for LuaJIT.
Initially discovered in: mlua-rs/mlua#653