can I dlsym malloc function?

[CaffeMrDe]

platform
window10
mingw 8.1.0 g++/gcc

static void initMemMonitor()
{
    malloc_f  = (malloc_t)dlsym(RTLD_NEXT, "malloc");
    if (!malloc_f) {
        fprintf(stderr, "unable to get malloc symbol!\n");
        exit(1);
    }
    fprintf(stderr, "malloc monitor : successfully wrapped!\n");
}

static thread_local bool enterDumpFunc = false; // 防止栈溢出标志位

void *malloc(size_t size)
{
    if(!malloc_f || !malloc_default)
    {
        initMemMonitor();
    }

    void *ret = nullptr;
    if(!enterDumpFunc)
    {
        enterDumpFunc = true;
        ret = malloc_f(size);
    }else
        ret = malloc_default(size);
    return ret;
}

result: stack frame recursion

1 malloc            test7.cpp 39  0x402808 
2 dlsym             dlfcn.c   557 0x401fc1 
3 initMemMonitor    test7.cpp 26  0x402784 
4 malloc            test7.cpp 41  0x402825 
5 dlsym             dlfcn.c   557 0x401fc1 
6 initMemMonitor    test7.cpp 26  0x402784 
7 malloc            test7.cpp 41  0x402825 
8 __tmainCRTStartup               0x40132e 
9 WinMainCRTStartup               0x4014cb 

How can I solve it? Thanks

[traversaro]

Hello @CaffeMrDe, I am not sure how to address your problem. To debug further, did you check if this works correctly if you use directly Win32 APIs like GetProcAddress without passing via dlfcn-win32 ?

Anyhow, I marked the issue as help wanted so if anyone from the community knows how to solve this problem or has any pinput, they are welcome to comment!

[pali]

@CaffeMrDe Well, if I understand correctly, you are trying to wrap system malloc() function (more precisely malloc() from msvcrt.dl) via dlsym+RTLD_NEXT like on linux/ELF systems. The problem is that dlsym() implementation in dlfcn-win32 project internally uses malloc() function, so you are not able to resolve malloc symbol via dlsym+RTLD_NEXT. I do not think that on Windows it is possible to write RTLD_NEXT support without dynamic allocation.

@traversaro GetProcAddress() does not support RTLD_NEXT flag. And I'm not aware of any Win32 API function which could provide RTLD_NEXT-like feature. This is really unique for dlfcn-win32 and in past I was in doubt if something like this is even possible to implement on Windows (but it is - we have it).

[traversaro]

Thanks @pali !

[pali]

@CaffeMrDe As a simple "hack", you can replace malloc() in dlfcn-win32 project by some other dynamic allocator. For example by LocalAlloc() from WinAPI/kernel32.dll (and free() by LocalFree()). Then you should be able to wrap msvcrt.dll's malloc(), but you cannot wrap kernel32.dll's LocalAlloc().

[CaffeMrDe]

Okay, I see what you mean! Thank you for your help!