intended use of extent hooks

[bchalios]

I want to use jemalloc to manage memory within a programming model runtime system, I think in a similar fashion to what is mentioned in #825, however I have some questions regarding how are the extent hooks meant to be used.

My understanding was/is that the extent_alloc_t hook is used to provide memory for a new arena. Subsequent allocation calls from this arena will return memory from the address range that the extent_alloc_t hook will provide. Is that true?

I was experimenting with the interface and I saw that subsequent attempts to allocate memory from a newly created arena using mallocx() invokes the extent_alloc_t hook again.

The scenario I want to work with would be something is along the lines of:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <jemalloc/jemalloc.h>
#include "jemalloc_util.h"
#include <sys/mman.h>

void *pre_alloc;
void *pre_alloc_end;

static void *
my_hooks_alloc(extent_hooks_t *extent_hooks, void *new_addr, size_t size,
		size_t alignment, bool *zero, bool *commit, unsigned arena_ind)
{
	printf("In wrapper alloc_hook: new_addr:%p "
		"size:%lu(%lu pages) alignment:%lu "
		"zero:%s commit:%s arena_ind:%u\n",
		new_addr, size, size / 4096, alignment,
		(*zero) ? "true" : "false",
		(*commit) ? "true" : "false",
		arena_ind);

	void *ret = pre_alloc;
	if (ret >= pre_alloc_end) {
		printf("> Not enough memory\n");
		return NULL;
	}
	pre_alloc = (char *)pre_alloc + size;

	if (*zero)
		memset(ret, size, 0);

	printf("> hook returning range: [%p,%p)\n", ret, (char *)ret + size);
	return ret;
}

extent_hooks_t hooks = {
	my_hooks_alloc,
	NULL,
	NULL,
	NULL,
	NULL,
	NULL,
	NULL,
	NULL,
	NULL
};

int main(int argc, char *argv[])
{
	size_t ret, sz;
	unsigned arena_ind = -1;
	extent_hooks_t *new_hooks;
	size_t hooks_len, memsize = 4096 * 4096;

	pre_alloc = mmap(NULL, memsize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	if (!pre_alloc) {
		perror("Could not pre-allocate memory");
		exit(1);
	}
	pre_alloc_end = pre_alloc + memsize;

	new_hooks = &hooks;

	printf("Setting up new hooks\n");
	sz = sizeof(arena_ind);
	ret = mallctl("arenas.create", (void *)&arena_ind, &sz, (void *)&new_hooks, sizeof(extent_hooks_t *));
	if (ret) {
		dbg_print(ret, "mallctl error creating arena with new hooks");
		exit(1);
	}

	printf("created arena: %u\n", arena_ind);

	printf("----------------------------------------------\n");
	void *p = mallocx(1024, MALLOCX_ARENA(arena_ind) | MALLOCX_TCACHE_NONE);
	printf("Allocated from new: %p\n", p);
	void *q = mallocx(1024, MALLOCX_ARENA(arena_ind) | MALLOCX_TCACHE_NONE);
	printf("Allocated from new: %p\n", q);
	void *z = mallocx(10, MALLOCX_ARENA(arena_ind) | MALLOCX_TCACHE_NONE);
	printf("Allocated from new: %p\n", z);

        return 0;
}

Running this gives me:

Setting up new hooks
In wrapper alloc_hook: new_addr:(nil) size:2097152(512 pages) alignment:4096 zero:true commit:true arena_ind:16
> hook returning range: [0x7f655e14b000,0x7f655e34b000)
created arena: 16
----------------------------------------------
In wrapper alloc_hook: new_addr:(nil) size:2097152(512 pages) alignment:4096 zero:false commit:false arena_ind:16
> hook returning range: [0x7f655e34b000,0x7f655e54b000)
In wrapper alloc_hook: new_addr:(nil) size:4096(1 pages) alignment:4096 zero:false commit:true arena_ind:16
> hook returning range: [0x7f655e54b000,0x7f655e54c000)
Allocated from new: 0x7f655e54b000
Allocated from new: 0x7f655e54b400
In wrapper alloc_hook: new_addr:(nil) size:2097152(512 pages) alignment:4096 zero:false commit:false arena_ind:16
> hook returning range: [0x7f655e54c000,0x7f655e74c000)
In wrapper alloc_hook: new_addr:(nil) size:4096(1 pages) alignment:4096 zero:false commit:true arena_ind:16
> hook returning range: [0x7f655e74c000,0x7f655e74d000)
Allocated from new: 0x7f655e74c000

The allocation hook is invoked when the arena is getting created (twice).
The first two allocations use the initial allocation of my allocation hook. The next allocation invokes the allocation hook again.

My question hence is: when exactly is each allocation hook invoked? I 've looked both in the documentation and the github issues, but was not able to find something, however, if I missed it please direct me to it.

Thanks in advance.

[interwq]

The alloc hook is invoked whenever the arena needs additional memory from the OS, e.g. when all local mapped memory are in use, and we need more memory to satisfy the current request (which is typical done through mmap).

The hooks are also used when an arena need memory for internal metadata, e.g. In wrapper alloc_hook: new_addr:(nil) size:2097152(512 pages) should be for the internal radix tree usage. These generally only happens during initialization / warmup. Note that if there is memory can be reused, alloc hooks may not be invoked.

[interwq]

Closing for now. Please feel free to let us know if any questions.

[bchalios]

Thanks a lot for your answer. I think it makes more sense now. Additionally when is the rest of the hooks are called? Is there somewhere documentation about that?

I 'm particularly interested in what happens when the dalloc hook is called. All the tests I 've done so far, have not seen it being called for my newly created arena.

[davidtgoldblatt]

Depending on the configuration (i.e. opt.retain is true), dalloc might never get called; in that case we'll save address space for later rather than explicitly deallocate it.

[mcguigan]

Following up with the original question, is there any documentation about when the rest of the hooks are called? I'm also interested in this, and can't find any documentation (so far).

[interwq]

The best place for documentation is http://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks

Regarding the exact locations where the hooks are called, it would be tricky to document since: 1) it depends on options used (e.g. purge vs dalloc); and 2) internally jemalloc may utilize the hooks for metadata (e.g. may call alloc when a new thread comes in). It's also related to internal design, such as splitting a large extent vs allocating a new one, when both can satisfy an allocation request.