excessively large page table on 64-bit platforms

[vicuna]

Original bug ID: 4448
Reporter: @xavierleroy
Assigned to: @xavierleroy
Status: closed (set by @xavierleroy on 2009-02-01T08:38:06Z)
Resolution: fixed
Severity: minor
Category: ~DO NOT USE (was: OCaml general)
Monitored by: "Vladimir Shabanov" adamc toots till @Chris00 smimram @oandrieu @yakobowski @alainfrisch @mmottl

Bug description

As reported on the caml-list, some users of 64-bit platforms noticed large page tables being allocated, e.g. "Growing page table to 104640820 entries". The goal of this PR is to better understand the situations where this occurs. Please add notes to this PR describing in particular:

• Your platform: processor, OS version (esp. which Linux distro), kernel version.

• The output of the following command:
  grep USE_MMAP ocamlc -where/caml/config.h

[vicuna]

Comment author: smimram

Here are two runs of liquidsoap:

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 11770701 entries
Growing heap to 1440k bytes

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 129384191 entries
Growing heap to 1440k bytes

The processor is an "AMD Athlon(tm) 64 Processor 3500+", running Debian testing, and the kernel is 2.6.22-2-amd64. ocaml comes from the 3.10.0-8 Debian package.

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

[vicuna]

Comment author: @xavierleroy

While you're at it, and if your OS is Lunix, the output of the following command is interesting as well:

cat /proc/sys/kernel/randomize_va_space

(Thanks, Basile.)

[vicuna]

Comment author: smimram

Here it is:

$ cat /proc/sys/kernel/randomize_va_space
1

and by the way, the problem disappears if I

echo 0 > /proc/sys/kernel/randomize_va_space

[vicuna]

Comment author: adamc

Linux distro is Fedora 5.

$ uname -a
Linux [hostname] 2.6.18.8pavel_64bit #5 SMP PREEMPT Tue May 8 12:28:30 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

$ cat /proc/sys/kernel/randomize_va_space
1

[vicuna]

Comment author: @Chris00

I suspect some Linux distros that applied address randomization patches
to the stock Linux kernel.

In fact, randomization seems to be in the stock kernel: I have compiled

Linux 2.6.23.1 #1 SMP Tue Nov 13 22:17:50 CET 2007 x86_64 GNU/Linux

downloaded from kernel.org a few days ago (no patches applied) and I have

$ cat /proc/sys/kernel/randomize_va_space
1

Thanks for your work,
ChriS

[vicuna]

Comment author: @mmottl

According to Adam, who filed the initial bug report, turning off address randomization in the kernel makes the problem go away. Though this solution is good enough for us for the while being, it would certainly be helpful if we could find a more general solution. Some applications might be so sensitive that people don't want to turn off this important security feature.

I am not sure whether this might help, but it seems that Linux has at least some respect for the "start" argument to mmap, i.e. takes it as a hint where to place the mapping. By starting out with "NULL" for the initial mapping and then just storing the start of the page following the current mapping in a global variable for the next allocation looks like a very simple thing to try. It may not be a perfect solution, since the kernel is not required to return a mapping at this address, but it may be good enough and does not require a substantial rewrite of the page handling code.

[vicuna]

Comment author: Vladimir Shabanov

$ cat /proc/sys/kernel/randomize_va_space
1

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

System is Debian testing, 2.6.21-2-amd64 kernel.

native executable, works ok, output is always the same:
$ OCAMLRUNPARAM="v=12" ./_build/game.opt
Growing heap to 960k bytes
Growing page table to 72391 entries
Growing heap to 1440k bytes
Growing page table to 90522 entries

bytecode executable:
$ OCAMLRUNPARAM="v=12" ./_build/game
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 141518746 entries <-- this number vary from run to run
Growing heap to 1440k bytes

Problem disappears after
$ echo 0 > /proc/sys/kernel/randomize_va_space
page table entries count is small and no more vary from run to run.

native output with randomize_va_space=0:
Growing heap to 960k bytes
Growing page table to 89517 entries
Growing heap to 1440k bytes
Growing page table to 107540 entries

bytecode output with randomize_va_space=0:
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 71741 entries
Growing heap to 1440k bytes
Growing page table to 98490 entries

[vicuna]

Comment author: Vladimir Shabanov

Forgot the processor:
$ cat /proc/cpuinfo | grep "model name"
model name : AMD Athlon(tm) 64 Processor 3500+

[vicuna]

Comment author: @mmottl

I can confirm now that passing a hint to mmap seems to fix the problem in the OCaml-runtime, though I haven't been able to test yet whether this will generally work well. If you change the function caml_aligned_mmap in byterun/unix.c as follows:

char * caml_aligned_mmap (asize_t size, int modulo, void **block)
{
char *raw_mem;
uintnat aligned_mem;
static char *last_addr = NULL;
raw_mem = (char *) mmap(last_addr, size + Page_size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (raw_mem == MAP_FAILED) return NULL;
last_addr = raw_mem + size + 2 * Page_size;
block = raw_mem;
raw_mem += modulo; / Address to be aligned */
aligned_mem = (((uintnat) raw_mem / Page_size + 1) * Page_size);
return (char *) (aligned_mem - modulo);
}

then the problem with exploding page tables seems to go away. In one case it seemed to me that one may have to add more than just two page tables to get nearby address space, but I unfortunately don't have time now to find out when this may be necessary. This is at least a starting point for a cheap solution that may work well enough until a better implementation of page table management is available.

[vicuna]

Comment author: @xavierleroy

Markus's suggestion looks like a good temporary workaround. I applied it in the 3.10 release branch. Meanwhile, I'm experimenting with replacing the page table by a hash table, with good results so far. This is candidate for inclusion in the trunk after more testing.

[vicuna]

Comment author: @xavierleroy

In CVS trunk, reimplemented the page table as a sparse data structure (hash table). Preliminary tests are OK, both correctness-wise and performance-wise. More testing is welcome.

[vicuna]

Comment author: @xavierleroy

The new implementation of page tables in 3.11 seems to work. I'm closing this PR.