Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
GC does not reclaim bigarrays fast enough causing Out_of_memory #7750
Original bug ID: 7750
In a long-running application we make heavy use of Bigarrays (easily pushing 250MiB/s per application). With 4 of these applications running on a server with 128GiB of RAM, the application stalled with Out_of_memory exceptions. Apparently each of them was eating 30GiB of memory with mostly stale Bigarrays.
After triggering compaction on one of the applications through RPC, memory use dropped to multiple MiBs (see additional info). Memory use as observed by the OS also fell steeply as all the stale bigarrays were disposed off. (This also proved that it wasn't a memory leak in other parts of our code).
In the output one can see that the ratio of minor/major collections is heavily skewed in favor of minor collections (36000/137). If many bigarrays live on the major heap, this means they will linger for a long time before getting cleaned up. Also, because the (default) space_overhead of 80% wasn't hit (1,3GiB/1,8GiB) major collection also wasn't sped up.
I created a small test-case that I believe sufficiently represents the issue at hand. The provided test-application easily takes 11GiB of memory, where only about 4-6 (?) would be expected. Settings GC parameters space_overhead or max_overhead does not help. I believe that it might help our normal application a bit since a more normal memory allocation pattern is expected there, but that remains to be tested.
The test application essentially does the following:
As for the trigger of the bad behavior, I believe the fixing of #7158 caused many fewer major garbage collections in our case (we also make use of Lwt_preemptive) - this issue started occuring after we upgraded the compiler from 4.03.0 to 4.05.0.
Secondly, I believe that the last question posed in #7180 may be related here:
In the test case, one can see that immediately after creating the bigarrays 'p' is capped to 0.3 and the remaining slices are rather small. In essence, it means that the garbage collector forgets about the excess and slows down significantly.
One potential fix would be to leave that excess in extra_heap_resources for the next iteration. Another improvement is probably to not only look at the ratio of free/total, but also have either a time-based or absolute-value based trigger (e.g. whenever promoted_words crosses a configurable boundary)
Steps to reproduce
ocamlbuild -lbigarray test.native && ./test.native
BEWARE: easily eats around 11-12GiB of memory
let minor_heap_size = Gc.((get ()).minor_heap_size)
let () = Printf.printf "Minor heap size = %dB\n%!" (minor_heap_size*8)
let display_stat state =
(* Create a bigarray and fill it to ensure memory is actually faulted in )
(* Flush the minor heap 'count' times
(* Eat about 2MiB minor * 1000 = 2G of heap space *)
let () =
(* Repeatedly create 'inner_count' bigarrays,
let () = loop_outer 100   
Comment author: @stijn-devriendt
An update is due here. After further investigation it appears we found an issue with our code, where we passed a pre-allocated buffer into caml_ba_alloc.
Unfortunately, when doing this, the size variable (https://github.com/ocaml/ocaml/blob/trunk/byterun/bigarray.c#L93) remains zero and the call to caml_alloc_custom receives that zero as extra size. Because of this, the GC is not sped up.
I think all that remains is to assess whether it's expected that the provided example eats a significant amount of memory and whether limiting the major GC to p=0.3 doesn't risk throttling it too much. We did find that caml_alloc_custom calls caml_check_urgent_gc, which might reduce the risk of letting memory allocation get out of control.