Repository navigation
Reset the major GC's pacing clock after any synchronous GC work. - #13736
Conversation
51d34ae to
69d1f5c
Compare
|
This change seems reasonable in the circumstances described. I've tested the changes against the sample program in #13123 (comment) which seems to improve the memory usage on that small program. I'll also setup a longer running liquidsoap program to see how that performs. |
|
@nojb pointed me to this PR after I observed problematic behavior in our application when using Gc.compact in OCaml 5.3: forcing compaction caused the process to consume significantly more memory instead of reducing it. I believe this constitutes a bug rather than merely a performance optimization, as the current label suggests. Therefore, I would support backporting this fix to the 5.3 maintenance branch. |
|
cc @Octachron OK to cherry-pick to 5.3? |
|
I don't see any reason to not backport in the maintenance branch (we don't need to ask our release manager every time we push something there), and the patch is reasonably simple so this should be easy/doable. (I'm not sure it will fix all your worries with the compactor, though.) Of course we are not talking about making a new 5.3 point release --- no such thing is planned for now. |
|
I don't think that I agree, backporting fix which changes GC behavior to branches that are not actively maintained seems to be potentially more harmful than useful. Since the fix is available in 5.4.0 which contains more runtime fixes, I am unconvinced that this specific fix in needed in the maintenance branch of 5.3 . |
I don't see how it could be harmful. The current behaviour in 5.3.0 is obviously bad, increasing the heap size of OCaml programs that call |
|
Cherry-picking the change increases the number of version of the runtime in existence, without testing that the changes Typically, I am not sure that I would recommend without hesitation to Debian or any downstream packager to pick up this fix for a hypothetical packaged OCaml 5.3 rather than upgrading to 5.4. |
|
My intuition is that this part of the GC has not changed much between 5.3 and 5.4, the patch is just as clearly an improvement for 5.3 than it was for 5.4. I'm happy to require a bit more testing to feel extra safe (but for me this is not a release-management question). @nojb, could you maybe try to backport locally, and confirm that this actually improves things for your Lexifi use-cases? |
|
I am not following you. The maintenance branch contains the set of new patches that ought to be part of any new release of this version of the compiler, even if no such release is planned on our side. This is a maintenance and release question. |
|
For me, the release manager role is to make decisions on what goes or does not go into a release during the freeze period that comes before a planned release. (The point is to avoid last-minute changes that endanger quality because we pressure ourselves into putting more stuff in the release.) |
|
Maintenance branches are still in their critical bug fix-only period, they are not unfrozen, and should be subject to the highest level of scrutiny because otherwise it would endanger the expectation that patch-level releases are always safe for downstream users. Your proposal changes the semantics of commits in the maintenance branch depending on the hidden signal "Was a future release planned when this commit was merged?" which is unreadable outside of active contributors. |
|
Note that on the subject of this PR, if both of you judge that 5.3 is not really usable without this fix because of the surprising GC behavior and you would both rather debug unlikely regression of this PR than GC issues without it, this is better criterion for me to backport this patch. |
Here are some numbers from @alainfrisch's investigation of this issue on our codebase, with 5.3.0. The following is a trace obtained by registering a GC alarm and printing the heap size (obtained with Stock 5.3.0: after every compaction the next full major GC is only done after a multiple of the last heap size before the last compaction, which can be a large multiple of the heap size after the last compaction. With the patch, the behaviour is more reasonable. |
|
I feel sorry for triggering this discussion... To clarify : we (@nojb, me, and LexiFi) don't really care about having the patch in the maintenance branch for 5.3. We'll likely cherry-pick anyway on our side. I understand that despite the problem being well understood and the fix easy, it's never 100% safe to assume that the fix is necessarily good for 5.3, and that actual testing is thus needed. For us, the situation seems clearly better with the fix than without, and upgrading to 5.4 would require much more testing. |
|
Sorry for my bickering with @gasche. At the same time, if LexiFi ends up cherry-picking the patch, this solves my concern of having an untested version of the runtime in maintenance branch, and it seems better to share the patched 5.3 . |
## Summary Backport [ocaml#13736](ocaml#13736) to `5.3.0-semgrep`. Synchronous major collections advance the major GC work counter without corresponding allocation. After `Gc.major`, `Gc.full_major`, or `Gc.compact`, the work counter can therefore remain far ahead of the allocation counter, suppressing subsequent automatic major GC slices until allocation catches up. On a large heap this can cause rapid heap growth or OOM after an explicit collection. The upstream fix resets the allocation and work counters to the same logical clock after synchronous major GC work. This is relevant to Semgrep because its memory-limit handler calls `Gc.compact` before reraising an OOM or memory-limit exception. ## Backport Notes The three runtime changes apply to OCaml 5.3 without adaptation and match upstream ocaml#13736: - Add `caml_reset_major_pacing`. - Invoke it after explicit major collection work, including each `Gc.full_major` and `Gc.compact` cycle. - Omit the conflicting upstream `Changes` entry, since this is a Semgrep-fork backport. No compiler or Semgrep-specific code is changed. ## Validation - Clean Flambda bootstrap of the exact branch: `./configure --enable-flambda && make -j8` - Confirmed the resulting compiler reports `flambda: true`. - Compiled an identical post-compaction pacing workload with unpatched and patched 5.3 runtimes. The workload compacts a 320 MiB live heap, drops it, and then performs short-lived major allocations: - Unpatched: reached the 768 MiB guard after 17,900 iterations; heap reached 801 MiB with only 2 automatic major cycles. - Patched: completed 100,000 iterations; final heap returned to 0 MiB with 10,169 automatic major cycles.
…p/semgrep-proprietary#6784) Pulls in semgrep/ocaml#26 , a backport of ocaml/ocaml#13736 . We hypothesize that this fix will improve GC pacing and OOMs owing to missing major collection cycles, which @ajbt200128 and @xavilien have observed from traces. We expect the following sequence of events will not happen going forward: 1. we trip max_memory or raise OOM exn 2. we trigger a synchronous compaction, work_counter runs way past alloc_counter 3. Major cycles and incremental compactions stop happening until someone manually calls Gc.compact Test plan: Observe, if our hypothesis is correct, reduced tail heap sizes. I'd also like to see benchmark numbers just to make sure this pacing change doesn't totally blow our perf out of the water somehow. synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
…p/semgrep-proprietary#6784) Pulls in semgrep/ocaml#26 , a backport of ocaml/ocaml#13736 . We hypothesize that this fix will improve GC pacing and OOMs owing to missing major collection cycles, which @ajbt200128 and @xavilien have observed from traces. We expect the following sequence of events will not happen going forward: 1. we trip max_memory or raise OOM exn 2. we trigger a synchronous compaction, work_counter runs way past alloc_counter 3. Major cycles and incremental compactions stop happening until someone manually calls Gc.compact Test plan: Observe, if our hypothesis is correct, reduced tail heap sizes. I'd also like to see benchmark numbers just to make sure this pacing change doesn't totally blow our perf out of the water somehow. synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
…p/semgrep-proprietary#6784) Pulls in semgrep/ocaml#26 , a backport of ocaml/ocaml#13736 . We hypothesize that this fix will improve GC pacing and OOMs owing to missing major collection cycles, which @ajbt200128 and @xavilien have observed from traces. We expect the following sequence of events will not happen going forward: 1. we trip max_memory or raise OOM exn 2. we trigger a synchronous compaction, work_counter runs way past alloc_counter 3. Major cycles and incremental compactions stop happening until someone manually calls Gc.compact Test plan: Observe, if our hypothesis is correct, reduced tail heap sizes. I'd also like to see benchmark numbers just to make sure this pacing change doesn't totally blow our perf out of the water somehow. synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
The major GC has
alloc_counterandwork_counterclocks to decide whether and how much work to do in the usual slices. In normal operation thework_countertracks thealloc_counter. If a large amount of major GC work is done synchronously (e.g. by callingGc.full_major()orGc.compact()then thework_counteris advanced a lot without any allocation, so it gets well ahead of thealloc_counter. This has the effect of turning off major GC until such time as allocation "catches up". If the heap is quite large, it's easy to get to OOM before that happens.This PR fixes this by adding a function to reset the major GC's pacing clock, and calling it after any synchronously-invoked major GC work.