Skip to content

Reset the major GC's pacing clock after any synchronous GC work. - #13736

Merged
damiendoligez merged 1 commit into
ocaml:trunkfrom
NickBarnes:nick-reset-major-pacing
Feb 4, 2025
Merged

damiendoligez merged 1 commit into
ocaml:trunkfrom
NickBarnes:nick-reset-major-pacing

Conversation

@NickBarnes

Copy link
Copy Markdown
Contributor

The major GC has alloc_counter and work_counter clocks to decide whether and how much work to do in the usual slices. In normal operation the work_counter tracks the alloc_counter. If a large amount of major GC work is done synchronously (e.g. by calling Gc.full_major() or Gc.compact() then the work_counter is advanced a lot without any allocation, so it gets well ahead of the alloc_counter. This has the effect of turning off major GC until such time as allocation "catches up". If the heap is quite large, it's easy to get to OOM before that happens.
This PR fixes this by adding a function to reset the major GC's pacing clock, and calling it after any synchronously-invoked major GC work.

@NickBarnes
NickBarnes force-pushed the nick-reset-major-pacing branch from 51d34ae to 69d1f5c Compare January 16, 2025 15:08
@kayceesrk kayceesrk added the Performance PR or issues affecting runtime performance of the compiled programs label Jan 18, 2025
@damiendoligez damiendoligez self-assigned this Jan 22, 2025
@tmcgilchrist

Copy link
Copy Markdown
Contributor

This change seems reasonable in the circumstances described. I've tested the changes against the sample program in #13123 (comment) which seems to improve the memory usage on that small program. I'll also setup a longer running liquidsoap program to see how that performs.

@damiendoligez damiendoligez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alainfrisch

Copy link
Copy Markdown
Contributor

@nojb pointed me to this PR after I observed problematic behavior in our application when using Gc.compact in OCaml 5.3: forcing compaction caused the process to consume significantly more memory instead of reducing it. I believe this constitutes a bug rather than merely a performance optimization, as the current label suggests. Therefore, I would support backporting this fix to the 5.3 maintenance branch.

@nojb

nojb commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

cc @Octachron OK to cherry-pick to 5.3?

@gasche

gasche commented Nov 19, 2025

Copy link
Copy Markdown
Member

I don't see any reason to not backport in the maintenance branch (we don't need to ask our release manager every time we push something there), and the patch is reasonably simple so this should be easy/doable. (I'm not sure it will fix all your worries with the compactor, though.) Of course we are not talking about making a new 5.3 point release --- no such thing is planned for now.

@Octachron

Copy link
Copy Markdown
Member

I don't think that I agree, backporting fix which changes GC behavior to branches that are not actively maintained seems to be potentially more harmful than useful.

Since the fix is available in 5.4.0 which contains more runtime fixes, I am unconvinced that this specific fix in needed in the maintenance branch of 5.3 .

@nojb

nojb commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

I don't think that I agree, backporting fix which changes GC behavior to branches that are not actively maintained seems to be potentially more harmful than useful.

I don't see how it could be harmful. The current behaviour in 5.3.0 is obviously bad, increasing the heap size of OCaml programs that call Gc.compact and Gc.full_major. Upgrading to 5.4.0 is not trivial for large/industrial codebases...

@Octachron

Copy link
Copy Markdown
Member

Cherry-picking the change increases the number of version of the runtime in existence, without testing that the changes
do not create unexpected regressions in the GC, which tends to exhibit coupled behaviour.

Typically, I am not sure that I would recommend without hesitation to Debian or any downstream packager to pick up this fix for a hypothetical packaged OCaml 5.3 rather than upgrading to 5.4.

@gasche

gasche commented Nov 19, 2025

Copy link
Copy Markdown
Member

My intuition is that this part of the GC has not changed much between 5.3 and 5.4, the patch is just as clearly an improvement for 5.3 than it was for 5.4. I'm happy to require a bit more testing to feel extra safe (but for me this is not a release-management question). @nojb, could you maybe try to backport locally, and confirm that this actually improves things for your Lexifi use-cases?

@Octachron

Copy link
Copy Markdown
Member

I am not following you. The maintenance branch contains the set of new patches that ought to be part of any new release of this version of the compiler, even if no such release is planned on our side. This is a maintenance and release question.

@gasche

gasche commented Nov 20, 2025 •

Copy link
Copy Markdown
Member

For me, the release manager role is to make decisions on what goes or does not go into a release during the freeze period that comes before a planned release. (The point is to avoid last-minute changes that endanger quality because we pressure ourselves into putting more stuff in the release.)
(following this view) If the maintenance branch of an old version has no planned release in the future, our usual development processes suffice to decide what should or shouldn't go in, just like we don't ask the release-manager before merging things in the trunk.

@Octachron

Copy link
Copy Markdown
Member

Maintenance branches are still in their critical bug fix-only period, they are not unfrozen, and should be subject to the highest level of scrutiny because otherwise it would endanger the expectation that patch-level releases are always safe for downstream users.

Your proposal changes the semantics of commits in the maintenance branch depending on the hidden signal "Was a future release planned when this commit was merged?" which is unreadable outside of active contributors.

@Octachron

Copy link
Copy Markdown
Member

Note that on the subject of this PR, if both of you judge that 5.3 is not really usable without this fix because of the surprising GC behavior and you would both rather debug unlikely regression of this PR than GC issues without it, this is better criterion for me to backport this patch.

@nojb

nojb commented Nov 20, 2025

Copy link
Copy Markdown
Contributor

@nojb, could you maybe try to backport locally, and confirm that this actually improves things for your Lexifi use-cases?

Here are some numbers from @alainfrisch's investigation of this issue on our codebase, with 5.3.0. The following is a trace obtained by registering a GC alarm and printing the heap size (obtained with Gc.quick_stat) after every major GC and after compaction (the compaction is triggered every time the heap has grown more than X megabytes since the last major GC).

Stock 5.3.0: after every compaction the next full major GC is only done after a multiple of the last heap size before the last compaction, which can be a large multiple of the heap size after the last compaction.

2025-11-19 09:21:50.181 ~??????? [gc] Heap size : 21 Mb
2025-11-19 09:21:50.625 ~??????? [gc] Heap size : 29 Mb
2025-11-19 09:21:50.720 ~??????? [gc] Heap size : 88 Mb
2025-11-19 09:21:51.148 ~??????? [gc] Heap size : 92 Mb
2025-11-19 09:21:52.363 ~36a01b6 [gc] Heap size : 142 Mb
2025-11-19 09:21:52.505 ~36a01b6 [gc] After compaction, heap size : 104 Mb
2025-11-19 09:21:55.562 ~36a01b6 [gc] Heap size : 373 Mb
2025-11-19 09:21:55.923 ~36a01b6 [gc] After compaction, heap size : 285 Mb
2025-11-19 09:22:01.319 ~36a01b6 [gc] Heap size : 981 Mb
2025-11-19 09:22:02.126 ~36a01b6 [gc] After compaction, heap size : 434 Mb
2025-11-19 09:22:09.168 ~36a01b6 [gc] Heap size : 1471 Mb
2025-11-19 09:22:10.091 ~36a01b6 [gc] After compaction, heap size : 484 Mb
2025-11-19 09:22:14.943 ~36a01b6 [gc] Heap size : 1781 Mb
2025-11-19 09:22:15.818 ~36a01b6 [gc] After compaction, heap size : 514 Mb
2025-11-19 09:22:19.797 ~36a01b6 [gc] Heap size : 1901 Mb
2025-11-19 09:22:20.704 ~36a01b6 [gc] After compaction, heap size : 530 Mb
2025-11-19 09:22:24.341 ~36a01b6 [gc] Heap size : 1962 Mb
2025-11-19 09:22:25.308 ~36a01b6 [gc] After compaction, heap size : 554 Mb
2025-11-19 09:22:29.089 ~36a01b6 [gc] Heap size : 2026 Mb
2025-11-19 09:22:30.099 ~36a01b6 [gc] After compaction, heap size : 585 Mb
2025-11-19 09:22:33.687 ~36a01b6 [gc] Heap size : 2118 Mb
2025-11-19 09:22:34.691 ~36a01b6 [gc] After compaction, heap size : 598 Mb

With the patch, the behaviour is more reasonable.

2025-11-19 10:03:23.841 ~??????? [gc] Heap size : 21 Mb
2025-11-19 10:03:27.208 ~??????? [gc] Heap size : 29 Mb
2025-11-19 10:03:27.412 ~??????? [gc] Heap size : 88 Mb
2025-11-19 10:03:28.209 ~??????? [gc] Heap size : 92 Mb
2025-11-19 10:03:33.977 ~a2451bf [gc] Heap size : 142 Mb
2025-11-19 10:03:34.236 ~a2451bf [gc] After compaction, heap size : 104 Mb
2025-11-19 10:03:36.438 ~a2451bf [gc] Heap size : 199 Mb
2025-11-19 10:03:40.714 ~a2451bf [gc] Heap size : 305 Mb
2025-11-19 10:03:41.330 ~a2451bf [gc] After compaction, heap size : 267 Mb
2025-11-19 10:03:46.989 ~a2451bf [gc] Heap size : 509 Mb
2025-11-19 10:03:47.701 ~a2451bf [gc] After compaction, heap size : 393 Mb
2025-11-19 10:03:52.606 ~a2451bf [gc] Heap size : 728 Mb
2025-11-19 10:03:53.538 ~a2451bf [gc] After compaction, heap size : 416 Mb
2025-11-19 10:03:59.168 ~a2451bf [gc] Heap size : 775 Mb
2025-11-19 10:04:00.125 ~a2451bf [gc] After compaction, heap size : 427 Mb
2025-11-19 10:04:06.318 ~a2451bf [gc] Heap size : 797 Mb
2025-11-19 10:04:07.276 ~a2451bf [gc] After compaction, heap size : 436 Mb
2025-11-19 10:04:12.159 ~a2451bf [gc] Heap size : 815 Mb
2025-11-19 10:04:13.171 ~a2451bf [gc] After compaction, heap size : 444 Mb
2025-11-19 10:04:17.320 ~a2451bf [gc] Heap size : 839 Mb
2025-11-19 10:04:18.568 ~a2451bf [gc] After compaction, heap size : 453 Mb
2025-11-19 10:04:21.927 ~a2451bf [gc] Heap size : 845 Mb
2025-11-19 10:04:22.882 ~a2451bf [gc] After compaction, heap size : 462 Mb
2025-11-19 10:04:25.800 ~a2451bf [gc] Heap size : 872 Mb
2025-11-19 10:04:27.090 ~a2451bf [gc] After compaction, heap size : 469 Mb
2025-11-19 10:04:29.411 ~a2451bf [gc] Heap size : 881 Mb
2025-11-19 10:04:30.305 ~a2451bf [gc] After compaction, heap size : 478 Mb
2025-11-19 10:04:32.405 ~a2451bf [gc] Heap size : 895 Mb
2025-11-19 10:04:33.331 ~a2451bf [gc] After compaction, heap size : 485 Mb
2025-11-19 10:04:35.269 ~a2451bf [gc] Heap size : 905 Mb
2025-11-19 10:04:36.277 ~a2451bf [gc] After compaction, heap size : 486 Mb
2025-11-19 10:04:38.255 ~a2451bf [gc] Heap size : 910 Mb
2025-11-19 10:04:39.172 ~a2451bf [gc] After compaction, heap size : 493 Mb
2025-11-19 10:04:41.218 ~a2451bf [gc] Heap size : 924 Mb
2025-11-19 10:04:42.181 ~a2451bf [gc] After compaction, heap size : 497 Mb
2025-11-19 10:04:44.185 ~a2451bf [gc] Heap size : 948 Mb
2025-11-19 10:04:45.362 ~a2451bf [gc] After compaction, heap size : 503 Mb
2025-11-19 10:04:47.289 ~a2451bf [gc] Heap size : 939 Mb
2025-11-19 10:04:48.180 ~a2451bf [gc] After compaction, heap size : 519 Mb
2025-11-19 10:04:50.234 ~a2451bf [gc] Heap size : 993 Mb
2025-11-19 10:04:51.449 ~a2451bf [gc] After compaction, heap size : 525 Mb
2025-11-19 10:04:53.501 ~a2451bf [gc] Heap size : 982 Mb
2025-11-19 10:04:54.440 ~a2451bf [gc] After compaction, heap size : 526 Mb
2025-11-19 10:04:56.458 ~a2451bf [gc] Heap size : 987 Mb
2025-11-19 10:04:57.481 ~a2451bf [gc] After compaction, heap size : 534 Mb
2025-11-19 10:04:59.560 ~a2451bf [gc] Heap size : 1003 Mb
2025-11-19 10:05:00.810 ~a2451bf [gc] After compaction, heap size : 541 Mb
2025-11-19 10:05:02.816 ~a2451bf [gc] Heap size : 1024 Mb
2025-11-19 10:05:04.102 ~a2451bf [gc] After compaction, heap size : 550 Mb
2025-11-19 10:05:06.344 ~a2451bf [gc] Heap size : 1033 Mb
2025-11-19 10:05:07.638 ~a2451bf [gc] After compaction, heap size : 559 Mb
2025-11-19 10:05:10.357 ~a2451bf [gc] Heap size : 1058 Mb
2025-11-19 10:05:11.785 ~a2451bf [gc] After compaction, heap size : 570 Mb

@alainfrisch

alainfrisch commented Nov 20, 2025 •

Copy link
Copy Markdown
Contributor

I feel sorry for triggering this discussion...

To clarify : we (@nojb, me, and LexiFi) don't really care about having the patch in the maintenance branch for 5.3. We'll likely cherry-pick anyway on our side.

I understand that despite the problem being well understood and the fix easy, it's never 100% safe to assume that the fix is necessarily good for 5.3, and that actual testing is thus needed. For us, the situation seems clearly better with the fix than without, and upgrading to 5.4 would require much more testing.

@Octachron

Copy link
Copy Markdown
Member

Sorry for my bickering with @gasche.

At the same time, if LexiFi ends up cherry-picking the patch, this solves my concern of having an untested version of the runtime in maintenance branch, and it seems better to share the patched 5.3 .

dijkstracula added a commit to semgrep/ocaml that referenced this pull request Jul 17, 2026
## Summary

Backport [ocaml#13736](ocaml#13736) to `5.3.0-semgrep`.

Synchronous major collections advance the major GC work counter without corresponding allocation. After `Gc.major`, `Gc.full_major`, or `Gc.compact`, the work counter can therefore remain far ahead of the allocation counter, suppressing subsequent automatic major GC slices until allocation catches up. On a large heap this can cause rapid heap growth or OOM after an explicit collection.

The upstream fix resets the allocation and work counters to the same logical clock after synchronous major GC work. This is relevant to Semgrep because its memory-limit handler calls `Gc.compact` before reraising an OOM or memory-limit exception.

## Backport Notes

The three runtime changes apply to OCaml 5.3 without adaptation and match upstream ocaml#13736:

- Add `caml_reset_major_pacing`.
- Invoke it after explicit major collection work, including each `Gc.full_major` and `Gc.compact` cycle.
- Omit the conflicting upstream `Changes` entry, since this is a Semgrep-fork backport.

No compiler or Semgrep-specific code is changed.

## Validation

- Clean Flambda bootstrap of the exact branch:
  `./configure --enable-flambda && make -j8`
- Confirmed the resulting compiler reports `flambda: true`.
- Compiled an identical post-compaction pacing workload with unpatched and patched 5.3 runtimes. The workload compacts a 320 MiB live heap, drops it, and then performs short-lived major allocations:
  - Unpatched: reached the 768 MiB guard after 17,900 iterations; heap reached 801 MiB with only 2 automatic major cycles.
  - Patched: completed 100,000 iterations; final heap returned to 0 MiB with 10,169 automatic major cycles.
semgrep-ci Bot pushed a commit to semgrep/semgrep that referenced this pull request Jul 22, 2026
…p/semgrep-proprietary#6784)

Pulls in semgrep/ocaml#26 , a backport of
ocaml/ocaml#13736 . We hypothesize that this fix
will improve GC pacing and OOMs owing to missing major collection
cycles, which @ajbt200128 and @xavilien have observed from traces. We
expect the following sequence of events will not happen going forward:

1. we trip max_memory or raise OOM exn
2. we trigger a synchronous compaction, work_counter runs way past
alloc_counter
3. Major cycles and incremental compactions stop happening until someone
manually calls Gc.compact

Test plan: Observe, if our hypothesis is correct, reduced tail heap
sizes. I'd also like to see benchmark numbers just to make sure this
pacing change doesn't totally blow our perf out of the water somehow.

synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
semgrep-ci Bot pushed a commit to semgrep/semgrep that referenced this pull request Jul 28, 2026
…p/semgrep-proprietary#6784)

Pulls in semgrep/ocaml#26 , a backport of
ocaml/ocaml#13736 . We hypothesize that this fix
will improve GC pacing and OOMs owing to missing major collection
cycles, which @ajbt200128 and @xavilien have observed from traces. We
expect the following sequence of events will not happen going forward:

1. we trip max_memory or raise OOM exn
2. we trigger a synchronous compaction, work_counter runs way past
alloc_counter
3. Major cycles and incremental compactions stop happening until someone
manually calls Gc.compact

Test plan: Observe, if our hypothesis is correct, reduced tail heap
sizes. I'd also like to see benchmark numbers just to make sure this
pacing change doesn't totally blow our perf out of the water somehow.

synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
ajbt200128 pushed a commit to semgrep/semgrep that referenced this pull request Jul 29, 2026
…p/semgrep-proprietary#6784)

Pulls in semgrep/ocaml#26 , a backport of
ocaml/ocaml#13736 . We hypothesize that this fix
will improve GC pacing and OOMs owing to missing major collection
cycles, which @ajbt200128 and @xavilien have observed from traces. We
expect the following sequence of events will not happen going forward:

1. we trip max_memory or raise OOM exn
2. we trigger a synchronous compaction, work_counter runs way past
alloc_counter
3. Major cycles and incremental compactions stop happening until someone
manually calls Gc.compact

Test plan: Observe, if our hypothesis is correct, reduced tail heap
sizes. I'd also like to see benchmark numbers just to make sure this
pacing change doesn't totally blow our perf out of the water somehow.

synced from Pro 58f2ce8c9312c1a59f1323803ed1ebe185362604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Performance PR or issues affecting runtime performance of the compiled programs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants