-
Notifications
You must be signed in to change notification settings - Fork 717
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CRIUSupport: GC Heap re-sizing on restore #18217
Comments
Personally, I think this should all be opt-in. If we make the dynamic adjustment by default we are essentially overriding a specified user request. If someone says they want -Xmx12g at checkpoint and they restore on a 4g node, there are clearly potential downsides to this. But what if that's what they want, how can they express a behaviour they have already expressed? The documentation already clearly states that this is the outcome if one sets Xmx in this manner. We could make an option called |
I am with Tobi on this one. It would be hard to explain to customer why JVM throw OOM if -Xmx has been specified explicitly, but JVM decided to use less heap memory by some "internal" reason. Creation of new non-default behaviour sounds reasonable for me. |
Should row 1 ("no options") say something like Implicit softmx 6g for the last column ("16g restore") ? I don't think we can support a softmx value greater than the mx value chosen at startup, correct ? |
I understand what is being said here, but just wanted to mention that the CRIU support feature has not technically GAed on its own at OpenJ9 (we are supporting Open Liberty's GA with the function as we have implemented for them). So, this should give us some freedom to change behavior if we want to. |
Is this question about rows 3 and 4 in the table ? Just making sure because those are the ones that talks about "dynamic adjustment". I generally agree with Dmitri and you that we should probably not change the default behavior by applying a dynamic adjustment factor if a user explicitly specified an i.e. Dynamic adjustment can be applied if That is my thinking now, but happy to hear thoughts. |
For your other two questions:
|
The default heuristics is to attempt to set it to 75%, hence 12g. We have a choice here to silently set it to 6g or output a message saying something like "ideally you'd get 12g but we cant, so here is 6g". |
Yes. [4] and [5] refers to the section below the table. In hindsight I should have used letters :) |
For scenario Restore 16g "Implicit softmx12g": "(error message [8])" does not make sense. I think returning the error for |
Note that while making these decisions that we also want to consider how we react on the other important h/w env change (from snapshot to restore), that is the CPU (HW thread count) that affects GC threads The current behaviour is rather dynamic - the restore side will adapt to the current environment, even if h/w thread count is larger than the original snapshot GC thread count (what was rather large work to accommodate for). This is true if either GC thread count was left default or -Xgcthreads was explicitly set on snapshot side. [But while we adapt, I'll again re-iterate that we make some compromises (the larger the change factor in h/w thread count is - the more visible effects of the compromises may be), hence it's optimal to version snapshots for various restore deployments.] We may want to consider aligning the behavior. For example to obey snapshot's Xgcthreads, what we seem to plan to do with Xmx? |
I'm ok with either way, so seems like we converging to the original proposal (case 2 is default and 3/4 are opt-in) If we introduce a new option (that would be specified on snapshot side), I think it should have 'restore' in its name, for example -XautoSoftmxOnRestore is more clear than just -XautoSoftmx Alternatively, we could extent the behavior of the existing -Xsoftmx option. If that is specified on snapshot side (along Xmx) it basically means that softmx functionality is activated from now on and restore should continue to auto adjust regardless of explicit Xmx. Since Xsoftmx takes an argument, it could also limit to which point auto-on-restore could really adjust (by default softmx is 0, so it effectively disables auto adjust on restore, if Xsoftmx is not specified). I'm not pushing that approach, but perhaps someone likes it.... |
Case 1, restore 16G, I believe we should proceed with softmx6G and no error message, since it's less intrusive. Note that what will help with potential confusion is that we will report in restore's VGC (re-)initialize stanza that softmx was set (this is just a general change that we recently added, regardless if softmx was explicitly set or implicitly activated on restore). |
Case 7 (Xmx Xms set on snapshot) should be 2 sub-cases, with and without auto-adjust-softmx option. If not specified, we should be consistent with case 2 in a way if we firmly obey snapshot's Xmx we should also obey Xms, so we proceed, no softmx set. If the option is specified, for both 4G and 16G restore failing is probably ok. The existing messages 'softmx too large for Xms/Xmx' might not be as confusing (as for the one that Dmitri commented about for case 1), since here we would have that extra auto-adjust-softmx option. |
I want to understand the thinking on "error message" when softmx value chosen is too large for the restore environment and hence reduced (e.g. 16g case in row 1). As I wrote in an earlier comment, I was not really picturing a message any place our customers usually look, e.g. javacore or tracepoint output from the JVM. Having a message (this is why I called it a "warning" instead of an "error") could be useful for our own service work in case we wanted to understand why a certain value was picked. In such a case, it would be like we made it work silently from a customer perspective (since they never see the message to begin with) and hopefully not confusing. |
Interesting, I did not know that we would adapt on the restore side even a user explicitly sets number of GC threads on the checkpoint side (a rare case I feel) via the -Xgcthreads option. One other way of becoming more consistent vs what we are discussing for -Xmx and -Xms in this issue is to change the behavior for -Xgcthreads to not adapt on the restore side if the user had specified -Xgcthreads on the checkpoint side. Having said that, there may well be an inherent/significant difference in the difficulty level of the problem of adapting GC threads vs adapting Java heap memory sizes and so one need not have complete consistency in how the two problems are handled, though obviously if we can reasonably be consistent, then we can consider it. |
Regarding Aleks' comment #18217 (comment) I am fine with I prefer that over the alternative in that comment, i.e. having the use of mx and softmx simultaneously on the checkpoint side mean that dynamic adjustment is to be applied on the restore side. One use case that needs to be kept in mind is that the Open Liberty server script can fail to restore under certain (hopefully rare) situations, and in such cases, it falls back to start Open Liberty up normally (in JVM mode, rather than a restore). Perhaps we want to think through this scenario with respect to all the cases in the table anyway, but this was one reason I was thinking a new option such as |
Regarding Aleks' comment #18217 (comment) |
Update: In Row 1 (No Options (default mode))
Row 2
Row 3 & 4
Row 7
|
@kangyining Please write a documentation that captures the behaviours above. |
Proposed doc: -Xsoftmx use cases in snapshot restore:
For example: if we do snapshot on a machine with 8GB RAM running docker, the heuristic will set the max heap memory to be 6GB. When we restore on another machine with 4GB RAM, we will set the max heap memory to be full 4GB instead of the default 3GB. Note here, JVM won’t generate output explicit warning message for softmx change, but record in verbose gc. And if JVM decides to do nothing, it outputs nothing which suggests softmx is not introduced.
Note, if -Xmx is set to be larger than the available memory, a few errors could happen: OOM, excessive paging, or a bus error; if -Xms is set to be larger than the available memory or softmx value, an error message will be sent and a fatal error occurs. 2.2 If -Xmx or -Xms is specified together with -XdynamicHeapAdjustment: For example: if the machine has 8GB memory but -Xmx12G, then we will still use implicit softmx 6G as default heuristics give. |
|
Another question rises about the -Xsoftmx at the snapshot/restore side: On snapshot side, we probably shouldn't preserve its command line value since we can set it by the Java_com_ibm_java_lang_management_internal_MemoryMXBeanImpl_setMaxHeapSizeImpl() function. The command line option behaves really like an intermediate call. On restore side, if we specify the softmx at command line we probably want to use it as a lifetime one, since this is somewhat similar to a second "Xmx" for restore machine. Should we support these two different modes of softmx or we have a better approach for these? |
Just so I understand, what is the down side of obeying Java_com_ibm_java_lang_management_internal_MemoryMXBeanImpl_setMaxHeapSizeImpl() function after restore ? i.e. have it affect softmx value specified on restore ? I don't know if we need to change/complicate the behavior of softmx in order to make (I think) a rare scenario work differently (rare == someone has softmx explicitly specified on restore side and they call the MXBean api and they are looking for some autonomic behavior by the GC to give them some other behavior). |
@vijaysun-omr Hi Vijay, after some discussions with Aleks, I think I misunderstand the user-specified softmx option at restore previously. I'll just correct it here: For snapshot side command line, we just store the user-provided value, and it behaves really like an intermediate call towards Java_com_ibm_java_lang_management_internal_MemoryMXBeanImpl_setMaxHeapSizeImpl(). For restore side initialization, we have two (potentially) softmx sources, one from default heuristic introduced by this PR, and one from the command line. We will just prioritize the user-provided softmx value here. As you said, we will simply obey Java_com_ibm_java_lang_management_internal_MemoryMXBeanImpl_setMaxHeapSizeImpl() after initialization. |
When we snapshot on a machine with higher physical memory and restore on one with lower physical memory, the InstantOn JVM heap size will be larger than a usual JVM run on the restore machine. And the momory usage will be abnormally higher. Update the extension with the correct physical memory limit and utilize the existing softmax mechanism to solve the problem. Note we only update the softmax if it is smaller than the Xmx size. If it is smaller than the Xms size, then we upscale the softmx to match with the Xms size. Related: eclipse-openj9#17596 Detailed behavior table: eclipse-openj9#18217 (comment) Signed-off-by: Frank Kang frank.kang@ibm.com
Resolved by #18168 (and a couple of others PRs) |
Overview
The current documentation states the following:
We would like to offer support for users who would like to create a single image and restore it on various nodes with different memory limits. There are several scenarios to consider.
[1] When running in a container the JVM will set Xmx to 75% of the available memory if no Xmx is set.
[2] Softmx applies a soft limit on the max heap size. This limit cannot be larger than the max heap limit at startup. It cannot be lower than the minimum heap limit at startup. When running in a container the JVM can implicitly set softmx to 75% of the available memory upon restore.
[3] When Xmx is set at checkpoint (JVM start) this value is retained for the lifetime of the JVM.
[4] If Xmx is set to be larger than the available memory on the system there could be an OOM if one is unable to commit all required memory, there could be excessive paging or a bus error.
[5] Dynamic Max Heap adjustment (we can come up with a better name) is a mode that "overrides" the default Xmx value by applying a softmx that is 75% of the available memory.
[6] Softmx cannot be set larger than max heap limit at startup.
[7] Softmx cannot be set lower than min heap limit.
[8] If the implicit softmx is calculated to be higher than that max heap limit then an error message is outputed and softmx is set to the max heap limit.
Key Questions
The text was updated successfully, but these errors were encountered: