Use fewer compilation threads at client when JITServer is low on memory #11323

dmitry-ten · 2020-12-01T19:26:51Z

When the JITServer starts running low on physical memory, it is
detrimental for clients to have all compilation threads active and
sending requests because it makes JITServer use too many compilation
threads which consume a lot of memory.

At the end of each compilation, server tells the client if it's low
on physical memory. As long as that's the case, client will not
activate a new compilation thread and will keep suspending the existing
ones until only one thread is active.

Resolves: #11175

dmitry-ten · 2020-12-01T19:39:04Z

@mpirvu ready for review.
So, do we want to send updates about the state of JITServer's memory more frequently?
The potential candidate messages with which we could send updates are:

message for creating resolved methods: these are consistently created for all compilations but the downside is that for larger compilations you'd have dozens of updates per one compilation.
IProfiler_profilingSample message: this is less frequent but would only be sent when the compilation uses IProfiling.

mpirvu · 2020-12-01T20:56:22Z

So, do we want to send updates about the state of JITServer's memory more frequently?

When I opened the issue I thought we needed more frequent updates. The only danger right now is the server approaching the limit and the client not knowing and launching another comp thread. The part that suspends compilation threads is fine because we cannot react to it until the compilation is over, so the hint from the compilation reply is enough.
Let's see how it goes with current implementation and if we find we need more frequent updates we'll open another PR

mpirvu

I have a few suggestions

mpirvu · 2020-12-01T23:49:15Z

runtime/compiler/control/CompilationRuntime.hpp

@@ -1048,6 +1048,9 @@ class CompilationInfo
   void  addJITServerSslCert(const std::string &cert) { _sslCerts.push_back(cert); }
   const std::string &getJITServerSslRootCerts() const { return _sslRootCerts; }
   void  setJITServerSslRootCerts(const std::string &cert) { _sslRootCerts = cert; }
+
+   bool serverHasLowPhysicalMemory() { return _serverHasLowPhysicalMemory; }


We could mark this one as const

mpirvu · 2020-12-02T00:00:45Z

runtime/compiler/control/CompilationThread.cpp

@@ -549,6 +549,10 @@ TR_YesNoMaybe TR::CompilationInfo::shouldActivateNewCompThread()
 #if defined(J9VM_OPT_JITSERVER)
      else if (getPersistentInfo()->getRemoteCompilationMode() == JITServer::CLIENT && JITServerHelpers::isServerAvailable())
         {
+         // If the available memory on the server is low, do not activate more threads


I would move this piece of code up where the 'NO' decisions are taken. As it is now, we may take the path that leads to TR_yes above.

mpirvu · 2020-12-02T00:11:26Z

runtime/compiler/control/CompilationThread.cpp

-            compInfo->getSuspendThreadDueToLowPhysicalMemory() ? "LowPhysicalMem" : "");
+            compInfo->getSuspendThreadDueToLowPhysicalMemory() ? "LowPhysicalMem" : "",
+#if defined(J9VM_OPT_JITSERVER)
+            compInfo->serverHasLowPhysicalMemory() ? "ServerLowPhysicalMem" : ""


TR_VerboseLog::writeLineLocked needs another %s for the new string. It gets tricky though due to the ifdef. I am ok with being a little imprecise and say just "LowPhysicalMem", but the decision should come from either the local JVM and or the remote server:

bool lowPhysicalMem = compInfo->getSuspendThreadDueToLowPhysicalMemory(); #if defined(J9VM_OPT_JITSERVER) lowPhysicalMem = lowPhysicalMem || compInfo->serverHasLowPhysicalMemory(); #endif

Yeah, or we could print an empty string if JITServer is not enabled:

compInfo->getRampDownMCT() ? "RampDownMCT" : "", compInfo->getSuspendThreadDueToLowPhysicalMemory() ? "LowPhysicalMem" : "", #if defined(J9VM_OPT_JITSERVER) compInfo->serverHasLowPhysicalMemory() ? "ServerLowPhysicalMem" : #endif ""

I am fine either way

When the JITServer starts running low on physical memory, it is detrimental for clients to have all compilation threads active and sending requests because it makes JITServer use too many compilation threads which consume a lot of memory. At the end of each compilation, server tells the client if it's low on physical memory. As long as that's the case, client will not activate a new compilation thread and will keep suspending the existing ones until only one thread is active. Signed-off-by: Dmitry Ten <Dmitry.Ten@ibm.com>

dmitry-ten · 2020-12-02T15:49:13Z

@mpirvu updated.

mpirvu

LGTM

mpirvu · 2020-12-02T16:23:21Z

jenkins test sanity plinuxjit,xlinuxjit,zlinuxjit jdk11

mpirvu · 2020-12-02T20:57:08Z

Tests have passed, hence merging.

When the server is running low on memory, clients need to reduce the number of active compilation threads so that the remaining memory doesn't get exhausted as quickly. Previous implementation in eclipse-openj9#11323 would suspend threads when the server was low on memory, and resume them once more memory becomes available. However, it suffered from constant oscillations in thread activations: i.e. clients would be notified of low server memory, suspend threads, more memory on the server would become available, meaning all the clients would resume their threads, causing the server to run out of memory again, and the cycle would repeat. This commit improves the previous implementation by using 2 new enums to represent client's thread activation policy and server's memory state. `JITServer::CompThreadActivationPolicy` and `JITServer::ServerMemoryState`. `JITServer::ServerMemoryState` has 3 states indicating how much free memory is left on the server. Each client receives an update at the end of each compilation. - `NORMAL`: there is enough memory available - `LOW`: given the current number of clients, the server might run out of memory soon - `VERY_LOW`: the server is about to exhaust its memory and completely stop all compilations. `JITServer::CompThreadActivationPolicy` has 4 states indicating the manner in which client's compilation threads should be suspended/resumed. - `AGGRESSIVE`: the default policy, activates new compilation threads more aggressively than non-JITServer JVM. - `MAINTAIN`: (VERY_LOW < server_memory <= LOW) - server is running out of memory, active threads are allowed to continue but no new threads can start. If server's memory situation improves, thread activation can be resumed. - `SUSPEND`: (server_memory <= VERY_LOW) - server is about to exhaust memory, suspend all but 1 compilation thread. If server's memory improves back to NORMAL, thread activation can be resumed. - `SUBDUE`: (server_memory > LOW) - this policy can be entered from `MAINTAIN` or `SUSPEND` policies. Client is allowed to resume threads but activation thresholds are much higher than in `AGGRESSIVE` policy. Signed-off-by: Dmitry Ten <Dmitry.Ten@ibm.com>

dmitry-ten force-pushed the server-low-memory branch from b7f79d6 to 02384da Compare December 1, 2020 19:35

mpirvu added the comp:jitserver Artifacts related to JIT-as-a-Service project label Dec 1, 2020

mpirvu added this to In progress in JIT as a Service via automation Dec 1, 2020

mpirvu self-assigned this Dec 1, 2020

mpirvu reviewed Dec 2, 2020

View reviewed changes

dmitry-ten force-pushed the server-low-memory branch from 02384da to 1013f40 Compare December 2, 2020 15:48

mpirvu approved these changes Dec 2, 2020

View reviewed changes

mpirvu merged commit ceeb4c8 into eclipse-openj9:master Dec 2, 2020

JIT as a Service automation moved this from In progress to Done Dec 2, 2020

mpirvu mentioned this pull request Dec 4, 2020

Inform client JVMs sooner that JITServer is running out of memory #11363

Closed

dmitry-ten mentioned this pull request Jan 11, 2021

Improve JITServer client compilation thread activation policy #11628

Merged

mpirvu mentioned this pull request Jan 19, 2021

JITServer stops compiling after reaching memory limit #11154

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use fewer compilation threads at client when JITServer is low on memory #11323

Use fewer compilation threads at client when JITServer is low on memory #11323

dmitry-ten commented Dec 1, 2020 •

edited

Loading

dmitry-ten commented Dec 1, 2020

mpirvu commented Dec 1, 2020

mpirvu left a comment

mpirvu Dec 1, 2020

mpirvu Dec 2, 2020

mpirvu Dec 2, 2020

dmitry-ten Dec 2, 2020

mpirvu Dec 2, 2020

dmitry-ten commented Dec 2, 2020

mpirvu left a comment

mpirvu commented Dec 2, 2020

mpirvu commented Dec 2, 2020

Use fewer compilation threads at client when JITServer is low on memory #11323

Use fewer compilation threads at client when JITServer is low on memory #11323

Conversation

dmitry-ten commented Dec 1, 2020 • edited Loading

dmitry-ten commented Dec 1, 2020

mpirvu commented Dec 1, 2020

mpirvu left a comment

Choose a reason for hiding this comment

mpirvu Dec 1, 2020

Choose a reason for hiding this comment

mpirvu Dec 2, 2020

Choose a reason for hiding this comment

mpirvu Dec 2, 2020

Choose a reason for hiding this comment

dmitry-ten Dec 2, 2020

Choose a reason for hiding this comment

mpirvu Dec 2, 2020

Choose a reason for hiding this comment

dmitry-ten commented Dec 2, 2020

mpirvu left a comment

Choose a reason for hiding this comment

mpirvu commented Dec 2, 2020

mpirvu commented Dec 2, 2020

dmitry-ten commented Dec 1, 2020 •

edited

Loading