Only the diagnostic threads should be executing JitDump compilations #11772
Comments
|
The first thing we will notice is that in the jidump the The reason for this will become clear once we examine the verbose log. Opening that up we see the following:
The other compilation threads are still processing the queue, and the application threads are still running. This is why we see lots of stuff printed following the crash point of the compilation on The JitDump logic in JitDump.cpp sets a flag to disable all future compilations and it purges the entire compilation queue. However as we can note by the above printouts (inserted in code by me) the number of active threads does not decrease once we set the flag, or purge the compilation queue. This is very important. We rely on the normal compilation thread suspension logic to suspend the compilation threads when the following conditions are met:
A bit later the JitDump logic resumes the diagnostic thread ( However the logging tells us that the diagnostic thread is resumed, and then subsequently it started processing the queue, and encountered that the queue is empty and it forces itself to get suspended. This occurs because the diagnostic thread is enabled in between purging the compilation queue and queuing the JitDump compilation: There is actually a Remember, during all of this the application is still executing and the VM is still suggesting to the JIT compiler that we should compile methods, as their count goes down to zero. This is why we see compile requests coming in, but they're not satisfied because we have suspended further compilations: Eventually though we do reach a point where i.e. the options to enable tracing are not activated if a normal compilation thread picks up a JitDump compile. This is why our |
|
In my mind the solution to this problem seems to be the following:
@dsouzai what are your thoughts given the above analysis? Edit: Thinking about this more the following logic may not work:
because the JitDump compilation requests are synchronous, so we need to enable the diagnostic thread before the synchronous compile request. The better solution here would be to swap the order, but teach the self-suspension logic not to suspend the diagnostic thread at all. The JitDump process will just suspend the diagnostic thread before finishing. |
|
I also explored the idea of attempting to suspend all "normal" compilation threads right in the JitDump logic, and waiting for that to happen. I used the following code: Unfortunately this will not work. The reason because it is not a good idea to stop/suspend compilation threads in the JitDump logic and wait until they are all stopped/suspended before proceeding with the JitDump logic is because we have a deadlock scenario here because the stop/suspension request has to happen at a yield point the compilation threads. These yield points can be things like attempting to acquire VM access, attempting to acquire various monitors, or attempting to reserve trampoline space, etc. Basically anywhere that checks However there is now a timing hole between when the JitDump logic has requested compilation threads to stop/suspend and when they actually reach such a yield point. The deadlock scenario happens if a compilation thread happens to crash in between these two points. It will then go through the dump triggers and it will halt on the dump monitor which prevents multiple crashes from being processed at the same time. Hence such a thread will never stop/suspend because the previous JitDump process holds such a lock and is waiting for other compilation threads to stop/suspend, i.e. a deadlock scenario. For this reason I do not think it is a good idea to introduce any serialization logic in the JitDump request logic. Ensuring that only the diagnostic thread can process |
Yeah I think this is the best way of going about it; it's probably the simplest solution too. |
…ump compile Only the diagnostic thread should be processing JitDump compilations. This is to ensure proper tracing is active and proper synchronization is performed w.r.t. special events (interpreter shutdown, etc.). The logic here is split into two parts. In the event we are a diagnostic thread, the JitDump process will have ensured the method compilation queue has been purged and all further compilations have been suspended until the entire JitDump process in complete. This also means if we are a diagnostic thread, then the only entries in the queue should be JitDump compile requests. We will ensure this is the case via a fatal assert. In addition there can be scenarios in which a non-diagnostic compilation thread is still attempting to process entries while the JitDump process (in a crashed thread) is happening. This is because one compilation thread must always be active, and there is a timing hole between purging of the queue in the JitDump process, and when the diagnostic thread is resumed. This means a non-diagnostic thread could attempt to pick up a JitDump compilation request (see eclipse-openj9#11772 for details). We must ensure that this does not happen and simply skip the request if we are a non-diagnostic thread attempting to process a JitDump compilation. Signed-off-by: Filip Jeremic <fjeremic@ca.ibm.com>
The original design behind the diagnostic compilation thread was to have a by default suspended compilation thread which will only get activated when the JitDump process is triggered. The idea is that the "normal" compilation threads may not be in the correct state to start compiling, because we have just encountered an error. The diagnostic compilation thread has just been sitting and waiting for someone to activate it and is fully capable of triggering a compilation.
Unfortunately as we will describe below via logging, it seem that "normal" compilation threads are picking up JitDump compiles. This is something we do not want, because there are a number of things that we do for JitDump compilations which are special (among others which I am likely missing):
This can be observed via a simple
java -versioncommand:Note that "crashDuringCompile option is set" is seen twice. The first time is when we trigger the initial compilation, and the second time is when we are doing the JitDump compilation. I'm attaching both the jidump trace log and the verbose log to this issue for future reference. Let's dissect the verbose log to see what happen inside.
jitdump-verbose-log.zip
The text was updated successfully, but these errors were encountered: