Deadlock in indexer #81
Comments
|
I have had two runs now with identical stack traces, so pretty sure this is what is going on. Extracts of key threads: "main" #1 prio=6 os_prio=0 cpu=6380.63ms elapsed=721.72s tid=0x00007f29ec0267b0 nid=0x1eaa7c waiting for monitor entry [0x00007f29f128e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(java.base@17.0.3/Native Method)
- waiting on <no object reference available>
at org.eclipse.cdt.internal.core.pdom.PDOMManager.joinIndexer(PDOMManager.java:1173)
- locked <0x00000000e8e07aa0> (a [Z)
at org.eclipse.cdt.core.testplugin.util.BaseTestCase5.waitForIndexer(BaseTestCase5.java:165)
at org.eclipse.cdt.core.testplugin.util.BaseTestCase.waitForIndexer(BaseTestCase.java:234)
at org.eclipse.cdt.internal.pdom.tests.PDOMCPPBugsTest.test191679(PDOMCPPBugsTest.java:227)
"Worker-5: Update Monitor" #56 prio=5 os_prio=0 cpu=19.67ms elapsed=712.00s tid=0x00007f2848000e70 nid=0x1eab17 waiting for monitor entry [0x00007f2878487000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.eclipse.cdt.internal.core.pdom.PDOMManager.isIndexerIdle(PDOMManager.java:751)
- waiting to lock <0x00000000e2035f38> (a java.util.ArrayDeque)
at org.eclipse.cdt.internal.core.pdom.PDOMManager$7.done(PDOMManager.java:1153)
- locked <0x00000000e8e07aa0> (a [Z)
at org.eclipse.core.internal.jobs.JobListeners$$Lambda$127/0x0000000800da52f0.notify(Unknown Source)
"Worker-0: Notify Index Change Listeners" #43 prio=5 os_prio=0 cpu=709494.63ms elapsed=716.35s tid=0x00007f29ec8e7520 nid=0x1eaadc runnable [0x00007f29a49f3000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.core.internal.jobs.JobManager.schedule(JobManager.java:1353)
at org.eclipse.core.internal.jobs.InternalJob.schedule(InternalJob.java:390)
at org.eclipse.core.runtime.jobs.Job.schedule(Job.java:654)
at org.eclipse.cdt.internal.core.pdom.PDOMManager.enqueue(PDOMManager.java:712)
- locked <0x00000000e2035f38> (a java.util.ArrayDeque)
at org.eclipse.cdt.internal.core.pdom.PDOMManager$3.run(PDOMManager.java:1012)
- locked <0x00000000e2036538> (a java.util.HashMap)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
"Worker-2: Building" #52 prio=5 os_prio=0 cpu=28.64ms elapsed=712.01s tid=0x00007f28c0001820 nid=0x1eab13 waiting for monitor entry [0x00007f287888b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.eclipse.cdt.internal.core.pdom.PDOMManager.handlePostBuildEvent(PDOMManager.java:1499)
- waiting to lock <0x00000000e2036538> (a java.util.HashMap)
at org.eclipse.cdt.internal.core.pdom.CModelListener.resourceChanged(CModelListener.java:166) |
The 2022-12 M1 build from Platform may have a regression that is causing CDT build to fail more often due to deadlock in the indexer. This commit uses a post M1 build of Platform to try pick up fixes that have been committed to Platform since M1 was released. Fixes eclipse-cdt#81
|
I think this is the same regression identified in eclipse-platform/eclipse.platform#193 - I will try updating to newer I-build, and if that works we may need to discuss what to do about the M1 release this week. |
|
With the updated target platform it looks like the problem isn't resolved, but the Java deadlock detector code can identify it now at least: Found one Java-level deadlock:
=============================
"main":
waiting to lock monitor 0x00007f37957fa400 (object 0x00000000e9980f88, a [Z),
which is held by "Worker-11: C/C++ Indexer"
"Worker-11: C/C++ Indexer":
waiting to lock monitor 0x00007f358801b240 (object 0x00000000e09dbbf0, a java.util.ArrayDeque),
which is held by "Worker-7: Notify Index Change Listeners"
"Worker-7: Notify Index Change Listeners":
waiting to lock monitor 0x00007f359c000e70 (object 0x00000000e09dbcf8, a java.util.concurrent.ConcurrentLinkedQueue),
which is held by "Worker-11: C/C++ Indexer"Full jstack output jstack.txt |
The 2022-12 M1 build from Platform may have a regression that is causing CDT build to fail more often due to deadlock in the indexer. This commit uses 2022-09 R build of Platform to avoid the problem. Note that this won't resolve the issue in Platform, but will allow CDT to be built. Fixes eclipse-cdt#81
|
CDT's immediate issue is fixed (hopefully) with #82 - but that really just hides the problem hoping that Platform resolves the issue in eclipse-platform/eclipse.platform#193 |
|
the deadlock described in #81 (comment) CDT should not synchronize in the JobChangeAdapter AND outside the while loop you could just synchronize while waiting to be notified. |
|
Thank you @jukzi for the analaysis. I have no doubt that there is some improvements to the indexer joining code (and this issue needs to be reopened) but it was written more than a decade ago. So the behaviour change in Platform has regressed CDT and I can't see where CDT violated the API, but perhaps it is because CDT depended on undocumented behaviour. Anyway, I will see if I can provide a reduced test case, which will either show the remaining issue in platform, or at least allow me to understand the problem more fully. The existing code is way too complicated within CDT, and frustratingly the code tends not to lock up when running under the debugger. It fails quite often (not 100% of the time) when I run the tests in the IDE, but it has never failed when debugging. |
|
Well the timing in debugging is different, but you could for sure reproduce this obvious deadlock with breakpoints in the synchronized blocks. I understand this deadlock. I will also think about how he platform could get rid of it's synchronize. |
|
Me blaming PDOMManager.idleCondition was wrong. Sorry. The problem is That should be done outside the synchronize. Same pattern is already used in probably same problem in |
This is an experiment as recommended in eclipse-platform/eclipse.platform#193 (comment) to see if latest I-build works
|
BTW that code is totally wrong: It adds the listener to every job so that any(!) job that completes will set the idleCondition even if other jobs still running. And it synchronizes all(!) jobs done(). Thats why all these unrelated threads: What probably is wanted here is to only listen for fIndexerJob to be done. So you could instead add and remove the job to/from only that Job. |
|
You may be right - that heavy joinIndexer code is mostly used in tests to make sure everything is idle. |
This is an experiment as recommended in eclipse-platform/eclipse.platform#193 (comment) to see if latest I-build works
This is an experiment as recommended in eclipse-platform/eclipse.platform#193 (comment) to see if latest I-build works
Because of [changes](https://www.eclipse.org/eclipse/news/4.26/platform_isv.php#JobManager_Implementation) in Eclipse Platform where the jobmanager's behaviour changes (within the API), the consumers of the jobmanager can deadlock due to incorrect assumptions. In particular, where we call job.schedule(), the callback can happen in different threads to the IJobChangeListener's. As CDT was holding a lock while calling schedule that is also required in those listeners, we need to no longer lock when calling schedule. As the code already dealt with the case of when there was a delay between the job.schedule() and where & when it was run, we can move the schedule call out of the synchronized block. Fixes eclipse-cdt#81
Because of [changes](https://www.eclipse.org/eclipse/news/4.26/platform_isv.php#JobManager_Implementation) in Eclipse Platform where the jobmanager's behaviour changes (within the API), the consumers of the jobmanager can deadlock due to incorrect assumptions. In particular, where we call job.schedule(), the callback can happen in different threads to the IJobChangeListener's. As CDT was holding a lock while calling schedule that is also required in those listeners, we need to no longer lock when calling schedule. As the code already dealt with the case of when there was a delay between the job.schedule() and where & when it was run, we can move the schedule call out of the synchronized block. Fixes eclipse-cdt#81
Because of [changes](https://www.eclipse.org/eclipse/news/4.26/platform_isv.php#JobManager_Implementation) in Eclipse Platform where the jobmanager's behaviour changes (within the API), the consumers of the jobmanager can deadlock due to incorrect assumptions. In particular, where we call job.schedule(), the callback can happen in different threads to the IJobChangeListener's. As CDT was holding a lock while calling schedule that is also required in those listeners, we need to no longer lock when calling schedule. As the code already dealt with the case of when there was a delay between the job.schedule() and where & when it was run, we can move the schedule call out of the synchronized block. Fixes #81
Something in the recent changes to #79 has caused the indexer to deadlock more readily than before.
Running PDOMCPPBugsTest on the build machine locks up, and it does when running (but not debugging) a significant percentage of the time.
The text was updated successfully, but these errors were encountered: