Unable to find references for a thread from an ExecutorService created with Executors.newFixedThreadPool with many lost handle ExecutorService instances

[wadechandler]

I was looking for an issue which was causing thread exhaustion on a system. The end result was lost ExecutorService instances created by someone calling Executors.newFixedThreadPool without shutting down those instances or even tracking their references, but VisualVM couldn't tell me where they were. I had a handle on any number of the thousands of threads, but there reference calculation spun forever. I even left it sitting for many hours. This was using a heap dump. I had multiple dumps which I tried this on, and the result was the same. A simple test case should be to call Executors.newFixedThreadPool multiple times without tracking the instance from some thread, having the executors execute tasks matching the size of the pool, until OOMEs are reached; be sure and give it a ThreadFactory to make names to make them easier to find. My heap dump size info follows:

File size: 230 MB

Total bytes: 206,598,574
Total classes: 21,386
Total instances: 2,940,614
Classloaders: 1,089
GC roots: 7,960
Number of objects pending for finalization: 0

The steps beyond creating and then opening the heap dump:

1. Go to classes
2. Filter by entering "Thread"
3. Double click on java.util.Thread
4. Make sure the "References" pane is showing
5. Click on various instances of thread which match the ThreadFactory naming pattern used in the test
6. Notice the reference window keeps searching, but never will find other references than this
7. Go to classes
8. Filter by entering "ThreadPoolExecutor"
9. Click on various instances, and notice you can drill down to the workers->table and see references to threads with some matching the naming patterns used for the test

From the about windows details:
Version: 1.3.9 (Build 161004); platform 20161002-268c3bcd7a42

System: Mac OS X (10.11.6) , x86_64 64bit

Java: 1.8.0_102; Java HotSpot(TM) 64-Bit Server VM (25.102-b14, mixed mode)

I'm sure this is the case for some other things, but this is a definite and specific example

[thurka]

I am not sure I understand all the details. It would be great if you can share the heap dump, where this issue can be reproduced. Thanks.

[wadechandler]

I have tried to recreate the circumstances with a simple application, but it appears there is more to the pattern of my heap in this case. The bad part is I can not share the heap per its nature, and so much try to recreate the scenario. I will keep trying, but the general steps to drill down to the issue will be the same once we have a heap to share. It doesn't seem related to the exact number of instances and thread exhaustion either. Either way, in my case, when I click on the instance in instances, the "References" pain infinitely searches for instances and shows the hour glass. It takes a while in the attempted reproduction, but certainly finishes, and thus I need to keep trying for a reproduction.

[wadechandler]

I have now even built a Sprint Boot application with multiple services etc in an attempt to replicate, but still isn't the same. I will see what I can do on permissions with the heap dump.

[thurka]

OK, thanks.

[thurka]

Closing for now. Please reopen, once you have more info.