Add Unmounted Thread info to javacore #18218

fengxue-IS · 2023-10-01T19:19:22Z

Add -XX:[+|-]ShowUnmountedThreadStacks option
Refactor RasDumpGlobalStorage.showNativeSymbols field into a bit flag dumpFlags
Add new J9RAS_JAVADUMP_SHOW_UNMOUNTED_THREAD_STACKS bit flag

Signed-off-by: Jack Lu Jack.S.Lu@ibm.com

fengxue-IS · 2023-10-01T19:48:47Z

@babsingh @TobiAjila Do you have any recommendation on what option should this be en-able under?

babsingh · 2023-10-02T14:05:07Z

Option 1: Add opts= for the java dump agent; https://eclipse.dev/openj9/docs/xdump/#optsoptions; -Xdump:java:opts=DISABLE_VIRTUALTHREAD_STACKS. Currently, only heap dumps, z/OS system dumps and tool dumps have support for opts= as per the documentation.

Option 2: Add a standalone option. Existing option: -XX:[+|-]ShowNativeStackSymbols; https://eclipse.dev/openj9/docs/xxshownativestacksymbols. The new option can be -XX:[+|-]ShowVirtualThreadStacks. More easy to implement, document and for a user to use.

@pshipton Requesting your opinion. Shall we proceed with Option 2?

pshipton · 2023-10-02T14:09:00Z

Option 2 is fine with me. What is the downside of enabling this by default?

babsingh · 2023-10-02T14:18:42Z

What is the downside of enabling this by default?

This can create very large javacores in case of stress benchmarks, which can have 1 million+ virtual threads. In extreme cases, this can cause disk space issues on machines (jenkins) and artifactory. We should selectively enable this option for diagnosis where the user can monitor the size of the javacores and perform the required clean up at the end.

tajila · 2023-10-02T15:10:18Z

Option 2 looks good

fengxue-IS · 2023-10-04T18:37:34Z

sample output in javacore dump:

1XMVTHDINFO    Virtual Threads
NULL           ===============
NULL
3XMTHREADINFO      "vthread-unmounted" [unmounted] J9VMContinuation:0x0000000146805DA0, java/lang/VirtualThread:0x000000047FF03848
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.yieldImpl(Native Method)
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield0(Continuation.java:255)
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield(Continuation.java:236)
4XESTACKTRACE                at java/lang/VirtualThread.yieldContinuation(VirtualThread.java:447)
4XESTACKTRACE                at java/lang/VirtualThread.park(VirtualThread.java:598)
4XESTACKTRACE                at java/lang/Access.parkVirtualThread(Access.java:530)
4XESTACKTRACE                at jdk/internal/misc/VirtualThreads.park(VirtualThreads.java:54)
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:369)
4XESTACKTRACE                at javacore.lambda$main$0(javacore.java:14)
4XESTACKTRACE                at javacore$$Lambda/0x00000000300d05e0.run(Bytecode PC:0)
4XESTACKTRACE                at java/lang/VirtualThread.runWith(VirtualThread.java:347)
4XESTACKTRACE                at java/lang/VirtualThread.run(VirtualThread.java:317)
4XESTACKTRACE                at java/lang/VirtualThread$VThreadContinuation$1.run(VirtualThread.java:198)
4XESTACKTRACE                at jdk/internal/vm/Continuation.enter(Continuation.java:186)

3XMTHREADINFO      "random-core" [mounted] J9VMContinuation:0x00000001469147D0, java/lang/VirtualThread:0x000000047FF03938
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.enterImpl(Native Method)
4XESTACKTRACE                at jdk/internal/vm/Continuation.run(Continuation.java:213)
4XESTACKTRACE                at java/lang/VirtualThread.runContinuation(VirtualThread.java:229)                                                                                                                        
4XESTACKTRACE                at java/lang/VirtualThread$$Lambda/0x0000000000000000.run(Bytecode PC:4)
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)                                                                                                   
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doExec(ForkJoinTask.java:387)                                                                                                                        
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)                                                                                                       
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.scan(ForkJoinPool.java:1843)                                                                                                                         
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.runWorker(ForkJoinPool.java:1808)                                                                                                                    
4XESTACKTRACE                at java/util/concurrent/ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)                                                                                                           
                                                                                                                                                                                                                       
NULL           ------------------------------------------------------------------------

@babsingh can you please take a look

babsingh · 2023-10-04T20:31:49Z

@babsingh can you please take a look

Had an in-person review with @fengxue-IS. Few doubts about the high level design and minor formatting nitpicks. @fengxue-IS to clarify design requirements for the new feature with @tajila.

fengxue-IS · 2023-10-04T20:39:50Z

Per discussion with @babsingh, few points to clarify:

Should VirtualThread (ie. Continuation) stack be printed in a separate sub section or part of Thread Details
If the VirtualThread is mounted, hence Continuation state represent the Carrier stack, should it reference the carrier name/object or Virtual?
For fields/info which do not exist in VirtualThread, should those be printed with N/A or just avoid printing?
Currently, mounted VirtualThread stack are printed as part of ThreadDetail using the CarrierThread name, should this be fixed in this PR or separately?

@tajila

0xdaryl · 2023-10-04T21:04:41Z

I imagine the eyecatchers like 3XMTHREADINFO are used by external analysis tools. Are virtual threads sufficiently different that we should leave 3XMTHREADINFO for the platform/carrier threads and introduce something new like 3XMVTHREADINFO for the virtual threads? I'm not aware what external tooling there is that depends on this, but I'd hate to break any assumptions.

tajila · 2023-10-04T23:16:48Z

I imagine the eyecatchers like 3XMTHREADINFO are used by external analysis tools.

I think the defensive option is to name it differently and tools can opt in to collecting that info.

tajila · 2023-10-04T23:26:30Z

In general I think anything that is mounted should be printed as part of the standard javacore output. So in the case where -XX:-ShowVirtualThreadStacks we should still show all the vthreads that are mounted.

I think it makes sense to separate Thread details from Virtual Threads as tools may already know about former.

So for the following:

Should VirtualThread (ie. Continuation) stack be printed in a separate sub section or part of Thread Details

Separate section

If the VirtualThread is mounted, hence Continuation state represent the Carrier stack, should it reference the carrier name/object or Virtual?

In thread details we should print the carrier stack. In the Virtual Thread section we should print the vthread stack. This implies that if there is a single vthread mounted, we will have a VirtualThread section regardless of what options are passed.

For fields/info which do not exist in VirtualThread, should those be printed with N/A or just avoid printing?

If we do the above, I think we can avoid this issue

We can discuss tomorrow to see how feasible this approach is.

fengxue-IS · 2023-10-05T18:20:52Z

Proposed new format changes:

fix naming for mounted virtual thread in CurrentThread/ThreadDetail section
Add new 3XMTHREADINFO4 at end of each thread's info to provide thread type and linkage info (ie. Platform/Carrier/Virtual, for Virtual thread, also include the Carrier Thread associated)
Add new 1XMVTHDINFO Unmounted Threads section after 1XMTHDSUMMARY Threads CPU Usage Summary
Add new 3XM... level tags for unmounted threads info

ex:

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "virtualThread-1" J9VMThread:0x0000000145E2AD00, omrthread_t:0x0000000145024860, java/lang/Thread:0x000000047FF03938, state:R, prio=0
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000047FF10380)
3XMTHREADINFO1            (native thread ID:0x1191A83, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x000000017094C000, to:0x000000017098F000, size:0x43000)
3XMCPUTIME               CPU usage total: 0.003150000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=117744 (0x1CBF0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at javacore.exceptionCaller(javacore.java:7)
4XESTACKTRACE                at javacore.lambda$main$1(javacore.java:20)
4XESTACKTRACE                at javacore$$Lambda/0x00000000300d0730.run(Bytecode PC:0)
4XESTACKTRACE                at java/lang/VirtualThread.runWith(VirtualThread.java:347)
4XESTACKTRACE                at java/lang/VirtualThread.run(VirtualThread.java:317)
4XESTACKTRACE                at java/lang/VirtualThread$VThreadContinuation$1.run(VirtualThread.java:198)
4XESTACKTRACE                at jdk/internal/vm/Continuation.enter(Continuation.java:186)
3XMTHREADINFO3           No native callstack available for this thread
3XMTHREADINFO4           Type:Virtual, CarrierThread:0x000000047FF03548
NULL
NULL
NULL
1XMTHDINFO     Thread Details
NULL           
3XMTHREADINFO      "main" J9VMThread:0x0000000147092B00, omrthread_t:0x0000000146008850, java/lang/Thread:0x000000047FF034E8, state:P, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x2, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000047FF10380)
3XMTHREADINFO1            (native thread ID:0x1191A61, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
3XMTHREADINFO2            (native stack address range from:0x000000016FBB4000, to:0x000000016FBF7000, size:0x43000)
3XMCPUTIME               CPU usage total: 0.324328000 secs, current category="Application"
3XMTHREADBLOCK     Parked on: java/util/concurrent/CountDownLatch$Sync@0x000000047FFB69C0 Owned by: <unknown>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=39024 (0x9870)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/misc/Unsafe.park(Native Method)
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:221)
4XESTACKTRACE                at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
4XESTACKTRACE                at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1099)
4XESTACKTRACE                at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:230)
4XESTACKTRACE                at java/lang/VirtualThread.joinNanos(VirtualThread.java:840)
4XESTACKTRACE                at java/lang/Thread.join(Thread.java:2085)
4XESTACKTRACE                at java/lang/Thread.join(Thread.java:2178)
4XESTACKTRACE                at javacore.main(javacore.java:30)
3XMTHREADINFO3           No native callstack available for this thread
3XMTHREADINFO4           Type:Platform
NULL

...

1XMTHDSUMMARY  Threads CPU Usage Summary
NULL           =========================
NULL
1XMTHDCATINFO  Warning: to get more accurate CPU times for the GC, the option -XX:-ReduceCPUMonitorOverhead can be used. See the user guide for more information.
NULL
1XMTHDCATEGORY All JVM attached threads: 0.407217000 secs
1XMTHDCATEGORY |             
2XMTHDCATEGORY +--System-JVM: 0.389157000 secs
2XMTHDCATEGORY |  |          
3XMTHDCATEGORY |  +--GC: 0.002902000 secs
2XMTHDCATEGORY |  |          
3XMTHDCATEGORY |  +--JIT: 0.070909000 secs
1XMTHDCATEGORY |
2XMTHDCATEGORY +--Application: 0.018060000 secs
NULL
1XMVTHDINFO    Unmounted Threads
NULL           ===============
NULL
3XMVTHDINFO        "vthread-unmounted" [unmounted] J9VMContinuation:0x0000000146805DA0, java/lang/Thread:0x000000047FF03848
3XMVTHDINFO1             Type:Virtual, etc...
3XMVTHDINFO2             Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.yieldImpl(Native Method)
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield0(Continuation.java:255)
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield(Continuation.java:236)
4XESTACKTRACE                at java/lang/VirtualThread.yieldContinuation(VirtualThread.java:447)
4XESTACKTRACE                at java/lang/VirtualThread.park(VirtualThread.java:598)
4XESTACKTRACE                at java/lang/Access.parkVirtualThread(Access.java:530)
4XESTACKTRACE                at jdk/internal/misc/VirtualThreads.park(VirtualThreads.java:54)
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:369)
4XESTACKTRACE                at javacore.lambda$main$0(javacore.java:14)
4XESTACKTRACE                at javacore$$Lambda/0x00000000300d05e0.run(Bytecode PC:0)
4XESTACKTRACE                at java/lang/VirtualThread.runWith(VirtualThread.java:347)
4XESTACKTRACE                at java/lang/VirtualThread.run(VirtualThread.java:317)
4XESTACKTRACE                at java/lang/VirtualThread$VThreadContinuation$1.run(VirtualThread.java:198)
4XESTACKTRACE                at jdk/internal/vm/Continuation.enter(Continuation.java:186)
NULL
NULL
3XMVTHDINFO        "ForkJoinPool-1-worker-1" [mounted] J9VMContinuation:0x00000001469147D0, java/lang/Thread:0x000000047FF03548
3XMVTHDINFO1             Type:Carrier, J9VMThread:0x0000000145E2AD00, VirtualThread:0x000000047FF03938
3XMVTHDINFO2             Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.enterImpl(Native Method)
4XESTACKTRACE                at jdk/internal/vm/Continuation.run(Continuation.java:213)                                                                                                                                     
4XESTACKTRACE                at java/lang/VirtualThread.runContinuation(VirtualThread.java:229)                                                                                                                             
4XESTACKTRACE                at java/lang/VirtualThread$$Lambda/0x0000000000000000.run(Bytecode PC:4)                                                                                                                       
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)                                                                                                        
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doExec(ForkJoinTask.java:387)                                                                                                                             
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)                                                                                                            
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.scan(ForkJoinPool.java:1843)                                                                                                                              
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.runWorker(ForkJoinPool.java:1808)                                                                                                                         
4XESTACKTRACE                at java/util/concurrent/ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

fengxue-IS · 2023-10-16T18:55:11Z

Personal testing passed, @tajila @babsingh can you take another look please

tajila · 2023-10-23T20:46:33Z

@fengxue-IS Can you please post some sample output for what the latest changes look like with mounted and unmounted threads?

fengxue-IS · 2023-10-24T14:22:55Z

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "vthread-mounted" J9VMThread:0x000000012FEA8900, omrthread_t:0x000000012FEA7650, java/lang/Thread:0x000000047FF026C8, state:R, prio=0
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000047FF00AD8)
3XMTHREADINFO1            (native thread ID:0x179D7C, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x000000016E290000, to:0x000000016E2D3000, size:0x43000)
3XMCPUTIME               CPU usage total: 0.004342000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=98176 (0x17F80)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at javacore.exceptionCaller(javacore.java:7)
4XESTACKTRACE                at javacore.lambda$main$1(javacore.java:20)
4XESTACKTRACE                at javacore$$Lambda/0x00000000300e8730.run(Bytecode PC:0)
4XESTACKTRACE                at java/lang/VirtualThread.runWith(VirtualThread.java:347)
4XESTACKTRACE                at java/lang/VirtualThread.run(VirtualThread.java:317)
4XESTACKTRACE                at java/lang/VirtualThread$VThreadContinuation$1.run(VirtualThread.java:198)
4XESTACKTRACE                at jdk/internal/vm/Continuation.enter(Continuation.java:186)
3XMTHREADINFO3           No native callstack available for this thread
3XMTHREADINFO4           Type:Virtual, java/lang/CarrierThread:0x000000047FF03EB8
NULL
NULL
NULL
1XMTHDINFO     Thread Details
NULL
3XMTHREADINFO      "main" J9VMThread:0x000000012F896D00, omrthread_t:0x000000012F80C250, java/lang/Thread:0x000000047FF02538, state:P, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x2, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000047FF00AD8)
3XMTHREADINFO1            (native thread ID:0x179D5F, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
3XMTHREADINFO2            (native stack address range from:0x000000016D590000, to:0x000000016D5D3000, size:0x43000)
3XMCPUTIME               CPU usage total: 0.368686000 secs, current category="Application"
3XMTHREADBLOCK     Parked on: java/util/concurrent/CountDownLatch$Sync@0x000000047FF02630 Owned by: <unknown>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/misc/Unsafe.park(Native Method)
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:221)
4XESTACKTRACE                at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
4XESTACKTRACE                at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1099)
4XESTACKTRACE                at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:230)
4XESTACKTRACE                at java/lang/VirtualThread.joinNanos(VirtualThread.java:840)
4XESTACKTRACE                at java/lang/Thread.join(Thread.java:2085)
4XESTACKTRACE                at java/lang/Thread.join(Thread.java:2178)
4XESTACKTRACE                at javacore.main(javacore.java:30)
3XMTHREADINFO3           No native callstack available for this thread 
3XMTHREADINFO4           Type:Platform
NULL
NULL
3XMTHREADINFO      "JIT Compilation Thread-000" J9VMThread:0x000000012F8AB500, omrthread_t:0x000000012F80CC60, java/lang/Thread:0x000000047FF028D8, state:R, prio=10                            
3XMJAVALTHREAD            (java/lang/Thread getId:0x4, isDaemon:true)                                                                                                                           
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000047FF00AD8)                                                                                                   
3XMTHREADINFO1            (native thread ID:0x179D62, native priority:0xB, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)                                                       
3XMTHREADINFO2            (native stack address range from:0x000000016D628000, to:0x000000016D72B000, size:0x103000)                                                                            
3XMCPUTIME               CPU usage total: 0.053490000 secs, current category="JIT"                                                                                                              
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)                                                                                                                       
3XMTHREADINFO3           No Java callstack associated with this thread                                                                                                                          
3XMTHREADINFO3           No native callstack available for this thread                                                                                                                          
3XMTHREADINFO4           Type:Platform                                                                                                                                                          
NULL                                                                                                                                                                                            
NULL                                   

...

1XMTHDSUMMARY  Threads CPU Usage Summary
NULL           =========================
NULL
1XMTHDCATINFO  Warning: to get more accurate CPU times for the GC, the option -XX:-ReduceCPUMonitorOverhead can be used. See the user guide for more information.
NULL
1XMTHDCATEGORY All JVM attached threads: 0.453985000 secs
1XMTHDCATEGORY |             
2XMTHDCATEGORY +--System-JVM: 0.433884000 secs
2XMTHDCATEGORY |  |          
3XMTHDCATEGORY |  +--GC: 0.002659000 secs
2XMTHDCATEGORY |  |      
3XMTHDCATEGORY |  +--JIT: 0.069855000 secs
1XMTHDCATEGORY |
2XMTHDCATEGORY +--Application: 0.020101000 secs
NULL
1XMVTHDINFO    Unmounted Threads
NULL           =================
NULL
3XMVTHDINFO        "vthread-unmounted" [unmounted] J9VMContinuation:0x0000000138806260, java/lang/Thread:0x000000047FF027B8                                                                     
3XMVTHDINFO1             Type:Virtual
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.yieldImpl(Native Method)                                                                                                           
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield0(Continuation.java:255)                                                                                                      
4XESTACKTRACE                at jdk/internal/vm/Continuation.yield(Continuation.java:236)                                                                                                       
4XESTACKTRACE                at java/lang/VirtualThread.yieldContinuation(VirtualThread.java:447)                                                                                               
4XESTACKTRACE                at java/lang/VirtualThread.park(VirtualThread.java:598)                                                                                                            
4XESTACKTRACE                at java/lang/Access.parkVirtualThread(Access.java:530)
4XESTACKTRACE                at jdk/internal/misc/VirtualThreads.park(VirtualThreads.java:54)                                                                                                   
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:369)                                                                                               
4XESTACKTRACE                at javacore.lambda$main$0(javacore.java:14)
4XESTACKTRACE                at javacore$$Lambda/0x00000000300e85e0.run(Bytecode PC:0)
4XESTACKTRACE                at java/lang/VirtualThread.runWith(VirtualThread.java:347)                                                                                                         
4XESTACKTRACE                at java/lang/VirtualThread.run(VirtualThread.java:317)                                                                                                             
4XESTACKTRACE                at java/lang/VirtualThread$VThreadContinuation$1.run(VirtualThread.java:198)                                                                                       
4XESTACKTRACE                at jdk/internal/vm/Continuation.enter(Continuation.java:186)                                                                                                       
NULL
3XMVTHDINFO        "ForkJoinPool-1-worker-1" [mounted] J9VMContinuation:0x0000000128804940, java/lang/Thread:0x000000047FF03EB8                                                                 
3XMVTHDINFO1             Type:Carrier, J9VMThread:0x000000012FEA8900, java/lang/VirtualThread:0x000000047FF026C8                                                                                
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.enterImpl(Native Method)                                                                                                           
4XESTACKTRACE                at jdk/internal/vm/Continuation.run(Continuation.java:213)                                                                                                         
4XESTACKTRACE                at java/lang/VirtualThread.runContinuation(VirtualThread.java:229)                                                                                                 
4XESTACKTRACE                at java/lang/VirtualThread$$Lambda/0x0000000000000000.run(Bytecode PC:4)                                                                                           
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)                                                                            
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doExec(ForkJoinTask.java:387)                                                                                                 
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)                                                                                
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.scan(ForkJoinPool.java:1843)                                                                                                  
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.runWorker(ForkJoinPool.java:1808)                                                                                             
4XESTACKTRACE                at java/util/concurrent/ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)                                                                                    
NULL                                                                                                                                                                                            
NULL                                                                                                                                                                                            
NULL           ------------------------------------------------------------------------

FYI @tajila

runtime/rasdump/javadump.cpp

fengxue-IS · 2023-11-08T22:42:22Z

Im running personal builds to verify the change, will mark PR ready once testing have passed.

fengxue-IS · 2023-11-10T21:35:39Z

personal build & local testing passed. @keithc-ca can you take another look please.

runtime/rasdump/javadump.cpp

keithc-ca

Please make the requested changes.

fengxue-IS · 2024-01-02T19:45:38Z

Updated code per review comments.

keithc-ca

Please squash.

runtime/rasdump/javadump.cpp

Refactor RasDumpGlobalStorage.showNativeSymbols field into a bit flag dumpFlags Add new J9RAS_JAVADUMP_SHOW_UNMOUNTED_THREAD_STACKS bit flag Add VThread info to javacore - Fix writeThreadName API to correctly reflect mounted virtual thread name - Add new tag for virtual thread related info - Move get VThread name code inside j9sig_protect call Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>

keithc-ca · 2024-01-02T21:45:57Z

Jenkins test sanity alinux jdk17,jdk21

keithc-ca · 2024-01-02T21:53:18Z

Please create a docs issue.

Ideally we would also have some tests that exercise this and openj9.dtfj should be able to parse the new information (see com.ibm.dtfj.javacore.parser.j9.section.thread). Please create one (or two) issues to continue this work.

keithc-ca · 2024-01-03T00:49:20Z

All the test failures are related to CRIU, not due to this change.

babsingh

LGTM. @tajila's approval is pending before this can be merged.

tajila · 2024-01-04T23:31:30Z

jenkins test sanity alinux64 jdk21

tajila · 2024-01-04T23:31:48Z

@fengxue-IS we should rtry to get this into 0.43

pshipton · 2024-01-04T23:38:41Z

we should rtry to get this into 0.43

It's after M2. If it causes any regressions we may not catch them before RC1. If we can get this merged before the weekend that would help, if the Semeru extended testing runs.

pshipton · 2024-01-04T23:44:46Z

We should have a docs issue opened for this.

fengxue-IS · 2024-01-05T15:23:16Z

Docs issue have been opened at eclipse-openj9/openj9-docs#1241,
I also created a PR for 0.43 if we decide to include it.

pshipton · 2024-01-05T18:28:29Z

Since it's disabled by default it should be low risk to include in 0.43

fengxue-IS force-pushed the javacore branch 2 times, most recently from 178333f to edb2029 Compare October 3, 2023 20:19

fengxue-IS changed the title ~~[WIP] Add VThread info to javacore~~ Add VThread info to javacore Oct 4, 2023

fengxue-IS marked this pull request as ready for review October 4, 2023 18:34

fengxue-IS requested a review from babsingh October 4, 2023 18:34

fengxue-IS force-pushed the javacore branch 6 times, most recently from 684e69c to c4e3264 Compare October 12, 2023 18:47

fengxue-IS force-pushed the javacore branch from c4e3264 to 2f43c55 Compare October 16, 2023 18:40

fengxue-IS changed the title ~~Add VThread info to javacore~~ Add Unmounted Thread info to javacore Oct 16, 2023

babsingh requested a review from tajila October 17, 2023 19:32

keithc-ca requested changes Oct 24, 2023

View reviewed changes

fengxue-IS marked this pull request as ready for review November 10, 2023 21:34

keithc-ca requested changes Nov 16, 2023

View reviewed changes

fengxue-IS force-pushed the javacore branch from 7c05ba5 to 5d9f615 Compare November 23, 2023 17:30

keithc-ca requested changes Jan 2, 2024

View reviewed changes

fengxue-IS force-pushed the javacore branch from 5d9f615 to 4232a95 Compare January 2, 2024 19:41

keithc-ca requested changes Jan 2, 2024

View reviewed changes

runtime/rasdump/javadump.cpp Outdated Show resolved Hide resolved

fengxue-IS force-pushed the javacore branch from 4232a95 to d8387b4 Compare January 2, 2024 20:25

keithc-ca approved these changes Jan 3, 2024

View reviewed changes

babsingh approved these changes Jan 4, 2024

View reviewed changes

This was referenced Jan 4, 2024

Document new -XX:[+|-]ShowUnmountedThreadStacks option eclipse-openj9/openj9-docs#1241

Closed

Support javacore "Unmounted Threads" section in DTFJ #18686

Open

tajila approved these changes Jan 4, 2024

View reviewed changes

pshipton added comp:vm depends:doc comp:ras labels Jan 4, 2024

fengxue-IS mentioned this pull request Jan 5, 2024

(0.43) Add -XX:[+|-]ShowUnmountedThreadStacks option #18692

Merged

keithc-ca merged commit 0cd684e into eclipse-openj9:master Jan 9, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Unmounted Thread info to javacore #18218

Add Unmounted Thread info to javacore #18218

fengxue-IS commented Oct 1, 2023 •

edited

fengxue-IS commented Oct 1, 2023

babsingh commented Oct 2, 2023 •

edited

pshipton commented Oct 2, 2023

babsingh commented Oct 2, 2023

tajila commented Oct 2, 2023

fengxue-IS commented Oct 4, 2023

babsingh commented Oct 4, 2023

fengxue-IS commented Oct 4, 2023 •

edited

0xdaryl commented Oct 4, 2023

tajila commented Oct 4, 2023

tajila commented Oct 4, 2023

fengxue-IS commented Oct 5, 2023 •

edited

fengxue-IS commented Oct 16, 2023

tajila commented Oct 23, 2023

fengxue-IS commented Oct 24, 2023

fengxue-IS commented Nov 8, 2023

fengxue-IS commented Nov 10, 2023

keithc-ca left a comment

fengxue-IS commented Jan 2, 2024

keithc-ca left a comment

keithc-ca commented Jan 2, 2024

keithc-ca commented Jan 2, 2024

keithc-ca commented Jan 3, 2024

babsingh left a comment

tajila commented Jan 4, 2024

tajila commented Jan 4, 2024

pshipton commented Jan 4, 2024

pshipton commented Jan 4, 2024

fengxue-IS commented Jan 5, 2024

pshipton commented Jan 5, 2024

Add Unmounted Thread info to javacore #18218

Add Unmounted Thread info to javacore #18218

Conversation

fengxue-IS commented Oct 1, 2023 • edited

fengxue-IS commented Oct 1, 2023

babsingh commented Oct 2, 2023 • edited

pshipton commented Oct 2, 2023

babsingh commented Oct 2, 2023

tajila commented Oct 2, 2023

fengxue-IS commented Oct 4, 2023

babsingh commented Oct 4, 2023

fengxue-IS commented Oct 4, 2023 • edited

0xdaryl commented Oct 4, 2023

tajila commented Oct 4, 2023

tajila commented Oct 4, 2023

fengxue-IS commented Oct 5, 2023 • edited

fengxue-IS commented Oct 16, 2023

tajila commented Oct 23, 2023

fengxue-IS commented Oct 24, 2023

fengxue-IS commented Nov 8, 2023

fengxue-IS commented Nov 10, 2023

keithc-ca left a comment

Choose a reason for hiding this comment

fengxue-IS commented Jan 2, 2024

keithc-ca left a comment

Choose a reason for hiding this comment

keithc-ca commented Jan 2, 2024

keithc-ca commented Jan 2, 2024

keithc-ca commented Jan 3, 2024

babsingh left a comment

Choose a reason for hiding this comment

tajila commented Jan 4, 2024

tajila commented Jan 4, 2024

pshipton commented Jan 4, 2024

pshipton commented Jan 4, 2024

fengxue-IS commented Jan 5, 2024

pshipton commented Jan 5, 2024

fengxue-IS commented Oct 1, 2023 •

edited

babsingh commented Oct 2, 2023 •

edited

fengxue-IS commented Oct 4, 2023 •

edited

fengxue-IS commented Oct 5, 2023 •

edited