High CPU usage with multiple detached windows

[rjwills28]

I have been testing the performance of Phoebus, measuring the CPU and memory usage. I have found that the CPU usage is very high when opening multiple screens in detached windows in Phoebus. I have tested opening 10 windows from the command line using the following command:

./phoebus.sh -resource file:/path/to/bob/testBob0.bob?target=window -resource file:/path/to/bob/testBob1.bob?target=window -resource file:/path/to/bob/testBob2.bob?target=window ...

The CPU usage of Phoebus was ~120%, while opening the same number of standalone windows in CS-Studio was of order 40%.

In addition I see a lot of threading related exceptions thrown when initially opening 6 or more detached windows in this way. Below is a snippet but I have also attached the full log: phoebus_thread_exceptions.txt

2024-10-16 12:50:25 SEVERE [org.phoebus.ui.application.PhoebusApplication] UI Freezeup
...
*********************************
*** JavaFX Application Thread ***
*********************************
"JavaFX Application Thread" prio=5 Id=34 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@68d8eaf8 owned by "QuantumRenderer-0" Id=29
	at java.base@21.0.4/jdk.internal.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@68d8eaf8
	at java.base@21.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
	at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
	at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
	at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
	at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
	at app//com.sun.javafx.tk.quantum.QuantumToolkit.runWithRenderLock(QuantumToolkit.java:440)
	at app//com.sun.javafx.tk.quantum.GlassViewEventHandler$ViewEventNotification.run(GlassViewEventHandler.java:882)
	...

...
2024-10-16 12:50:26 SEVERE [org.phoebus.ui.application.PhoebusApplication] UI Updates resume

Further info

• I do not see this error when opening 10 screens in tabs (i.e. without the ?target=window extension).
• I have tested against the master GitHub branch as well as the v4.7.3 release and see the same behaviour in both.
• I am running Phoebus on a machine with 4 cores, 8GB RAM running Fedora 40 OS.
• I am using no special JVM environment variables although have tested using -Dprism.forceGPU=true, which only increased the CPU usage.

Is this something that has been noticed before with multiple detached windows?

[kasemir]

Your finding is interesting, but not sure what we can do about it.
Just to confirm, I assume your issue is not literally about opening the displays, where opening a new window make take a little longer than adding a tab within an existing window. You're concerned about the runtime CPU usage later on, when the windows have been open for a while, right?

The "UI Freezeup" messages are from the https://github.com/ControlSystemStudio/phoebus/blob/master/core/ui/src/main/java/org/phoebus/ui/monitoring/ResponsivenessMonitor.java
There was a similar tool in Eclipse/RCP. It helps us to detect UI thread slowdowns. The stack trace would then for example show that we're opening and parsing a file on the UI thread, which we should not do. In your case, the "JavaFX Application Thread" is not blocked by anything that we're doing, it's simply waiting for the "QuantumRenderer-0" to draw stuff on the screen, namely a rectangle:

"JavaFX Application Thread" prio=5 Id=34 WAITING
   on java.util.concurrent.locks.ReentrantLock$NonfairSync@68d8eaf8
  owned by "QuantumRenderer-0" Id=29
	at java.base@21.0.4/jdk.internal.misc.Unsafe.park(Native Method)

"QuantumRenderer-0" daemon prio=5 Id=29 RUNNABLE
	at app//com.sun.pisces.PiscesRenderer.clearRectImpl(Native Method)
	at app//com.sun.pisces.PiscesRenderer.clearRect(PiscesRenderer.java:327)
	at app//com.sun.prism.sw.SWGraphics.clear(SWGraphics.java:370)
	at app//com.sun.javafx.tk.quantum.ViewPainter.doPaint(ViewPainter.java:468)
	...
	Number of locked synchronizers = 2
	- java.util.concurrent.ThreadPoolExecutor$Worker@6b09fbe3
	- java.util.concurrent.locks.ReentrantLock$NonfairSync@68d8eaf8

We do know that JavaFX needs more and more time as the scene graph grows. That's why we're trying to keep the scene graph small. For example, our plots are not like the JavaFX charts where every tick mark on the axis is a "Line", every number on the axis is a "Label" and so on, because that way one plot would easily become a sub-scene with 1000+ nodes. Instead, we create plots in background threads as images, and in the JavaFX scene graph it's then a single node for displaying the picture. See https://github.com/ControlSystemStudio/phoebus/tree/master/app/display/representation-javafx/src/test/java/org/csstudio/display/builder/representation/javafx/sandbox for things that were tested.

So back to your finding. What you see is basically that this scene graph:

Window/Stage -> Tab -> { panel1, panel2, panel3, ...} 

is faster than this one:

Window/Stage1 -> panel1
Window/Stage2 -> panel2
Window/Stage3 -> panel3
...

Maybe you can try those displays that you already created for your tests on Windows and Mac? Maybe also on different Linux setups?
Bottom line, though, I'm afraid that's a JavaFX issue beyond our control. When I google "javafx slow multiple stages" I see no obvious answer. If you can create a standalone JavaFX example that demonstrates the issue, maybe submit a JavaFX bug report?