many extra render buffers & textures w/ even simple sketches on OS X

[allesblinkt]

Hello,

I recently ran into some severe performance problems, especially having OpenGL sketches with a big resolution (e.g. 3840 x 1080 pixels). The sketches are fairly simple (no offscreen drawing / FBOs, but still performance suffers severely. I am using the most recent version from git (1fcd354). Even for simple sketches, the GPU memory usage spiked to over 500 megabytes (just by running the sketch) with not much in it. After some investigation, I suspect two circumstances:

Too many render buffers

The sketch is initalized with size(3840, 1080, OPENGL); smooth(4); in setup(). There is no change in size or multisampling during the execution.

As for the render buffers I have the following associated with my processing sketch:

1. GL_DEPTH24_STENCIL_8 (1440x860 pixels)
2. GL_DEPTH24_STENCIL_8 (1440x860 pixels)
3. GL_RGBA8 (1440x860 pixels @4samples)
4. GL_DEPTH24_STENCIL_8 (1440x860 pixels @4samples)
5. GL_DEPTH_COMPONENT24

It seems like 3 and 4 are the buffers actually used, the others contain garbage and do not change.

Too many allocated textures

There also seem to be too many textures used by the main renderer.
There are four textures containing the (same) rendered image at all times (GL_TEXTURE_2D) and eight textures with no visible content (maybe it's just Apple's OpenGL Profiler), that also have the size of the sketch.

I hope that those hints are somehow useful for eliminating some performance problems in Processing.

Best,

Benjamin

[allesblinkt]

I believe that this might be somehow connected to this issue: #1089
Even big empty sketches (3840 x 1080 pixels) are eating away at least 512 Megabytes of GPU memory. So, getting a little bit bigger than that fills up my GPU memory and the GPU starts swapping back and forth textures (in this case even some of the render textures)

[codeanticode]

Hello, many thanks for your detailed report. I agree with your belief that this issue and #1089 are related. In fact, this other one #1714 may be related as well. Unfortunately, I think the large number of textures and renderbuffers created by a single running Processing program is not due to a bug in the Processing renderer, but to a more fundamental issue.

To begin with, all these problems affecting OpenGL rendering in OSX apparently originate in a series of low-level changes that Apple/Oracle made to Java, which particularly affects how a GL surface can be embedded inside an Applet object. You can find a detailed discussion of this issue in the LWJGL forum: http://lwjgl.org/forum/index.php/topic,4711.105.html.

JOGL was also affected by these changes in OSX/Java, and Sven Gothel (JOGL's main developer: https://github.com/sgothel), did a lot of work to make sure that JOGL works under different combinations of OSX and Java versions.

Eventually, recent releases of both LWJGL and JOGL now work with the latest updates of Java 1.6 and 1.7, but it seems that performance and resource usage won't be the same as before due to the workarounds required to have the CALayers rendering properly into the Applet surface.

So I wrote a number of test cases that show different usage scenarios, from rendering into a native window w/out any intermediate Applet object, to a full Applet-inside-Frame structure that emulates Processing's architecture, using both JOGL and LWJGL.

Following your report, I checked the resources used in each instance, and found some consistent patterns:

1. LWJGL, native window, no Applet: https://github.com/codeanticode/lwjgl-tests/blob/master/src/test/simple/ShaderTest.java
   No Textures, FBOs or Renderbuffers are being created, only two shader objects and one shader program that correspond to the shader is being set in the code. Performance is ok.

2. JOGL, native window, no Applet: https://github.com/codeanticode/jogl2-tests/blob/master/src/test/animator/AnimatorNEWT.java
   Same resources and performance as in 1)

3. LWJGL, running as an Applet: https://github.com/codeanticode/lwjgl-tests/blob/master/src/test/simple/AppletTest.java
   Several textures, FBOs and Renderbuffers are created, all with the same resolution. Only one rendebuffer contains the valid rendering output, the other appear to just contain garbage:

The shader resources are also interesting, because some additional shaders that are not loaded in the program appear to be running behind the scenes. This one seems some sort of motion blur effect (?!):

BTW, this test case doesn't run smoothly at all on my system when compared with 1), not sure if it is due to the additional textures/shaders or something else, I discussed this on the LWJGL forum: http://lwjgl.org/forum/index.php/topic,4995.0.html

4. JOGL, running with an Applet embedded inside a Frame (Processing-style): https://github.com/codeanticode/jogl2-tests/blob/master/src/test/applet/AppletInvoke.java
   Same as 3), several textures, FBOs and Renderbuffers, a few more than with LWJGL, maybe this is due to a different implementation of the CALayer workarounds (?). The weird "motion blur" shader also shows up, in addition to the shader resources created in the program:

This test program has an additional issue that consists in the screen not being updated properly (which I reported already in this bug entry in the JOGL tracker: https://jogamp.org/bugzilla/show_bug.cgi?id=735), however it doesn't affect the point of the argument being the increased resource usage.

What I conclude from all these tests is that the workarounds that both LWJGL and JOGL use in order to accomodate the recent changes in Java result in a substantially increased GL resource utilization, as well as stuttering in programs that are GPU-bound (#1714). The resource utilization becomes a serious problem for sketches with large screen sizes (#1089). I believe that both of these problems cannot be fixed from Processing, since they are the consequence of architectural limitations imposed on the OSX implementation of Java.

As a final note, I run the tests on OSX 10.8.3, and the results are consistent between Java 1.6 and 1.7. Sorry for the long post, but these are complex issues that I'd like to keep documented in detail because I have been trying to fix for quite a while now.

[allesblinkt]

Hello, you are very welcome. I was digging into the code and profiling, because I was releasing some application for a client and something did not feel quite right (substandard performance and high GPU memory usage). A very big part was the font texture issue ( #1775 ), which seems to be good now.

No need to excuse for the long post. I find this very interesting because until now, I did not really care or think about how the actual drawing surfaces were created, because I used frameworks like Processing to handle that cross-platform work for me.

Also, creating those test-cases makes a lot of sense in terms of isolating the cause of those issues. They all yield about the same results on my machine. I agree that most of the issues actually originate from the way the surface / window is now handled on Mac OS / Java.

What I will try next is to run and profile the test cases on Linux (and possibly Windows 7) to see what is going on there. GPU memory usage seems to be out of proportion on Linux also. Either it is also a JOGL problem or the problem is also in Processing. I'll keep you posted on that.

About that motion blur shader... It is really part of the Mac OS CoreAnimation layer. Its source-code is embedded in the QuartzCore framework:

$ strings /System/Library/Frameworks/QuartzCore.framework/QuartzCore | grep MOTION_BLUR_SAMPLES                
const int samples = MOTION_BLUR_SAMPLES;
CA_MOTION_BLUR_SAMPLES
#define MOTION_BLUR_SAMPLES %d

[codeanticode]

Hello Benjamin, thanks a lot for your help! Note that in commit 2acb28f I introduced some changes to reduce the creation of GL resources, so you should be able to see lower memory usage on the three platforms. On OSX, it would probably be still higher than on Linux and Windows, but at least lower than in 2.0b9. In relation to this issue, I think this thread on the jogl forum might be of interest: http://forum.jogamp.org/Increased-resource-utilization-with-Java-7-OSX-tc4029227.html

Please let me know if you find anything else while testing and debugging.

Quick question: what GL debugging tool are you using on Linux?

[allesblinkt]

Hello,
thanks a lot for handling this issue. I just tried the new code, seems better and working without any problems so far.
Under Linux I tried using gDEBugger, which did not perform well on my 64 bit Xubuntu machine, now I am using bugle's gldb-gui. Works, but does not look so nice. Although I recently upgraded my Linux box and could not compile it yesterday. Some weird issue with GLEW...

[codeanticode]

quick update: I was able to reduce the resource usage on OSX a bit more with commit 50d4dc9, a typical sketch that doesn't create any offscreen surfaces should use 5 textures (one texture2D, and the rest texture rectangles).

[codeanticode]

I'm closing this issue as we don't have control on how these additional GL resources are created. With the latest improvements, I believe we reduced resource usage to the minimum that is possible from Processing

[allesblinkt]

I think so too. Thank you very much for looking into the problem.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.