Memory footprint of images created with an ImageDataProvider

[ptziegler]

This is a topic that came up here with regards to the image data that is cached by an Image, in case an ImageDataProvider is used.

For example, take the following snippet:

Snippet

package gef;

import java.util.ArrayList;
import java.util.List;

import org.eclipse.swt.graphics.GC;
import org.eclipse.swt.graphics.Image;
import org.eclipse.swt.graphics.ImageData;
import org.eclipse.swt.graphics.ImageDataProvider;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;

public class DrawGC {

	public static void main(String[] args) {
		Shell shell = new Shell();
		shell.setText("DrawGC");
		Display display = shell.getDisplay();

		int width = 310;
		int height = 310;

		List<Image> images = new ArrayList<>();
		for (int i = 0; i < 2000; ++i) {
			ImageDataProvider imageDataProvider = zoom -> {
				if (zoom == 100) {
					return createImageData(display, width, height);
				}
				return null;
			};
			Image image = new Image(display, imageDataProvider);
//			Image image = new Image(display, width, height);
//			GC gc = new GC(image);
//			gc.dispose();
			images.add(image);
		}

		shell.addPaintListener(event -> {
			for (Image image : images) {
				event.gc.drawImage(image, 0, 0, width, height, 0, 0, 100, 100);
			}
		});

		shell.open();

		while (!shell.isDisposed()) {
			if (!display.readAndDispatch()) {
				display.sleep();
			}
		}

		images.forEach(Image::dispose);
	}
	
	private static ImageData createImageData(Display display, int width, int height) {
		Image image = new Image(display, width, height);
		try {
			return image.getImageData(100);
		} finally {
			image.dispose();
		}
	}
}

This example introduces 550MB of image data which is stored on the heap.

...which is used almost exclusively held by the cachedImageData map.

The entire process takes up 1.6GB of memory.

When I create the image without ImageDataProvider, the Java heap is nonexisent and the process only needs 1.2GB of memory. The underlying "problem" is that this cache is never cleared. So images with a long lifecycle now require (at least) 50% more memory, and will stay like that until eventually they are disposed. I also think the image data is redundant, because the same data is already stored in the native widget.

This overhead primarily exists on Windows, which I assume was done as part of the HiDPI support, in order to avoid the performance impact when the monitor zoom has changed and thus the images need to be updated. But perhaps there's a better way handle this?

Maybe it would already help if:

• the cache only stores the image data for "small" images? For example everything up to 16x16, which have a negligible memory cost.
• the cache stores the image data as soft references, to better work on memory-starved systems.

This might already be improved with #3429 but you'd still get the issue if the provider supports multiple zoom levels.

[Phillipus]

Thanks for opening this discussion as this has big impact on our RCP app on Windows when using many images.

@HeikoKlare Is this something that could be improved?

[HeikoKlare]

I don't see an easy way to improve this. As mentioned above, soft references may resolve issues on systems running out-of memory, but at a more foundational level the only reasonable option is to not cache data but reload it when required (which is the common trade-off between memory and performance).

So images with a long lifecycle now require (at least) 50% more memory, and will stay like that until eventually they are disposed. I also think the image data is redundant, because the same data is already stored in the native widget.

This assessment is not correct or at least not fully clear to me. What is meant by "the same data is already stored in the native widget"? Also, what's the baseline for the 50% more memory? The memory for the 100% image data was always consumed, just that it used to be in the handle (OS memory) before lazy image handle creation was implemented. Now it shows up as Java heap memory consumption until you create an actual 100% handle. I.e., if you called Image.win32_getHandle(image, 100) once, the cached data will disappear and be transformed into an actual handle (which should then be the same state as we had in SWT for the last decades).
At least that's how I think it is supposed to behave. Maybe there is some bug in it though 🙂

This overhead primarily exists on Windows, which I assume was done as part of the HiDPI support, in order to avoid the performance impact when the monitor zoom has changed and thus the images need to be updated. But perhaps there's a better way handle this?

The main purpose was to avoid that images need to be loaded twice from storage, once at image creation time (for metadata initialization) and once for the actual handle creation later on. This was introduced with lazy image handle creation (used for HiDPI support but not mandatory for it). As long as handles were created once the image was created, this was not necessary.

[ptziegler]

Also, what's the baseline for the 50% more memory?

This is based on some simple tests and on what @Phillipus reported is his product.

What is meant by "the same data is already stored in the native widget"?

This pretty much relates to what you explained below. The image data seems to exists both in the cache (and therefore on the Java heap), but also in a native resource. In other words: It's stored twice in memory.

I.e., if you called Image.win32_getHandle(image, 100) once, the cached data will disappear and be transformed into an actual handle (which should then be the same state as we had in SWT for the last decades).

Which I don't think is happening. This is also the reason why I specifically looked at the memory consumed by the process, not just the Java heap. Otherwise I can't explain why there's such a big difference between using an ImageDataProvider and painting on the image directly

if you called Image.win32_getHandle(image, 100) once, the cached data will disappear

This method specifically is never called in the snippet above. I even set a breakpoint in this method, but it's never hit. Instead, I only get this call stack:

[HeikoKlare]

Thank you for the additional input. I have some questions for clarification and an enhancement of #3429 that may (at least to a certain degree) address the memory consumption issue.

Also, what's the baseline for the 50% more memory?

This is based on some simple tests and on what @Phillipus reported is his product.

I actually meant what state/release of Eclipse is used as baseline to see that difference. I guess it's some release before lazy resource loading was introduced, but better we know that than guessing in the wrong direction.

What is meant by "the same data is already stored in the native widget"?

This pretty much relates to what you explained below. The image data seems to exists both in the cache (and therefore on the Java heap), but also in a native resource. In other words: It's stored twice in memory.

So if I understand correctly it is about a native handle being created, right? Because I do not see any widget being involved that might store the image (such as a tool item storing its image). Sorry for being nitpicky, just want to be sure that I don't miss something when trying to understand the case.

I.e., if you called Image.win32_getHandle(image, 100) once, the cached data will disappear and be transformed into an actual handle (which should then be the same state as we had in SWT for the last decades).

Which I don't think is happening. This is also the reason why I specifically looked at the memory consumed by the process, not just the Java heap. Otherwise I can't explain why there's such a big difference between using an ImageDataProvider and painting on the image directly

I've just done that and see memory consumption for the process dropping from 2,8GB to 2,4GB on my system. That's the saving you gain from converting the cached image data into a native handle. When applying #3429, the memory consumption of the process even drops to 1,8GB because then the persistent 100% handle is used. I have just adapted that PR so that even without the win32_getHandle(image, 100) call a persistent native handle for 100% is created from the cached data once requested (even via the GC.drawImage() use case) and also used instead of a temporary handle being created. That should bring us back to the original memory consumption (for the 100% use case).

if you called Image.win32_getHandle(image, 100) once, the cached data will disappear

This method specifically is never called in the snippet above.

That's correct and completely as intended.