Memory is out (Windows)

[denisgl7]

It's hard for me to accurately describe the problem, but it is!
It comes with new versions of the Air SDK 28, 29
This issue is definitely not in the Air SDK 26
I created this topic on the forum. Then I shut down the particle systems.
https://forum.starling-framework.org/topic/starling-context-restored-and-loss-of-textures
The problem is most likely the use of filters and it is played on the Windows platform
After applying filters(ColorMatrix, glow, dropshadow), the application runs for 15-20 minutes, and then crashes error 'memory is out' and what happens in the video
https://youtu.be/sUspvO4tIDg
Without filters such problem did not find
With the air sdk 26, everything is in order!
Tested scout. No increase of memory, no other problems I have not discovered!

[PrimaryFeather]

Thanks a lot for the report, Denis!

Hm, that it's happening only in new AIR versions suspiciously sounds like an AIR issue, then. Hm ...

When Starling starts, it traces the following String: [Starling] Context ready. Display Driver: ...
Could you tell me what's the output of your app with AIR 26 vs 29?

[denisgl7]

AIR 26 - [Starling] Context ready. Display Driver: DirectX9 (Standard)
AIR 29 - [Starling] Context ready. Display Driver: DirectX11 (Standard)
Is there anything else I can do?

[PrimaryFeather]

Thanks! Just as I expected — it seems to be a bug in the DirectX 11 implementation.
Now please try what happens when you use the following compiler argument (using AIR 29): -swf-version=37

Does that change anything? What's the trace output then?

[denisgl7]

I realized the application will work in DirectX 9. By the evening there will be results, Daniel!

[denisgl7]

Yes, I do not find problems when using DirectX9

[PrimaryFeather]

Okay, that's great news! At least you have a simple workaround then.

Still: it would be fantastic if we could figure out how to reproduce this problem in a much simpler program — because then we could give Adobe a chance to fix this problem in the DirectX 11 implementation.

Could you maybe try your luck when you've got a little time? Perhaps try a simple demo that displays an image and adds a filter, then disposes both and does the same, every second or so. Or whatever you think triggered the error in your code. That would be a huge help!

[denisgl7]

I'll try Daniel, but I can not immediately find the problem a simple project!

[PrimaryFeather]

Yeah, I know it's not going to be easy! It might be a combination of settings that's hard to pinpoint. 😟

[denisgl7]

In the previous 31 beta, it seemed to me that the problem was solved
And in the new version 31, the problem reappeared. But now I have no previous sdk to check. Where can I get it? Because under this link https://helpx.adobe.com/air/kb/archived-air-sdk-version.html intermediate versions are not published

[PrimaryFeather]

Hm, I don't know if there's an archive of all the intermediate versions — sorry!

So you say with the previous beta, the memory issue was apparently gone, even with DirectX 11?

[denisgl7]

Yes, this problem was not! But after the publication of the latest version again appeared. But I deleted the previous version

[hardcoremore]

Still: it would be fantastic if we could figure out how to reproduce this problem in a much simpler program — because then we could give Adobe a chance to fix this problem in the DirectX 11 implementation.

Hi Daniel,

I have created a simple test that shows memory leaks when using filters. Denis has redirected me here from Starling forum. Here is the post:

https://forum.starling-framework.org/d/21763-gpu-memory-leak-when-using-starling-filters-on-windows/12

[PrimaryFeather]

Thanks for creating the sample, Denis! Yes, we could use that sample to report the issue again - maybe someone from Harman can take a look. Does anyone know if there's a new issue tracker planned? Or are we supposed to still use the old one? As I'm currently on a vacation, I haven't been able to follow all the news lately. 😉

[hardcoremore]

@ajwfrost Hi Andrew, is there any hope for this to be fixed?

Thanks,

Caslav

[paulp-games]

We have the same bug in win 10(directx 11).

And we cant use compiler argument -swf-version=37 because we need x64 application.

@ajwfrost really looking forward to fixing this bug. Or switch between directx versions in air33.
Thanks.

[denisgl7]

I'm afraid that adobe will not fix this problem
But this is probably the biggest problem on the Windows platform.
All hope for harman

[paulp-games]

Good news, Harman answered me

FYI this is one that we’re aware of and are hoping to address in one of our Windows releases, it’s in our backlog so we’ll be reviewing it internally.

[ajwfrost]

@hardcoremore just looking at your sample app.. if you run this on your Windows platform with AIR 32, do you see an increase in GPU memory happening, as reported by Starling? I've seen that this is only with Direct3D 11 but wondering if it's also specific to a version of Windows or to a build of AIR i.e. 32-bit vs 64-bit.

One of our guys had built the demo app and ran it with the standard 32-bit AIR via ADL, but isn't seeing an increase in memory. Starling output is:
[Starling] Context ready. Display Driver: DirectX11 (Enhanced)

So we may need more details for reproducing this (or there's always the possibility that this is actually a DirectX issue that's been resolved by MSFT, I think the machine used was very up to date with Windows 10 feature releases etc.)

[hardcoremore]

Hi Andrew,

For me this bug is still present. Starling display stats is showing GPU memory usage 120MB while Task Manager for GPU in use shoes usage of 2GB which is how much my GPU has.

I am using this code to test:

package dfpck
{
	import flash.utils.setInterval;
	
	import starling.core.Starling;
	import starling.display.Image;
	import starling.display.Sprite;
	import starling.filters.ColorMatrixFilter;

	
	public class FilterMemoryLeak extends Sprite
	{
		public static const NUMBER_OF_IMAGES:int = 200;

		protected var _atlas:StarlingAssets;
		protected var _filterImages:Vector.<Image>;
		protected var _container:Sprite;
		
		public function FilterMemoryLeak()
		{
			super();
			
			_filterImages = new Vector.<Image>;
		}
		
		public function get atlas():StarlingAssets
		{
			return _atlas;
		}
		
		public function set atlas(value:StarlingAssets):void
		{
			_atlas = value;
		}
		
		public function startExample():void
		{
			_container = new Sprite();
			
			addChild(_container);

			createFilterImages();

			setInterval(this.createFilterImages, 500);				
		}
		
		public function randomNumberRange( maxNum:Number, minNum:Number, floor:Boolean = true):Number
		{
			var randomNumber:Number = 0;
			
			if(floor)
			{
				randomNumber = Math.random() * (maxNum - minNum + 1);
				randomNumber = Math.floor(randomNumber);
			}
			else
			{
				randomNumber = Math.random() * (maxNum - minNum);
			}
			
			return  randomNumber + minNum;
		}
		
		public function createFilterImages():void
		{
			var filterImage:Image;
			var filter:ColorMatrixFilter;
			
			while(_filterImages.length > 0)
			{
				filterImage = _filterImages[0];
				
				if(filterImage && filterImage.filter)
				{
					filterImage.filter.clearCache();
					filterImage.filter.dispose();
				}

				_container.removeChild(filterImage, true);

				_filterImages.shift();
			}

			for(var i:int = 0; i < NUMBER_OF_IMAGES; i++)
			{
				filterImage = new Image(atlas.atlas.getTexture('ball_pool_1_full'));
				filterImage.scale = 0.17;
				filterImage.alignPivot();
				filterImage.readjustSize();
				filterImage.name = i.toString();
				
				filterImage.x = randomNumberRange(Starling.current.stage.width - 100, 0, false);
				filterImage.y = randomNumberRange(Starling.current.stage.height - 100, 0, false);
				
				filter = new ColorMatrixFilter();
				filter.tint(0xFF0000, 1);
				
				filterImage.filter = filter;
				filterImage.filter.cache();
				
				_container.addChild(filterImage);
				
				_filterImages.push(filterImage);
			}
		}
	}
}

I am using:

Windows 10 Pro
Vresion 1903
OS build 18362.476

With profile:

[Starling] Context ready. Display Driver: DirectX11 (Enhanced)

GPU Details:

NVIDIA System Information report created on: 12/05/2019 20:36:51
System name: CASLAV-PC

[Display]
Operating System: Windows 10 Pro, 64-bit
DirectX version: 12.0
GPU processor: GeForce GTX 770
Driver version: 432.00
Driver Type: Standard
Direct3D API version: 12
Direct3D feature level: 11_0
CUDA Cores: 1536
Core clock: 1084 MHz
Memory data rate: 7.01 Gbps
Memory interface: 256-bit
Memory bandwidth: 224.32 GB/s
Total available graphics memory: 9959 MB
Dedicated video memory: 2048 MB GDDR5
System video memory: 0 MB
Shared system memory: 7911 MB
Video BIOS version: 80.04.C3.00.21
IRQ: Not used
Bus: PCI Express x16 Gen3
Device Id: 10DE 1184 28251462
Part Number: 2005 0000

[Components]

nvui.dll 8.17.14.3200 NVIDIA User Experience Driver Component
nvxdplcy.dll 8.17.14.3200 NVIDIA User Experience Driver Component
nvxdbat.dll 8.17.14.3200 NVIDIA User Experience Driver Component
nvxdapix.dll 8.17.14.3200 NVIDIA User Experience Driver Component
nvCplUIR.dll 8.1.940.0 NVIDIA Control Panel
nvCplUI.exe 8.1.940.0 NVIDIA Control Panel
nvWSSR.dll 26.21.14.3200 NVIDIA Workstation Server
nvWSS.dll 26.21.14.3200 NVIDIA Workstation Server
nvViTvSR.dll 26.21.14.3200 NVIDIA Video Server
nvViTvS.dll 26.21.14.3200 NVIDIA Video Server
nvLicensingS.dll 6.14.14.3200 NVIDIA Licensing Server
nvDispSR.dll 26.21.14.3200 NVIDIA Display Server
NVMCTRAY.DLL 26.21.14.3200 NVIDIA Media Center Library
nvDispS.dll 26.21.14.3200 NVIDIA Display Server
nvDevToolS.dll 26.21.14.3200 NVIDIA 3D Settings Server
PhysX 09.19.0218 NVIDIA PhysX
NVCUDA.DLL 26.21.14.3200 NVIDIA CUDA 10.1.120 driver
nvGameSR.dll 26.21.14.3200 NVIDIA 3D Settings Server
nvGameS.dll 26.21.14.3200 NVIDIA 3D Settings Server

I have tested this with latest Adobe AIR SDK not Harmans. Did something changed with Harman's SDK in this area, I will re test with Harman's SDK as well

[denisgl7]

Adobe Air 33.0.2.315 has the same problem

[ajwfrost]

Thanks for confirming this - I see the confusion here.. we'd been looking at the stats from Starling and from Scout, but the problem is that these are wrong i.e. we can only see the memory increase if we look at the stats from Task Manager.

Reproduced this with the latest build we have.. we'll get onto it and see where the discrepancy is...

Thanks!

[raresn]

Hey guys, Any luck with a workaround for this?
Our app doesn't start (Starling) - the graphic driver is Intel Q45/Q43 Express Chipset (Microsoft Corporation WDMM 1.1)
If we install an older version of the driver, or uninstall it and go with the windows default, the app starts. The problem si that it doesn't have the right resolution for 16:9 screens and it is really a problem for a lot of our clients.
Testing this with Air 32, latest Starling & Feathers.

[ajwfrost]

Our initial investigations didn’t find an obvious memory leak in the D3D11 usage, but that was focussing on textures so we’ll also need to look at the vertex buffers.

But the problem you describe above sounds different, if it’s just not starting at all on a particular GPU/driver. Do you get any errors out from Stage3D if you enable the debug mode? As a workaround, if you can use the Stage3D APIs to query the driver and find it’s not going to work, you could request a software profile instead?

Thanks

[raresn]

First of all, thank you for the quick response. Thank you for the suggestion.
Kind of a noob question i guess, but how do i request a software profile?
I will come back with additional information on this specific error.

[ajwfrost]

In the call to stage3D.requestContext3D()
https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/Stage3D.html#requestContext3D()
RenderMode can be "software":
https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display3D/Context3DRenderMode.html

If using Starling, you can add the render mode to the constructor, 'auto' is the default so switch this to 'software':
https://doc.starling-framework.org/v2.0/starling/core/Starling.html#Starling()

I guess you'd need to create an instance using hardware first so that you can detect what the driver info is.. so it depends a bit really on what the problem is when you say that the app doesn't start! You need to be able to start your app to be able to get to the point where you choose between auto and software rendering..

[raresn]

Really good info! Can't wait to try it on monday when we get to work.

[raresn]

Didn't mean to hijack this thread, but i do want to really thank you for the suggestion to switch to software - it worked and it was a really big deal for our business!
Still strugling to build the switch between hardware and software, but it is really great! Thank you.

[paulp-games]

Is there any hope for this to be fixed?
It's critical bug for our game

[ajwfrost]

@paulp-games We did some initial investigations on this one a while ago and couldn't spot anything obvious unfortunately - we focused on textures at that point. But it's on our list to look at again in our next batch of tasks so we will look more into the other memory buffers and try to track down what's happening.

[ajwfrost]

@paulp-games FYI we've just done a fix for a render-to-texture GPU memory leak and have tried the "FilterMemoryLeak" test case from above: this no longer has the GPU leak in it when we use our RTT fix. So these were the same root cause (#20) and will be fixed in our next release.. apologies it's taken so long!

thanks

[Adolio]

I would suggest to close this issue. At least on my side this issue has been fixed for a while.

@denisgl7 @paulp-games @ajwfrost What do you think?