Spout can't handle more than a certain number of Senders

[Stefano-Rascio]

Hi Lynn,

I tested Spout 2.006 on different machines and I found out that there is a limit related to the number of Senders that can be created before the SDK crashes.
After an extensive investigation I found out that the same limit applies in the case where Senders are created and destroyed progressively too (for instance: Sender1 created, Sender1 destroyed; Sender2 created, Sender2 destroyed; etc...).
The limit is the same in both cases and it's 156 on my machine.
In all these cases an exception is thrown inside the nvoglv64.dll because of an access violation reading location 0x000000000000000C (n.b.: this is only visible attaching the debugger to the processes). Moreover when a Receiver is created, after the limit has been reached, the following message is shown by the SDK:
Cannot create DirectX/OpenGL interop.
CreateDX11interop - error 1
CreateInterop - error 3

I've created two Openframeworks projects (open close senders and open n senders) which use the ofxXmlSettings and ofxSpout2 (only used to include the SpoutSDK) add-ons, to show the problem. There is an xml file in the bin/data folder through which is possible to set:

• the amount of sender that will be created and destroyed one after another (this is the case of the first project);
• the amount of sender that will be created (this is the case of the second project).

Thanks,
Stefano

[leadedge]

Thanks for your investigations Stefano.

This issue is related to the previous one where the maximum number of senders is set to 10. If a further sender is created, the shared memory of the previous sender is over-written. SpoutPanel was revised for that issue and since then I have assembled an update installation which allows the maximum number of senders to be changed in the settings program. Could you try this with your test programs?

[Stefano-Rascio]

The number of senders on my machine was already set to 250 but I've never been able to reach it (even closing and opening channels progressively). I tried the updated installation too but nothing changed, I got exactly the same issue.

[leadedge]

I set up a test program and I believe it is simply running out of graphics memory. DX11 reports failure to create a shared texture. DX9 does not for some reason but in any case the interop link fails. On my machine with GTX 640, for textures of 1920x1080 the maximum number of senders is 77. I can't fault create/release of senders.

[Stefano-Rascio]

According to my tests it's not running out of memory, I tried creating the maximum number of senders that my machine can handle and less than 20% of the gpu memory is used, moreover even creating and destroying a sender at a time in a for loop is giving the same outcome and this shouldn't be the case where is going out of memory since that at the end of each iteration there are no senders around.
Furthermore it doesn't seem an issue which is related to the size of a texture since that I tried textures of diverse sizes and I always get the same behaviour.

[leadedge]

I am looking into it. There is definitely something strange going on. Here is what I have found so far.

There is certainly plenty of VRAM left when it fails - monitored with GPU-Z - but spoutDirectX::CreateSharedDX11Texture definitely fails with "E_OUTOFMEMORY".

I revise the number I mentioned previously. I get 123 as the maximum if the texture size is less than or equal to 1920x1080. Any less gives the same number, but the number of senders reduces above 1920 x 1153 (122), down to 42 at 3840x2160, so it is definitely related to texture size.

I have worked more on creating/releasing senders. Each is released OK. I have checked the resource releases and all report a refcount of zero so I assume they are indeed freed. However I can reproduce what you see. Releasing a sender does not affect the maximum achieved, so I don't know why the memory is not released. The next thing I would try is to delete the sender object, but have not had time to do it.

There is a compatibility check- spoutGLDXinterop::GLDXready() - that is called at the very start. This creates a shared texture for the test and it might fail at this point depending on the texture size. If it does it will switch to CPU texture share mode, so you have to go back into settings to restore texture share mode for the next test. Otherwise it creates an additional staging texture and the number of senders is reduced further.

The fact that spoutDirectX::CreateSharedDX11Texture definitely fails with "E_OUTOFMEMORY" is the key.

It could be because it is a shared texture, or because previous textures are linked with the GL/DX interop, or it could be due to how the driver manages GPU memory and makes available a limited space for resources. At this stage I don't know.

There are OpenGL resources created as well, textures and an fbo, which will consume GPU memory. Leave the "Buffer" option OFF or two pbos are also created.

I will try to isolate this but will need to make some test code and it might take a while.

[leadedge]

I have found that the problem arises only for a DirectX shared texture. If the texture is not shared you can create a large number of them (I have not tested the limit). I tested by just creating a texture and no sender, so the limit is with texture creation only.

It doesn't matter how small the texture is. However increasing texture size gives the same limit until they reach a certain point and then the number reduces. It could also be related to the aspect ratio but I have not tested that completely.

For example I could create 31 large textures of 16384x16256. But the interesting thing is that I can open more applications that create such a texture. I stopped at 40. Each one took longer and longer to initialize until it was virtually useless.

So I tried opening multiple copies of the Spout demo sender. They all open OK, but after about 20 of them they all start to slow down. I guess it depends on the system, but any more is not useable for me.

I believe it could be due to the interop lock. The specs say "The Lock/Unlock calls serve as synchronization points between OpenGL and DirectX. They ensure that any rendering operations that affect the resource on one driver are complete before the other driver takes ownership of it."

I have also looked into the sender release problem and again believe it is due to shared textures. Each shared texture has a share handle generated. For DX9 this is done by the call to CreateTexture. For DX11, GetSharedHandle is used to retrieve it. The share handle retrieved by GetSharedHandle can't be released using CloseHandle (https://msdn.microsoft.com/en-us/library/windows/desktop/bb174562(v=vs.85).aspx). Although I can't find any evidence for it, I am guessing that the texture may not be released from memory because this share handle exists even though it may not be accessed.

It is possible to use an alternative "CreateSharedHandle" where you can specify a flag to create a handle that can be closed. However, this means changes to the compiler environment as well as other code changes and would take some time to set up even to test whether this would resolve it. I will do this eventually but can't at the present time.

So meanwhile there is a limit to the number of senders that can be created in the one application due to shared texture limits, most likely imposed by the driver. Although there does not seem to be a limit on the number of sender applications that can be opened together, there is a practical limit depending on the performance of the system.

If you have any further ideas I would be pleased to hear. Also if you can give the update a run through I will make that the default download.

[leadedge]

Update has been available for some time, so closing this issue