Massive Memory Leak in Multitileset Scenes

[Reag]

Background

We built a Unity scene where several tilesets can be loaded and unloaded as needed. Immediately, we noticed that a large memory-leak was occurring, ballooning our default memory footprint of 2-3 gig to over 10gig of ram. After a rather extensive testing process, I was able to reproduce the bug in the cesium samples. Below is a report on what I have found, what I suspect, and what knowledge I lack to further investigate this problem.

Describe the bug

Whenever more than one Cesium3DTileset.cs is running on a machine (Be it in the same scene, different editor windows, or even different clients), all but one of these scripts will begin to leak untracked ram. This ram appears to be allocated by the C++ engine or Reinterop, as Unity's memory snapshots cannot identify it beyond being marked as 'private'.

To Reproduce

Experiment 1

Open the VR01_CesiumDenver sample. Copy the CesiumGeoreference GameObject several times. Attach the script in Appendix 1 to one of the objects. Build the sample and activate the AutoReload GUI Button. Watch the memory clime.

Experiment 2

Open the VR01_CesiumDenver sample. Attach the script in Appendix 1 to the tileset. Build the sample and run several instances of it. Select one of the ran clients thats not the first or last one to open. Activate the AutoReload GUI Button and watch the memory explode.

Experiment 3

This experiment was conducted on our product, and may not be fully relevant. We loaded our main content scene, containing about 15 small tilesets over a geographic area of around 20 km. We then adjusted the Maximum Screen Space Error of one of the tilesets and watched our memory usage clime.

Diagnostic Notes

Two tilesets running at the same time didn't appear to be enough to cause the system to enter into the memory leaking state, some of the time. Unfortunately, I could not isolate what exactly you would have to do to cause the issue. I did note that if you where in the memory leaking state, the cesium-requrest-cache.sqlite-wal (write ahead log) would start to be written too. We did play around with the CesiumRuntimeSettings, setting the Requests Per Cache Prune and Max Items to both very large and very small numbers. This seemed to have an effect, but not one I could isolate in a useful way.

Screenshots/Video

Appendix 1

Linked Here is the script we used for testing.

Other Technical Notes

We ran the development samples for this, where we followed the Developer Setup Guide exactly. This means that we compiled Reinterop~, Native~, and built our own DLLs. It was certainly an interesting process.

I suspect the tilesets are fighting for control of the SqlLite database, leading to issues and cached requests. Its possible that this is causing the requests to be 'lost' in application's memory, thus leaking. I unfortunately do not have the C++ skills to do a detailed analysis of the native code, so this remains speculation.

In addition, I'm happy to provide the memory snapshots and profiler data we took as part of our testing protocol. If you feel they, or any other technical information, would be helpful, please contact me at gilado@oversight-ar.com.

[csciguy8]

Thanks for the detailed report @Reag .

I'll link some related issues to this
CesiumGS/cesium-unreal#1325
CesiumGS/cesium-unreal#1362