Disposing and recreating NativeEngine has big leaks when loading a GLB

[ryantrem]

When just loading the cube, there is no noticeable leak when disposing and recreating NativeEngine (though this might be due to it being tiny to start with). However, when loading a glb, there is a noticeable memory leak. For example, if we load https://raw.githubusercontent.com/microsoft/experimental-pcf-control-assets/master/surface.glb in the Babylon React Native Playground app, then each time we tap "Toggle EngineScreen" we leak about 30mb. I observed this in an iOS Release build (haven't tested debug or Android yet).

I also noticed a similar issue with XR. Even with just the cube, if I enter XR, and then hit "Toggle EngineScreen" (rather than stopping XR), it leaks around 120mb. I'm guessing this means we are not freeing up certain native resources when the JS NativeEngine is disposed. If the issue causing the leak for glbs does not turn out to be the same root cause as the leak for XR mode, then we should open a separate issue.

[CedricGuillemet]

I'm taking quick a look to see if this repro with babylon native playground.

[CedricGuillemet]

I can see 6 leaked vertex buffers from bgfx when disposing the engine with that .glb.
I don't think this can sum up to 30mb. I'm investigating with BN React Playground ...

[ryantrem]

With a large model with lots of verts (this one for example: https://raw.githubusercontent.com/RonnieAshlock/TestAssets/master/GLB/CherryPicker.glb), I still see some large leaks that look like they are all vertex buffers. I see allocation stacks like this:

   0 libsystem_malloc.dylib malloc
   1 libglInterpose.dylib 0x103c015ff
   2 Playground Babylon::VertexBufferData::VertexBufferData(Napi::TypedArrayOf<unsigned char> const&, bool)
   3 Playground Babylon::NativeEngine::CreateVertexBuffer(Napi::CallbackInfo const&)
   4 Playground Napi::ObjectWrap<Babylon::NativeEngine>::DefineClass(napi_env__*, char const*, unsigned long, Napi::ObjectWrap<Babylon::NativeEngine>::PropertyDescriptor const*, void*)::'lambda1'(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long) const
   5 Playground std::__1::__function::__value_func<facebook::jsi::Value (facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)>::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*&&, unsigned long&&) const /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1873
   6 Playground std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)>::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long) const /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:2548
   7 Playground facebook::jsc::JSCRuntime::createFunctionFromHostFunction(facebook::jsi::PropNameID const&, unsigned int, std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)>)::HostFunctionMetadata::call(OpaqueJSContext const*, OpaqueJSValue*, OpaqueJSValue*, unsigned long, OpaqueJSValue const* const*, OpaqueJSValue const**) /Users/ryantremblay/Repos/BabylonReactNative2/Apps/Playground/node_modules/react-native/ReactCommon/jsi/JSCRuntime.cpp:1151
   8 JavaScriptCore JSC::JSCallbackObject<JSC::JSNonFinalObject>::call(JSC::JSGlobalObject*, JSC::CallFrame*)
   9 JavaScriptCore JSC::LLInt::setUpCall(JSC::CallFrame*, JSC::CodeSpecializationKind, JSC::JSValue, JSC::LLIntCallLinkInfo*)

And also like this:

   0 IOAccelerator IOAccelResourceCreate
   1 Metal -[MTLIOAccelResource initWithDevice:options:args:argsSize:]
   2 Metal -[MTLIOAccelBuffer initWithDevice:pointer:length:options:sysMemSize:vidMemSize:args:argsSize:deallocator:]
   3 AGXMetalA11 0x1be1da85c
   4 AGXMetalA11 0x1be228e8c
   5 libMTLCapture.dylib 0x103d1ec83
   6 Playground bgfx::mtl::RendererContextMtl::createVertexBuffer(bgfx::VertexBufferHandle, bgfx::Memory const*, bgfx::VertexLayoutHandle, unsigned short)
   7 Playground bgfx::Context::rendererExecCommands(bgfx::CommandBuffer&)
   8 Playground bgfx::Context::renderFrame(int)
   9 Playground bgfx::Context::swap()
  10 Playground bgfx::Context::frame(bool)
  11 Playground Babylon::Graphics::Impl::FinishRenderingCurrentFrame()
  12 Playground Babylon::CreateJsRuntimeDispatcher(Napi::Env, facebook::jsi::Runtime&, std::__1::shared_ptr<facebook::react::CallInvoker>)::'lambda'(std::__1::function<void (Napi::Env)>)::operator()(std::__1::function<void (Napi::Env)>) const::'lambda'()::operator()() const
  13 Playground std::__1::__function::__value_func<void ()>::operator()() const /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1873
  14 Playground std::__1::function<void ()>::operator()() const /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:2548
  15 Playground facebook::react::Instance::JSCallInvoker::scheduleAsync(std::__1::function<void ()>&&)::$_2::operator()(facebook::react::JSExecutor*) const /Users/ryantremblay/Repos/BabylonReactNative2/Apps/Playground/node_modules/react-native/ReactCommon/cxxreact/Instance.cpp:284

The first block above is also reported as leaks in the XCode Instruments leaks diagnostic tool, so it seems like we are not properly releasing these vertex buffers.

[CedricGuillemet]

It corresponds to what I found: vertex buffers are leaking. Only the ones created by .glb loaded. built meshes (box, spheres,..) are not leaking. I can repro that in babylon native. It looks like a reference counting issue/auto release in JS code.

With the fix, I'll also add checks on the CI to detect bgfx leaks. Any leak will fail the build.

There is also a leak on vertex layouts.

[CedricGuillemet]

First issue is fixed by this PR on babylon js BabylonJS/Babylon.js#9315
Second issue is in bgfx: vertexlayout not used by createVertexBuffer but used by setVertexBuffer and properly destroyed are leaking.
I'll finish my PR with leaks detection on babylonnative once that problem is fixed.

[CedricGuillemet]

Vertex buffer leak has been fixed with this PR : BabylonJS/Babylon.js#9318

[ryantrem]

I also verified the vertex buffer leak fix in Babylon React Native Playground on both Android and iOS using @babylonjs/core@4.2.0-rc.2.