Double Precision Worlds

[SpiderPig523]

My game will support 64bit floats for world positions as I want to create planet sized environments. I'm wondering if newton has support for this or do I need to make my own partitioning system so that collisions always use 32bit floats and have bodies removed if they get too far away?

[JulioJerez]

yes the engine support double.
has you ever build the engine using cmake?
all those options are there.

[SpiderPig523]

Oh okay, I'll take a look. 😃

[SpiderPig523]

I think I'm getting some form of memory corruption when using netwon compiled with double precision.

Most of the time it crashes here. But every time I launch my app it crashes in several different places. No changes or re-compiling or anything.

The playercapsule demo works with double precision. Haven't tried the other demos yet.

If I go back to using newton with float precision my app works again. I can post some other callstack screenshots if it helps.

I am linking my VS22 project to newton compiled as a static library. I also have defined D_NEWTON_USE_DOUBLE in the preprocessor definitions so the headers reflect what has been compiled into the lib file.

[JulioJerez]

I do not get any crashes in double precision in any of the demos.
In the file ...\newton-4.00\sdk\dCore\ndMemory.cpp there is a define that enables the Windows memory integrity check:
#define D_MEMORY_SANITY_CHECK
Try uncommenting that. It may help detect the first time the memory becomes corrupted.
It is extremetely slow.

Also, the code you are showing is guaranteed to produce a memory leak or a crash.
ndShape and all of its subclasses are internal engine data structures.
They use intrusive reference counting, so deleting one of them manually will corrupt memory and eventually crash the application.
These objects are meant to be handled through the ndShapeInstance class.

For example:
ndShapeInstance* myInstance = new ndShapeInstance(new ndShapeBox(...));

ndShapeInstance takes ownership of the ndShape pointer.

So my guess is that the crash happens because when Collider goes out of scope, it attempts to delete a reference-counted shape.
I was considering replacing raw shape references with a smart pointer, but that would be a very large interface change and would break every existing application. Because of that, it is better to leave that change for the next major version update.

[SpiderPig523]

Thanks, I enabled D_MEMORY_SANITY_CHECK and consistently got the following error.

So my guess is that the crash happens because when Collider goes out of scope, it attempts to delete a reference-counted shape.

I'm using raw pointers so nothing should be going out of scope at the end of that function. 😕

[JulioJerez]

there is no ndMemory::CheckMemory() in current sdk.
that was a function of previous version where free list chucks had their own ptr and OS memory check did not work on those.
so it was hard to track memory corruption.
Try sync and try again.

edit:
I re added the function again.
/// check memory heap integrity using operating system support.
D_CORE_API static bool CheckMemoryHeap();

now the check is also call form the free list

The error you are getting is not form the allocation, that error have before.
you can try to check memory before the you call the function. like this
ndAssert (ndMemory::CheckMemoryHeap());

you need to sync to get that.

[SpiderPig523]

Synced and rebuilt my project.

Error shows an issue in ndMemory.cpp.

[SpiderPig523]

Here's the call stack with source code.

[JulioJerez]

It shouldn’t be this difficult. The code you’re showing me is something I have never seen before.
Are you creating bodies from inside a Newton update loop?

I’m having trouble understanding what you are doing.
Creating a class that collects an ndShapeBox is not something I have seen anyone do before,
and once you go in that direction, I lose confidence in the rest of the implementation.
An ndShape... is an argument used to construct an ndShapeInstance.
All interactions with a shape must go through ndShapeInstance.
Also, if you are using one, the correct way to destroy it is by calling Release(), not delete.

If you don’t want to step through the demos, you can disable them in CMake.
There is an option called Build Tests, which builds a set of tutorials located in:
C:\development\newton-dynamics\newton-4.00\tests

Take a look at tutorial rigidBody_test.cpp.
It shows how to assemble the most basic rigid body.

[SpiderPig523]

Ok I have changed my code to reflect your suggestions. The body was being created in the world loop but have changed that to create it before the loop.

`Collider* CreateCubeCollider(Decimal width, Decimal height, Decimal depth) {
auto collider = new Collider();
collider->m_shape_instance = new ndShapeInstance(new ndShapeBox(width, height, depth));

return collider;

}
`

A entity can have a collider changed so I set an entities collider like this.

m_newton_body->GetAsBodyKinematic()->SetCollisionShape(*m_collider->m_shape_instance); m_newton_body->GetAsBodyKinematic()->SetMassMatrix(m_mass, *m_collider->m_shape_instance); m_world_index = m_physics_world->AddBody(this);

Now this all works with float precision. But still does not work with double precision.

[JulioJerez]

on check memory on each allocation is actually too slow and does even detect when the bug happens..
It just say something overrun memory.

top find out when that happen, I made the memory integrity check available to the application.
it is this function ndMemory::CheckMemoryHeap()
It is possible that bug it is in Newton, but since I am running with double precision and I do not have the crash,
you are going to have to do some detective work to debug it
startion from this function

Collider* CreateCubeCollider(Decimal width, Decimal height, Decimal depth) 
{
auto collider = new Collider();
collider->m_shape_instance = new ndShapeInstance(new ndShapeBox(width, height, depth));
return collider;
}

you can change it to this

Collider* CreateCubeCollider(Decimal width, Decimal height, Decimal depth) 
{
ndAssert(ndMemory::CheckMemoryHeap());
auto collider = new Collider();
collider->m_shape_instance = new ndShapeInstance(new ndShapeBox(width, height, depth));
ndAssert(ndMemory::CheckMemoryHeap());
return collider;
}

and set a break point. at the begining.
when the program crash, you get go t other caller function in the trace stack, and you insert
ndAssert(ndMemory::CheckMemoryHeap()); aft every suption call,

this way you can narrow down to what function is causing the memory overrun.
that the last function that experience the crash in not the one that cause it.

You have to sync to get that change.

[JulioJerez]

actually, the stack show that this happen early.

place the line below in few places in the function CreateBody()
ndAssert(ndMemory::CheckMemoryHeap());

[SpiderPig523]

I did some testing and it appears to be the creation of ndBodyDynamic that causes the problem. It crashes at line 41 all the time. If I comment out the "m_newton_body = new ndBodyDynamic();" My app will work. I also had "ndAssert(ndMemory::CheckMemoryHeap());" right before the creation of ndBodyDynamic and it didn't crash there. (Its not in the screen shot)

Maybe this in the output window might help?

[SpiderPig523]

I can't place "ndAssert(ndMemory::CheckMemoryHeap());" any further down into the class ndBodyDynamic as the source is not compiled directly into my app. Although if you need me to do that to further track this down I will. 😄

[JulioJerez]

put the check before and after new and see if it gets the error.

my guess is that the source is and you code do not have the same compile options.
how is the source compiled?

try to place more line,
ndBodyDynamic::ndBodyDynamic()
:ndBodyKinematic()

until you get the line that override the memory.

I just try in the stuff I am working now.

ndBody* ndMeshBodyDynamic::CreateObject() const
{
	ndAssert(ndMemory::CheckMemoryHeap());
	ndBodyDynamic* const body = new ndBodyDynamic();
	ndAssert(ndMemory::CheckMemoryHeap());
	body->Deserialize(this);
	return body;
}

an that works in float32 and float64.
you must have some kind of alignment or size problem at compile time.

[SpiderPig523]

my guess is that the source is and you code do not have the same compile options.
how is the source compiled?

I looked over my compile options and most are the same. I'm trying to match them up exactly now.

Still get the error after new.

[SpiderPig523]

I might have to create a very minimal VS project that imports the library and creates a single body with no rendering. Maybe then we can narrow things down a bit better. I could probably send you the whole project for testing if needed.

[JulioJerez]

I just try to build the sdk and demo in 32 bit, and it works fine.
It seems that every time we go over one problem, you find another one in an even a more basic engine foundation.
I am running out of options.

Is your engine open source? or something I can download and test?
I am not saying it is you, there could be something wrong in the engine, but whatever that is I am not aware of.
I do not know what else to do.

I might have to create a very minimal VS project that imports the library and creates a single body with no rendering. Maybe then we can narrow things down a bit better. I could probably send you the whole project for testing if needed.

ahh yes, that sound like a good idea.
there are some basic tutorial, but at this point, see if you can just make a test the way you understand it.
Maybe there is a pathology that you are seeing and I am too bias to reproduce.

[SpiderPig523]

Yeah that's okay. I'll build a minimal project and see if the issue persists.

[SpiderPig523]

Okay, try this project if you can and let me know what you think. It's very minimal and I'm not sure if I should set up newton better to show the bug better - but it crashes when using d_ndNewton_d.lib and not using f_ndNewton_d.lib

For both lib files I turn ":option("NEWTON_BUILD_SHARED_LIBS" "build shared library" OFF)" to OFF

For d_ndNewton_d.lib I turn "option("NEWTON_DOUBLE_PRECISION" "generate double precision" ON)" to ON

Those are the only things I change when building newton.

NewtonDoubleTests.zip

[JulioJerez]

ok I build it and I get the crash.
I will try to link it to the newton lid

edit:
there is not project to build the library. do you have that?

[JulioJerez]

alright. there are few problems. and they are my fault.
This came from the time I was debugging an issue with memory alignment.
I commented out this define #define D_NEWTON_ENFORCE_CLASS_ALIGNMENT
and I check it the code.

That define has no effect when building with avx2 since floats are aligned to 32 bytes.
it become a big problem when not using Avx2 because in newton all structs are aligned to 32 bytes,
but a class with no avx, if the first member is a vector the is aligned to 16 bytes, that's a huge issue.
the 32 byte alignment has to be enforced.

I am guessing this is probably a cause of all those problems that you are getting that seems so weird.

the other point is that I see you are copying the code to a folder and making a build.
in case you are not doing it, here is a way to do it more reliable.

you can open cmake and in the option, you select and installation folder.

after you set all the other options that you want. You create the solution.
then open visual studio and build the the engine.
then, in visual studio, you can build the ALL Build->Install folder.

this will copied the entire source code to a flat directory that you can use either use as source or lib.
It also copy the compiled libraries and or dlls in your application destination path.

to using, it you only need to include one header

#define D_NEWTON_USE_DOUBLE
#include "ndNewtonInc.h"

I used that method and uploaded you test.
NewtonDoubleTests.zip

please sync, build and see if that works for you now.

[SpiderPig523]

Okay thanks.

you can open cmake and in the option, you select and installation folder.

Yes that is much easier! 😄

EDIT - Got it too build. I'll test with my engine now.

[SpiderPig523]

With this setup ofbx is interfering greatly with my game engine. All of ofbx's structs have the same names as my own. Is there a using namespace ofbx somewhere? I'd need to comment that out so I can compile.

[SpiderPig523]

Found it in ndFbxMeshLoader.h. Commenting it out fixed my issue and newton seems to be fine without it.

[SpiderPig523]

Works in my game engine now. Thanks for your help on this one Julio. 😃

[SpiderPig523]

...I have run into another crash but I think I'll sleep on it. 🤣

[JulioJerez]

ahh yes, the openfbx, tinyxml, and vhacd are copy as part of the source because they are use by the ndMesh to import external file.
I guess you are using openfbx in your app.
vhacd and tinyxml are under under a name space already, openflx is the later addition and I forgot to name space it.
I will do it today.
Thanks for that tip.

what is the other crash?

[SpiderPig523]

This is the new error I get when I shoot about half a dozen cubes into the world. (double precision build)

[JulioJerez]

Ah, I see, the engine reuse a few buffers to save memory.
Typically, m_bvhSceneManager ends up holding about twice as many items as the main body list.
so they is not need to check bound.

However, that is a mistake, if the scene is changing (which seems to be the case here),
there’s a potential issue right after adding bodies. In that frame, the body array could temporarily have more entries than m_bvhSceneManager, since the scene is sorted after the frame update.
This will trigger an assert in debug, but more important It may cause a memory overrun,
if m_bvhSceneManager was actually small in size.
I will be extremely rare that that was the case, but by murphy's law, if there is a known chance of fail, it will fail.

I’ve added a check to ensure that m_bvhSceneManager is always at least as large as the main body list.
I committed the fix to my working branch. later I will update the main branch.

That’s a solid catch, nice work, and thanks for reporting that issue.

[SpiderPig523]

Ah okay. No problem at all! 😄

Thanks for all your help. I'll sync soon!

[JulioJerez]

I’ve updated the namespace from ofbx to ndOfbx.

OpenFBX is such an impressive little library, it’s hard to believe how good it is for its size.
I’m honestly surprised Autodesk hasn’t taken action against it.

Back in 2009, I bought 3ds Max 2010 and included the FBX libraries in my project.
After reaching out to support, I got a friendly reminder that while the FBX file format itself is royalty-free,
the official FBX binary libraries are not. Because of that, I had to remove them.

I guess that’s one of the reasons people have reverse-engineered the format,
which also explains why there are so many incompatible versions floating around.

So it was a great find coming across this tiny library.
It’s really solid, and it works directly with the original format rather than an interpretation.

Anyway, if you sync and re-import, everything should work without any modifications.

Found it in ndFbxMeshLoader.h. Commenting it out fixed my issue and newton seems to be fine without it.

It is fine for now, but I am working on the serialization, so that file will be important to run the demos.
and if you want to import ndMesh ready to use by a third party apps.

ndMesh will be a way for end users to shared and debug issues.
It will also be an interface to import models like URDF for physics and other graphics files.
It also does fbx, but I will have other formals.
The hopes is that this simplify the usage of the more advance engine features.