SofaCUDA Setup - DLL needs recompile?

[MHNeobird]

Hello, I am new to Sofa and FEM and also to CUDA.

My goal is to create an accurate simulation of stacked objects, with collisions and deformations.
I managed to create an xml file with a floor and multiple stacked cubes, below an example with one cube.
When I use runsofa to step through the simulation I see that at least collision works, although very slowly.

<FreeMotionAnimationLoop parallelCollisionDetectionAndFreeMotion="true" parallelODESolving="true" />
<LCPConstraintSolver tolerance="0.001" maxIt="1000" mu="0.6" computeConstraintForces="True" />
<CollisionPipeline verbose="0" depth="6" draw="0" />
<ParallelBruteForceBroadPhase />
<ParallelBVHNarrowPhase />
<LocalMinDistance alarmDistance="0.01" contactDistance="0.005" angleCone="10" />
<CollisionResponse response="FrictionContactConstraint" responseParams="mu=0.001" />

<Node name="Floor">
    <MeshSTLLoader name="Loader-Floor" filename="./Meshes\cube1x1x1.stl" translation="0.02500001 -0.15 0.03000001" rotation="0 0 0" scale3d="0.95 0.15 1.14" />
    <MeshTopology src="@Loader-Floor" />
    <MechanicalObject template="Vec3d" src="@Loader-Floor" />
    <TriangleCollisionModel simulated="0" moving="0" />
    <LineCollisionModel simulated="0" moving="0" />
    <PointCollisionModel simulated="0" moving="0" />
    <OglModel src="@Loader-Floor" color="blue" />
</Node>

<Node name="Box-27-0-978">
    <MeshGmshLoader name="VolumeMesh-Loader-27-0-978" filename="./Meshes\cube4x4x4.msh" translation="0.0341 0.007700002 0.982725" rotation="0 0 0" scale3d="0.2698 0.1463 0.17955" />
    <MeshSTLLoader name="SurfaceMesh-Loader-27-0-978" filename="./Meshes\cube4x4x4.stl" translation="0.0341 0.007700002 0.982725" rotation="0 0 0" scale3d="0.2698 0.1463 0.17955" />
    <MeshTopology src="@VolumeMesh-Loader-27-0-978" />
    <EulerImplicitSolver rayleighMass="0.001" rayleighStiffness="0.0001" />
    <CGLinearSolver template="CompressedRowSparseMatrixMat3x3d" iterations="200" tolerance="1E-09" threshold="1E-09" />
    <MechanicalObject template="Vec3d" src="@VolumeMesh-Loader-27-0-978" />
    <UniformMass totalMass="1.5" />
    <ParallelTetrahedronFEMForceField youngModulus="1E+08" poissonRatio="0.3" computeGlobalMatrix="true" plasticYieldThreshold="0.01" plasticMaxThreshold="0.05" plasticCreep="0.5" computeVonMisesStress="1" showVonMisesStressPerElement="1" />
    <LinearSolverConstraintCorrection />
    <Node name="Visual-27-0-978">
        <OglModel name="Model-Visual-27-0-978" src="@../SurfaceMesh-Loader-27-0-978" color="red" />
        <BarycentricMapping input="@.." output="@Model-Visual-27-0-978" />
    </Node>
    <Node name="Surface-27-0-978">
        <MeshTopology src="@../SurfaceMesh-Loader-27-0-978" />
        <MechanicalObject template="Vec3d" src="@../SurfaceMesh-Loader-27-0-978" />
        <TriangleCollisionModel />
        <LineCollisionModel />
        <PointCollisionModel />
        <BarycentricMapping />
    </Node>
</Node>

So I wanted to accelerate the simulation using CUDA, but I am not sure how to proceed.
With the files I downloaded comes a SofaCUDA.dll file, so I guess I can skip the compilation outlined in the docs?
When I set it as requiredPlugin I can see it is recognized by runsofa

[INFO] [SofaCUDA] CUDA: 1 device(s) found.
[INFO] [SofaCUDA] CUDA: 0 : "NVIDIA GeForce RTX 3060 Laptop GPU", 5067/6143 MB, 240 cores at 1.425 GHz, revision 8.6
[INFO] [SofaCUDA] , timeout enabled
[INFO] [SofaCUDA] CUDA: Using device 0 : "NVIDIA GeForce RTX 3060 Laptop GPU"

but when I change the template to CudaVec3d I get errors like

[ERROR] [InfoComponent(Not created (ParallelTetrahedronFEMForceField))] Object type ParallelTetrahedronFEMForceField<> was not created
The object is in the factory but cannot be created.
Requested template : None
Used template : None
Also tried to create the object with the template 'Vec3d' but failed for the following reason(s):
1. Since the attribute 'mstate' has not been specified, a mechanical state with the datatype 'Vec3d' has been searched in the current context, but not found.

and also

[ERROR] [InfoComponent(Not created (BarycentricMapping))] Object type BarycentricMapping<Vec3d,Vec3d> was not created

So my question is basically where is my error and do i need to recompile the dll because not everything is in there?

Edit: I found the CudaTetrahedronTLEDForcefield, but when using it I get the following error

[INFO] [SofaCUDA] GPU initialised for TLED: no error
[ERROR] [SofaCUDA] CUDA error: invalid configuration argument returned from MechanicalObjectCudaVec3t_vMEq_kernel
sofa::helper::BackTrace::dump
mycudaLogError
mycudaGetMultiProcessorCount
mycudaCheckError
CudaSortGPU
sofa::component::statecontainer::MechanicalObject<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<2,double>,sofa::type::Vec<2,double>,double> >::vOp
sofa::component::statecontainer::MechanicalObject<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<2,double>,sofa::type::Vec<2,double>,double> >::vOp

[alxbilger]

ParallelTetrahedronFEMForceField is the parellelized version of TetrahedronFEMForceField on the CPU. To get the GPU version, you have to use TetrahedronFEMForceField with the template CudaVec3d

[MHNeobird]

Hello, thanks for the reply.

Yes indeed, I found that out myself while experimenting and I also managed to adjust the Barycentric mapping for visuals and surface
<BarycentricMapping template="CudaVec3f,Vec3d" input="@.." />
I also replaced the LcpConstraintSolver for a GenericConstraintSolver, because the FreeMotionAnimationLoop was throwing errors with the LcpConstraintSolver.
My simulation is now running using CUDA, but that leads to new errors when I have a lot of boxes:

[ERROR]   [SofaCUDA] CUDA error: out of memory returned from cudaMallocHost
  sofa::helper::BackTrace::dump
  mycudaLogError
  mycudaGetMultiProcessorCount
  mycudaMallocHost
  sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::peudo_determinant_for_coef
  sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::reset
  sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::init
  sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::peudo_determinant_for_coef
  sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,float>,sofa::type::Vec<3,float>,float> >::init

I read about compute-sanitizer and when I use it like that
$ compute-sanitizer --tool memcheck runsofa
I get this, what is basically the same error

========= Program hit cudaErrorMemoryAllocation (error 2) due to "out of memory" on CUDA API call to cudaMallocHost.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: cuProfilerStop [0x7ff94ae0b905] in nvcuda64.dll
=========         Host Frame: sofa::component::container::SpatialGrid<sofa::component::container::SpatialGridTypes<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,float>,sofa::type::Vec<3,float>,float> > >::update [0x7ff94550ef62] in SofaCUDA.dll
=========         Host Frame: mycudaMallocHost [0x7ff944c3726c] in SofaCUDA.dll
=========         Host Frame: sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::peudo_determinant_for_coef [0x7ff94527af68] in SofaCUDA.dll
=========         Host Frame: sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::reset [0x7ff94527bfa8] in SofaCUDA.dll
=========         Host Frame: sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::init [0x7ff94526f685] in SofaCUDA.dll
=========         Host Frame: sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::peudo_determinant_for_coef [0x7ff94527a05a] in SofaCUDA.dll
=========         Host Frame: sofa::component::solidmechanics::fem::elastic::TetrahedronFEMForceField<sofa::gpu::cuda::CudaVectorTypes<sofa::type::Vec<3,float>,sofa::type::Vec<3,float>,float> >::init [0x7ff94526e296] in SofaCUDA.dll

How can I fix it?