Use of SoftRobots in High Performance Computing

[andersonnardin]

Hi all,

I have a few questions about using the SoftRobots plugin in High-Performance Computing. To explore its total capacity, I am also using the SOFA MultiThreading.

First, SOFA version 23.06 does not come with the SoftRobots tutorials folder. Is this permanent?

Second, in version 22.12 I cannot use the component "ParallelTetrahedronFEMForceField". It accuses the error:
Python exception:
ValueError: Object type ParallelTetrahedronFEMForceField was not created
The object 'ParallelTetrahedronFEMForceField' is not in the factory.
But the following exits:
: ParallelHexahedronFEMForceField (92% match)
: TetrahedronFEMForceField (85% match)
: HexahedronFEMForceField (76% match)
: StandardTetrahedralFEMForceField (75% match)
: TetrahedronDiffusionFEMForceField (73% match)
: TetrahedralCorotationalFEMForceField (70% match)
However, the same component is allowed in version 23.06. Was that a bug fixed?

Third (and most important), I have a scene where I am using the Multithreading plugin, components ParallelTetrahedronFEMForceField (just available in 23.06), ParallelBruteForceBroadPhase, and ParallelBVHNarrowPhase. The scene has a large number of tetrahedrons (seriously!) and runs in an HPC with a large number of CPUs. However, I verified no improvement in computational time compared to the same scene without the mentioned components (without Parallel...) or even when it runs on my Personal Computer with much fewer cores. Any clue about this behavior?

Thank you in advance!

[alxbilger]

Hi,

ParallelTetrahedronFEMForceField has been introduced in January. That is why it is not available in previous versions. See sofa-framework/sofa#3552.

Parallel components are usually not trivially parallelized. They have synchronization mechanism, which means the speed up does not match the number of CPUs. In addition, the speed up due to parallelism of a component affects only a part of a time step, not the entire time step.

Not knowing the scene, here are my guesses if you observe no improvement:

• If you use a direct linear solver, a large part of the matrix assembly can be due to the matrix data structure. It is not thread-safe, that is why it must be done sequentially, and it is rather time-consuming.
• The force computation is not (yet) parallelized. In simulations where the number of tets is more modest, this step is usually not the bottleneck.
• You have a speed-up, but compared to other steps of the simulation, it is not significant. For example, constraint solving.
• We never tried this component on an HPC. This component and its underlying task pool is maybe not efficient for this kind of architecture.

I advise to run a few time steps and to analyze the timers: https://www.sofa-framework.org/community/doc/using-sofa/performances/inspect-performances/

Alex

[andersonnardin]

HIi @alxbilger,

Thank you for your response.

1- Yes, I am using a linear solver (LinearSolverConstraintCorrection). What would be an alternative to avoid the sequential execution due to its presence?

2- Yes, I am computing forces, but this is unavoidable for me due to the nature of the study I am conducting. I look forward to a solution that includes the parallelized version of this.

3- I have used the AdvancedTimer profiler, and I cannot verify any speed-up. Both steps ConstraintSolver and Collision Detection take around 50% of the time either in my PC or in HPC.

4- I tried to run SOFA without the graphical interface to test it (using the argument -g batch), but it accuses the error:
[ERROR] [DAGSimulation(Simulation)] Cannot load file 'C:\Users...': extension ( py) is only supported if the plugin SofaPython3 is loaded. SofaPython3 must be loaded first before being able to load the file.

However, the SofaPython3 was loaded. This happens in versions 22.12 and 23.06 on both Windows and Linux. Any idea about how to solve it?

[alxbilger]

Keep in mind that the optimization of performances is about timings, not the relative percentages.

The ComputeForce timer is included in the FreeMotion timer, but in your case I guess this step is in an asynchronous thread (parallelCollisionDetectionAndFreeMotion set to true). That's why it is hidden. You can set parallelCollisionDetectionAndFreeMotion to false to benchmark the free motion step (or use Tracy), and set it back to true later.

You have 2 steps which are the bottlenecks:

1. The free motion step: in this step, you have:
   1. The force computation -> not really time consuming but it needs to be measured. Does not use parallelism for now
   2. The linear system assembly -> time consuming. Does not use parallelism for now
   3. The linear system solving -> really time consuming. Does not use parallelism for now. To leverage parallelism in this step, I recommend https://github.com/SofaDefrost/SofaCUDALinearSolver or https://github.com/sofa-framework/sofa/tree/master/applications/plugins/SofaPardisoSolver/. The latter is probably not up-to-date. I've never tried it myself.
2. The constraint solving -> not parallelized except with parallelInverseProduct.

If some steps of the simulation are not parallelized, they won't get any speed up if run on an HPC. So you must leverage as much parallelism as possible.

Another option to speed-up the linear system solving is the combination of a preconditioned conjugate gradient and an asynchronous linear solver (see for example https://github.com/sofa-framework/sofa/blob/master/examples/Component/LinearSolver/Preconditioner/FEMBAR_PCG_AsyncSparseLDLSolver.scn). With this configuration, the parallelism from ParallelTetrahedronFEMForceField is used in the conjugate gradient. The asynchronous solver factorizes the matrix in an different thread. However, you won't gain anything in the constraint solving.

[andersonnardin]

Hi @alxbilger

First, I toggled the parallelCollisionDetectionFreeMotion and found the timer ComputeForce. However, as you can see in the picture below, it takes less than 1%.

Second, I tried the combination of the preconditioned conjugate gradient and the asynchronous linear solver. In practice, I substituted the previous SparseLDLSolver for:

      # unit.addObject('SparseLDLSolver', name='preconditioner', template='CompressedRowSparseMatrixMat3x3d')
       unit.addObject('ShewchukPCGLinearSolver', name = 'PCG', iterations = '1000', preconditioner = '@preconditioner')
       unit.addObject('AsyncSparseLDLSolver', name='preconditioner', template='CompressedRowSparseMatrixMat3x3d')

Then, I have gotten the error:
[ERROR] [LinearSolverConstraintCorrection(LinearSolverConstraintCorrection)] Can not use the solver PCG because it is templated on GraphScatteredType

Third, before testing the parallelInverseProduct, I should compile the sofa master branch. I pulled the Ubuntu image from the recommended docker hub:
https://hub.docker.com/r/sofaframework/sofabuilder_ubuntu
However, I expected the container would have a graphical desktop environment, but I could not access it via VNC Viewer or browser. I verified the image was updated yesterday. Could you verify and let me know whether it works properly for you?

[alxbilger]

1. Yes, ComputeForce is really not the bottleneck.
2. Your LinearSolverConstraintCorrection should refer to the preconditioner, not the PCG
3. I never used docker with SOFA. I recommend opening a discussion on the SOFA repository for this specific issue.

[andersonnardin]

Hi @alxbilger ,

I think I misunderstood something. You said:
"Your LinearSolverConstraintCorrection should refer to the preconditioner, not the PCG"
But... doesn't PCG stand for PRECONDITIONED conjugate gradient? So, how should I refer to preconditioner, not PCG?
I checked again the example you sent, but I did not find the LinearSolverConstraintCorrection component. This is mine:

[alxbilger]

In the example you gave:

unit.addObject('ShewchukPCGLinearSolver', name = 'PCG', iterations = '1000', preconditioner = '@preconditioner')
unit.addObject('AsyncSparseLDLSolver', name='preconditioner', template='CompressedRowSparseMatrixMat3x3d')

The component preconditioner is the preconditioner for the preconditioned CG called PCG. Your LinearSolverConstraintCorrection should refer to the preconditioner (AsyncSparseLDLSolver), not ShewchukPCGLinearSolver. From your screenshot, it seems it is the case. But you still have the error [ERROR] [LinearSolverConstraintCorrection(LinearSolverConstraintCorrection)] Can not use the solver PCG because it is templated on GraphScatteredType?