Different Solutions Running in Serial v. Parallel Using Python Wrapper w/ Mesh Deformation

[j-signorelli]

Hi,

I am obtaining quite different solutions in parallel v. in serial when using the Python wrapper with mesh deformation. I set up a simple test case to aid in helping resolve this issue. I have attached my Python script as a .txt file at the bottom.

This simple test case is just flow in a channel, where I use the Python wrapper to enforce an oscillating wall. I use an initial flowfield from a steady, nondeforming solution. Below is the video of the pressure through time:

1core.mp4

When I run the same setup but on 3 processes, I get the following result:

3processes.mp4

There are very unusually sharp pressure gradients at the side as can be seen. This problem is occurring directly at the interface of domains decomposed onto different ranks:

Is there something I am missing in the Python file? Any help is really appreciated!

SU2_FSI_TestCase_PythonFile.txt

[pcarruscag]

What branch are you using? How do you compile it? Run it (python script)?

[j-signorelli]

Hi @pcarruscag , Thanks a ton for the quick response. I am using master branch v7.5.1. I compile with:

export CC=$(which mpicc)
export CXX=$(which mpicxx)
export MPICC=$(which mpicc)
export MPICXX=$(which mpicxx)

./meson.py build -Dcustom-mpi=true  -Denable-pywrapper=true -Denable-mkl=true -Dmkl_root=$MKL_DIR --prefix=~/software/SU2/install/
./ninja -C build install

I run by:

mpirun -n 3 python3 SU2_FSI_TestCase.py -f configfile.cfg --parallel

[j-signorelli]

I should note I am using Intel MPI. I haven't been able to get SU2 to work with it without setting up "custom MPI" like this.

[pcarruscag]

I'm not sure this is right

    # Obtain indices of all vertices that are being worked on on this rank
    for iVertex in range(nVertex_MovingMarker):
        if not SU2Driver.IsAHaloNode(MovingMarkerID, iVertex):
            iVertices_MovingMarker_PHYS.append(int(iVertex))

You need to set the displacement for all vertices (including Halos) or call a function of the wrapper to communicate them (CommunicateMeshDisplacements ? , it should be easy to find if you run dir(SU2Driver). But in your case just loop over all vertices.
Let me know if that works.

[j-signorelli]

Thanks @pcarruscag but sadly I have already tried setting every vertex including halos without any luck, and just tried again to confirm once more.

I also do call SU2Driver.CommunicateMeshDisplacement() for good measure in the Python file I attached, but this made no difference as mesh displacements are already communicated in the beginning of CMeshSolver::DeformMesh, which is called in CSinglezoneDriver::DynamicMeshUpdate, which is called in CSinglezoneDriver::Preprocess.

Please let me know if you have any other suggestions! I am not sure if this may be a bug in SU2.

[pcarruscag]

I see, the deformation looks smooth in both videos.
Is the flow solver converging to the same residuals in serial vs parallel?
Can you upload the screen output in both cases, the mesh and config would also help.

[j-signorelli]

Of course, thanks so much @pcarruscag ! I can confirm that the deformation in the output is smooth:

I also did verify that the flow solver is converging to the same residuals in serial v. parallel. My convergence criteria is an RMS_ENERGY < 1e-1.5. Attached is screen output for 3 procs, 1 proc, the config file, the mesh, and the initial flow dat file, all as text files so I can upload them here.

3cores.txt
1core.txt
config_file_testcase.txt
fluidMesh.txt
initial_flow_00000.txt

[pcarruscag]

I'll have a look but if the mach number of 0.01 matches the velocities you are getting that may be the root of the problem. Compressible solvers without low speed preconditioning are known for not very good solutions.
Have you tried the incompressible solver?

[j-signorelli]

Just tried it, same problem sadly:

1 process:

3 processes:

Thanks so much for looking into this @pcarruscag

[pcarruscag]

Took a bit but I found it #1933
It was just a problem with euler markers + deforming markers, not related to the wrapper or MPI.
We recently changed the python wrapper for version 8... so your script will not work directly with the branch in the PR.
You can use this script run.py.txt or git-cherrypick my commit into master and continue using yours.

Can we use your example as a regression test?

[j-signorelli]

Thank you so much for figuring this out! You can absolutely use the example.