Modify FEniCS tutorials to enable parallel runs

[IshaanDesai]

The FEniCS-Adapter now supports parallel runs: precice/fenics-adapter#71.
Correspondingly minor modifications to FEniCS scripts and preCICE configurations for all FEniCS tutorials are necessary to enable parallel runs.

[IshaanDesai]

For testing purposes the tutorial HT/paritioned-heat/fenics-fenics was manually copied to the adapter branch: https://github.com/precice/fenics-adapter/tree/parallelMidLevel
Now that this PR exists, the prototype within the adapter branch has been removed (precice/fenics-adapter@f541ddd) and testing is done within this PR to record the results of parallel runs of all tutorials.

[IshaanDesai]

The tutorial HT/partitioned-heat/fenics-fenics with a coupling interface which is parallel to the Y-axis now works in parallel. The domain decomposition is as follows:

The left participant is the Dirichlet participant and the right participant is the Neumann participant.

[BenjaminRodenberg]

Do we want to merge this PR first into develop or #124? @fsimonis @MakisH any opinions?

[BenjaminRodenberg]

While reviewing precice/fenics-adapter#71 I used the content of this PR. I tried to run the CHT tutorial and ran into the following error (looking like issue with the parallelization):

benjamin@benjamin-ThinkPad-X1-Yoga-2nd:~/tutorials/CHT/flow-over-plate/buoyantPimpleFoam-fenics$ mpirun -np 2 python3 Solid/heat.py
---[precice] ERROR:  For a parallel participant, only the mapping combinations read-consistent and write-conservative are allowed
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[38575,1],0]
  Exit code:    255
--------------------------------------------------------------------------

For the FSI test cases I also ran into errors (looking like issues with the adapter):

benjamin@benjamin-ThinkPad-X1-Yoga-2nd:~/tutorials/FSI/flap_perp/OpenFOAM-FEniCS$ python3 Solid/perp-flap.py
---[precice]  This is preCICE version 2.1.1
---[precice]  Revision info: v2.1.1-54-g25b9f155
---[precice]  Configuring preCICE with configuration "/home/benjamin/tutorials/FSI/flap_perp/OpenFOAM-FEniCS/Solid/../precice-config.xml"
---[precice]  I am participant "fenics"
Traceback (most recent call last):
  File "Solid/perp-flap.py", line 76, in <module>
    precice_dt = precice.initialize(coupling_boundary, read_function_space=V, write_object=V,
  File "/home/benjamin/.local/lib/python3.8/site-packages/fenicsprecice/fenicsprecice.py", line 363, in initialize
    raise Exception("Dimension of preCICE setup and FEniCS do not match")
Exception: Dimension of preCICE setup and FEniCS do not match
---[precice]  Implicitly finalizing in destructor

and

benjamin@benjamin-ThinkPad-X1-Yoga-2nd:~/tutorials/FSI/cylinderFlap/OpenFOAM-FEniCS$ python3 Solid/cyl-flap.py 
---[precice]  This is preCICE version 2.1.1
---[precice]  Revision info: v2.1.1-54-g25b9f155
---[precice]  Configuring preCICE with configuration "/home/benjamin/tutorials/FSI/cylinderFlap/OpenFOAM-FEniCS/Solid/../precice-config.xml"
---[precice]  I am participant "fenics"
Traceback (most recent call last):
  File "Solid/cyl-flap.py", line 92, in <module>
    precice_dt = precice.initialize(coupling_boundary, read_function_space=V, write_object=V,
  File "/home/benjamin/.local/lib/python3.8/site-packages/fenicsprecice/fenicsprecice.py", line 363, in initialize
    raise Exception("Dimension of preCICE setup and FEniCS do not match")
Exception: Dimension of preCICE setup and FEniCS do not match
---[precice]  Implicitly finalizing in destructor

[MakisH]

Do we want to merge this PR first into develop or #124? @fsimonis @MakisH any opinions?

If it should work with the develop branch of the fenics adapter, add it to the develop of the tutorials. If it depends on the v1 branch, merge it to the v1 branch. Unless I am missing something.

[IshaanDesai]

I tried to run the CHT tutorial and ran into the following error (looking like issue with the parallelization)

This PR currently only tests the HT/paritioned-heat/fenics-fenics tutorial. For FSI, the 2D-3D coupling needs to changed to a new mechanism where the non-FEniCS participant will handle the data modification. The corresponding change in the fenics-adapter is already included in the parallel design and is being separately also handled here: precice/fenics-adapter#102

[fsimonis]

Feel free to merge this PR first.

[BenjaminRodenberg]

I tried to run the CHT tutorial and ran into the following error (looking like issue with the parallelization)

This PR currently only tests the HT/paritioned-heat/fenics-fenics tutorial. For FSI, the 2D-3D coupling needs to changed to a new mechanism where the non-FEniCS participant will handle the data modification. The corresponding change in the fenics-adapter is already included in the parallel design and is being separately also handled here: precice/fenics-adapter#102

@IshaanDesai

• Then we should fix the CHT case here (Modify FEniCS tutorials to enable parallel runs #120). It's only a few changes in the config. I can also do this, if you want. done in
  6ef8088 . Now I'm also facing the 2D/3D error like in the FSI cases.
• I tested the FSI cases from this PR (Modify FEniCS tutorials to enable parallel runs #120) with the fenics-adapter from Support parallel runs fenics-adapter#71 is this expected to fail? I understand from your comment that this should work.

[IshaanDesai]

I tested the FSI cases from this PR (#120) with the fenics-adapter from precice/fenics-adapter#71 is this expected to fail? I understand from your comment that this should work.

All FSI cases currently are also 2D-3D and hence will not work with the current parallel design. Just changing the precice-config by setting <solver-interface dimensions="2"> instead of "3" should ideally work. I am still confirming this.

[BenjaminRodenberg]

Merged #127 into this PR. Note that CHT case is still causing problems. I think that there is a bug in the configuration of the mapping. If I git checkout develop CHT/flow-over-plate/buoyantPimpleFoam-fenics/precice-config.xml and change the dimension to 2. I.e.: Correct 2D/3D, but not working with parallel runs, everything is fine.

[BenjaminRodenberg]

55d7579 uses this configuration file. Results look good (for 1 and 2 ranks for Solid/heat.py). This is also a good step, if we want to make the tutorials more consistent (@uekerman).

However, I got some confusing messages from the adapter for this case, when running buoyantPimpleFoam -case Fluid/ & python3 Solid/heat.py:

---[precice]  it 1 of 200 | dt# 1 | t 0 of 1 | dt 0.01 | max dt 0.01 | ongoing yes | dt complete no | write-iteration-checkpoint | 
---[precice]  initializeData is skipped since no data has to be initialized.
output vtk for time = 0.0
Rank 0: Nothing to Receive
Rank 0: Nothing to Send
Solving linear variational problem.
Solving linear variational problem.

Nothing to Receive, Nothing to Send looks wrong to me, but the results look fine. I assume this is just a logging issue and debugging output that can be removed anyway. @IshaanDesai can you take a look at this?

[BenjaminRodenberg]

Intermediate summary

• HT and CHT are working nicely
• FSI cases fail due to a deadlock in the adapter that occurs, if point sources are used. @IshaanDesai is aware of this issue and will fix it in precice/fenics-adapter#71.

[IshaanDesai]

FSI cases with FEniCS as the structure participant do not work in parallel because fenics-2019.1.0 hangs while defining PointSources in parallel. This has been buggy in the past as said by the core developers here: https://fenicsproject.discourse.group/t/pointsource-definition-in-parallel-leads-to-deadlock/4729/3

Note: The changes to preCICE configurations for FSI cases can be retained as these configurations work in serial.

[BenjaminRodenberg]

Minor comment on dead code. Rest looks good to me. I did not try running the CHT and HT case again. Do you want me to run them?

[IshaanDesai]

Minor comment on dead code. Rest looks good to me. I did not try running the CHT and HT case again. Do you want me to run them?

Last clean-up is on-going. A last run from you will make this foolproof, so yes one run check would be good.