Add Surface Tension

[wilfonba]

Description

This PR adds the surface tension model of Schmidmayer et. al 2017 to MFC.

Fixes #276

Type of change

• New feature (non-breaking change which adds functionality)

Scope

• This PR comprises a set of related changes with a common goal

How Has This Been Tested?

• This is the 2D equilibration of pressure inside a water droplet in air. It's discretized onto a 50x50 Grid with reflective boundary conditions at x[y]_beg and ghost cell extrapolation for x[y]_end. The case file is available in examples/2D_equilibration/. This simulation was run on M2 Apple Silicone.

test.mp4

• This is the 2D relaxation of a square to a circle in 2D. It's discretized onto a 129x128 grid with reflective boundary conditions at x[y]_beg and ghost cell extrapolation for x[y]_end. The case file is available in examples/2D_recoverCircle. This simulation was run on 2 V100 GPUs.

test.mp4

• This is the 3D equilibration of pressure inside a water droplet in air. It's discretized on a 100x100x100 grid with reflecting boundary conditions at x[y,z]_beg and ghost cell extrapolation at x[yz]_end. This simulation was ran using 4 V100 GPUs on Pace Phoenix.

test.mp4

• This is the 3D relaxation of a cube to a sphere. It's discretized on a 200x200x200 grid with reflecting boundary conditions at x[y,z]_beg and ghost cell extrapolation at x[yz]_end. This simulation was ran using 8 MI250 GPUs on OLCF Frontier. While equilibration is not reached, the relaxation shown is the asymmetric and the expected behavior. One octant of the cube is simulated and reflections used for visualization. This case is given as an example in examples/3D_surface_tension.

test.mp4

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected
• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using Cray compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new feature performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR nsysRaw.txt
• Ran an Omniperf profile using './mfc.sh run XXX --gpu -t simulation --omniperf', and have attached the output file and play text results to this PR. omniRaw.txt
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature
  
  
• Ran examples/3D_weak_scaling with 32GB per GPU with 8 MI250 GPUs on both the master branch and this fork with the following time/time-step:
  Master: 0.555s
  This Fork: 0.539s
  I'm not sure why this fork is faster, but these results were consistent across three runs.

[anandrdbz]

@wilfonba check s_rectangle subroutine in m_patches.fpp, it only defines smooth_patch_id when ib is not true (lines 893-894), and you added a further check at line 928 where that variable is no longer defined, which is causing a bug in the 3D IBM case

[wilfonba]

I forgot I'd added that stuff. Thanks for taking a look @anandrdbz!

[sbryngelson]

@wilfonba sync w/ master

[sbryngelson]

should be able to debug these Test Suite / Github (macos, mpi, debug, false) (pull_request) from your own mac

[henryleberre]

lgtm on my side.

• I fixed ./mfc.sh count_diff and overall improved how it's written.
• I reverted the changes you made in the 3D_weak_scaling case where you hardcoded the time-stepping parameters.

[henryleberre]

Note: ./mfc.sh count_diff is called from master so the Check Line Counts workflow is expected to fail despite the fixes I pushed to this PR.

[wilfonba]

Let me fix the macOS no debug runners. Not sure why those would have failed, but I haven't ran the test suite with no debug recently

[sbryngelson]

Let me fix the macOS no debug runners. Not sure why those would have failed, but I haven't ran the test suite with no debug recently

Related to new issue #443

[sbryngelson]

Can merge this PR without that runner working