Home

Docs for the AMR/LB library (OpenFOAM ESI)

This repository provides implementations of OpenFOAM libraries to run load-balanced adaptive mesh refinement on both hexahedral and polyhedral meshes. This wiki page intends to provide the absolute minimum amount of information needed to get started with the library.

First, here are some principles this project was built on:

• Unit/integration tests are documentation. By browsing tests, you get to see examples of usage for the most important classes (entry points and interfaces for the library). tests/Make also shows you how to compile and link against the library if you wish to do so.
• Unit/integration tests also give a rough idea on what we promise to keep stable throughout the life of the project. If a class is not tested, it means there is a chance its interface will soon change.
• We are all for keeping a lean Git history. This library is the result of a port of AMR/LB capabilities from blastFoam and all of the related history is retained.

It's important to set AMRLB_PROJECT to the root directory of this project!

Git can be useful while exploring

Want to know what needs to be done to a case so it runs with load-balanced AMR?

# Grab the first commit in which I touched the case
myfirstcommit=$(git log --format='%H' --author=Elwardi --reverse -- tutorials/damBreak2D | head -1)
# You want the diffs between one commit before that (last commit from the original subtree) and the most recent commit
# However, files will most likely be moved around and renamed.
git diff $myfirstcommit~..HEAD -M -- tutorials/<path/to/damBreak/on/openfoam/v2012/repo> tutorials/damBreak2D
# Example: This will show add diffs (including renames) which happened between first subtree inclusion
# and the currently checked-out commit
git diff $myfirstcommit~..HEAD -M -- tutorials/tutorials/multiphase/interFoam/laminar/damBreak/damBreak tutorials/damBreak2D
# Note that some of the diffs might be irrelevant; e.g. solution controls ... etc
# Also, you don't have to guess the paths; you can see all renames/deletes with:
git log --diff-filter=RD --summary

The same applies for solvers and library parts.

Requirements for running AMR/LB

• You need a solver that supports dynamic meshes. This basically it calls mesh.update() on a dynamicFvMesh and does flux correction at the right places.
• You need the ability to dynamically load libraries, and you need to load these libraries before OpenFOAM's standard ones. This is important because they have the same "names". Usually, this is the case for $FOAM_USER_LIBBIN libs.
• The first mesh.update() call in each time step will trigger AMR and LB functionality. Subsequent calls in the same time step will not.

Known/Possible issues

• The library is not thoroughly tested against all decomposition methods yet. So, if you encounter problems related to load-balancing, try a different decomposition method (scotch and hierarchical are preferred).
• There are verified issues of the Load-balancer with some specific boundary conditions.

Supported OpenFOAM versions

• The master branch is tested against OpenFOAM v2006 (same as v2012).
• There is a branch v2212 which is supposed to support newer versions. You can check the up-to-date CI build status for the version you're interested in.

How to add AMR/LB support to new solvers

The best/laziest/simplest way is to leverage Git history on the reactingDyMFoam solver. Because this solver was introduced using a subtree from OpenFOAM's repo, the diffs shows all steps (with bug fixes and everything):

# Get last commit which touched the solver
mylastcommit=$(git log --format='%H' --author=Elwardi -- applications/solvers/combustion/reactingDyMFoam | head -1)
#  Get first commit which touched the solver (moved files from subtree path)
myfirstcommit=$(git log --format='%H' --author=Elwardi --reverse -- applications/solvers/combustion/reactingDyMFoam | head -1)

# Go through the diffs
git diff $myfirstcommit..$mylastcommit

Unfortunately, a patch of these diffs will probably not be suitable to apply to different solvers, so you have to manually apply the changes for now.

How to reconstruct my LB-powered cases?

The recommended way to decompose and reconstruct cases which have load-balancing active is redistributePar:

# for reconstruction
runParallel redistributePar -reconstruct

Some issues were reported about usage of reconstructPar for case reconstruction, although simple workarounds are possible. See issue #7

How to add new error indicators?

Error indicators are the components which generate candidates for refinement and unrefinement. Typically, refinement candidates are the mesh cells which the use desires to refine to the maximum level of refinement. Unrefinement candidates are actually mesh vertices but the API is simplified to select vertices of provided cells, to keep the selection process consistent with the refinement part.

A few error indicators are provided. To add more, copy an original one, make the ::update() member method do what you want, and compile the class into its own shared library which you can load at case-level with the libs keyword in system/controlDict (In the same way you load libraries of this toolkit).

A fast-paced way to come up with specialized error estimators is to take advantage of the code one:

• Used coded error indicator for fast prototyping as shown in the tutorials. You can include external headers with codeInclude, and even link against external libraries if you need to.
• Invoke updateMesh at least once on the case to compile the dynamic code.
• Copy the generated shared library to your $FOAM_USER_LIBBIN 5you can find it inside the dynamicCode folder in your case).
• Load the copied library by its filename in system/controlDict
• Adjust errorEstimator keyword in constant/dynamicMeshDict to use the new class.
• If you care about the code, copy *Template.[HC] files from dynamicCode/<yourErrorEstimatorName> somewhere, clean them up of any dynamic-loading artifacts (some global functions, sha1s .... etc). These are valid c++ files, so you can compile them into a separate library and use it instead.