Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix floating-point asymmetry of AMR cell-centered prolong/restrict operators #114

Merged
merged 6 commits into from
Apr 25, 2018

Conversation

felker
Copy link
Contributor

@felker felker commented Apr 24, 2018

Description

While I wasn't planning on immediately fixing the AMR asymmetries that manifested even after merging #112, the issues turned out to be limited to two separate causes that were easy to address:

  1. Using MPI with AMR is not reproducible nor does it preserve symmetry. @tomidakn explained that it is due to the derefinement counter:

This variable counts the number of the continuous steps that the derefinement criterion is satisfied. And when it reaches a threshold, it flags the MeshBlock to be derefined. But... I do not transfer this number when MeshBlocks are transferred to another node, and the load balancing is intrinsically asymmetric. So this affects the timing of derefinement asymmetrically

  1. The stencil calculations in src/mesh/mesh_refinement.cpp had an implicit directional bias due to left-to-right floating point associativity, much like PPM in Fix general asymmetry of x1f, dx1v Coordinates terms; ensure exact float64 symmetry of PPM stencils #98.

I only fixed the simple second issue here. Also, I copied and enrolled the user RefinementCondition() from the pgen/kh.cpp file to pgen/rt.cpp.

Testing and validation

AMR tested with the Rayleigh-Taylor problem with 2 levels of refinement (and mesh/derfef_count=5) on a 500x1000 base mesh with 50x50 MeshBlocks. Output was loaded with athena_read.py-- although it does not have bitwise correctness re: coordinate positions, it can compute the exact symmetry differences, which were exactly 0.0 for the PR version of the code:
rt-ppm-nx1-amr-fixed-nompi.pdf
No MPI used; icc 18 with -fp-model strict enabled.

To-do

  • Modify the R-T RefinementCondition() to some problem-specific threshold
  • Test in 1D and 3D
  • Fix reproducibility and symmetry issues with MPI+AMR
  • Test and fix any MHD+AMR asymmetries

Relevant lines in mesh_refinement.cpp have signed comments to
emphasize that strict operator associativity must be followed
Might not be necessary, since the fine() stencils arent biased
along a direction in each dimension. Imrpove readability
@felker felker added bug Broken functionality or unexpected result SMR/AMR Relating to static and adaptive mesh refinement labels Apr 24, 2018
@felker felker changed the title Fix floating-point asymmetry of AMR cell-centered prolong/restriction operationrs Fix floating-point asymmetry of AMR cell-centered prolong/restrict operators Apr 24, 2018
@felker
Copy link
Contributor Author

felker commented Apr 25, 2018

For comparison, here are the same AMR R-T results produce by the current version of Athena++:

@felker felker merged commit e37a4ac into master Apr 25, 2018
@felker felker deleted the amr-symmetry branch April 25, 2018 13:45
@felker felker removed their assignment May 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Broken functionality or unexpected result SMR/AMR Relating to static and adaptive mesh refinement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants