Skip to content

Subcycling

Jump to bottom Edit New page
Roland Haas edited this page May 8, 2024 · 10 revisions

Refinement level groups

CarpetX collects refinement levels that can be acted one at a given moment in the simulation in a structure active_levels which is a an interval of refinement levels [min_level, max_level). During the EVOL timebin this set is constructed as all the refinement levels that are at the current time and share a common time refinement factor (indicated by the subcylcing_level member of leveldata).

Which levels are considered active can change from timebin to timebin, for example some time bins (CCTK_TERMINATE for example) do not allow any access to data so no level is active, while other time bins (CCTK_ANALYSIS) may ignore the the subcycling state and group levels exclusively by time sine they do not involve any time stepping.

Evolution time step and communication pattern

For the purpose of this discussion is is most convenient to think of all levels that share a time step size as a single group, basically some multi-patch system with only Cartesian patches.

Assume we have 3 groups of levels G0, G1, G2 that are all initially aligned in time and refined in time by factors of 2 and time step on G0 being Delta_t=1.0. There are 3 data movement operations to consider: interprocessors ghost zone exchange (IP), spatial prolongation from coarse to fine (PR), interpolation in time using the prolongated RK values (IT) and restriction from fine to coarse (RE). Additionally there are physical boundary conditions (BC)

A full set of operations in the first two time steps for an RK4 scheme is then (parenthesis indicate sources of data):

  1. ODE step G0
    1. compute Y1(G0), IP(Y1(G0)), PR(Y1(G0)), BC(Y1(G0)), RE(Y1(G0))
    2. compute Y2(G0), IP(Y2(G0)), PR(Y1(G0)), BC(Y2(G0)), RE(Y1(G0))
    3. compute Y3(G0), IP(Y3(G0)), PR(Y1(G0)), BC(Y3(G0)), RE(Y1(G0))
    4. compute Y4(G0), IP(Y4(G0)), PR(Y1(G0)), BC(Y4(G0)), RE(Y1(G0))
    5. update G0 time to t=1.0
  2. ODE step G1
    1. PR(Ys(G0))
    2. compute Y1(G1), IP(Y1(G1)), PR(Y1(G1)), IT(Ys(G1)), RE(Y1(G1))
    3. compute Y2(G1), IP(Y2(G1)), PR(Y2(G1)), IT(Ys(G1)), RE(Y2(G1))
    4. compute Y3(G1), IP(Y3(G1)), PR(Y3(G1)), IT(Ys(G1)), RE(Y3(G1))
    5. compute Y4(G1), IP(Y4(G1)), PR(Y4(G1)), IT(Ys(G1)), RE(Y4(G1))
    6. update G1 time to t=0.5
  3. ODE step G2
    1. PR(Ys(G1))
    2. compute Y1(G2), IP(Y1(G2)), PR(Y1(G2)), IT(Ys(G2)), RE(Y1(G2))
    3. compute Y2(G2), IP(Y2(G2)), PR(Y2(G2)), IT(Ys(G2)), RE(Y2(G2))
    4. compute Y3(G2), IP(Y3(G2)), PR(Y3(G2)), IT(Ys(G2)), RE(Y3(G2))
    5. compute Y4(G2), IP(Y4(G2)), PR(Y4(G2)), IT(Ys(G2)), RE(Y4(G2))
    6. update G2 time to t=0.25
  4. ODE step G2
    1. compute Y1(G2), IP(Y1(G2)), PR(Y1(G2)), IT(Ys(G2)), RE(Y1(G2))
    2. compute Y2(G2), IP(Y2(G2)), PR(Y2(G2)), IT(Ys(G2)), RE(Y2(G2))
    3. compute Y3(G2), IP(Y3(G2)), PR(Y3(G2)), IT(Ys(G2)), RE(Y3(G2))
    4. compute Y4(G2), IP(Y4(G2)), PR(Y4(G2)), IT(Ys(G2)), RE(Y4(G2))
    5. update time G2 to 0.5
  5. RE(G2(t=0.5)) -> G1(t=0.5)
  6. ODE step G1
    1. compute Y1(G1), IP(Y1(G1)), PR(Y1(G1)), IT(Ys(G1)), RE(Y1(G1))
    2. compute Y2(G1), IP(Y2(G1)), PR(Y2(G1)), IT(Ys(G1)), RE(Y2(G1))
    3. compute Y3(G1), IP(Y3(G1)), PR(Y3(G1)), IT(Ys(G1)), RE(Y3(G1))
    4. compute Y4(G1), IP(Y4(G1)), PR(Y4(G1)), IT(Ys(G1)), RE(Y4(G1))
    5. update time G1 to 1.0
  7. RE(G1(t=1.0)) -> G0(t=1.0)
  8. ODE step G2
    1. PR(Ys(G1))
    2. compute Y1(G2), IP(Y1(G2)), PR(Y1(G2)), IT(Ys(G2)), RE(Y1(G2))
    3. compute Y2(G2), IP(Y2(G2)), PR(Y2(G2)), IT(Ys(G2)), RE(Y2(G2))
    4. compute Y3(G2), IP(Y3(G2)), PR(Y3(G2)), IT(Ys(G2)), RE(Y3(G2))
    5. compute Y4(G2), IP(Y4(G2)), PR(Y4(G2)), IT(Ys(G2)), RE(Y4(G2))
    6. update time G2 to 0.75
  9. ODE step G2
    1. compute Y1(G2), IP(Y1(G2)), PR(Y1(G2)), IT(Ys(G2)), RE(Y1(G2))
    2. compute Y2(G2), IP(Y2(G2)), PR(Y2(G2)), IT(Ys(G2)), RE(Y2(G2))
    3. compute Y3(G2), IP(Y3(G2)), PR(Y3(G2)), IT(Ys(G2)), RE(Y3(G2))
    4. compute Y4(G2), IP(Y4(G2)), PR(Y4(G2)), IT(Ys(G2)), RE(Y4(G2))
    5. update time G2 to 1.0
  10. RE(G2(t=1.0)) -> G1(t=1.0)
  11. RE(G1(t=1.0)) -> G0(t=1.0)

Analysis step and communication pattern

The ANALYSIS timebin traditionally runs after the simulation has fully progressed to a given time, that is until after all restriction has happened.

So the traditional algorithm is much simpler:

  1. find all refinement levels at a given time, ignoring their subcycling marker and timestep size
  2. call all scheduled functions
  3. there is no communication, no prolongation, no restriction, no interprocessor ghost exchange

This means in particular:

  1. one cannot compute derivatives (or any other stencil based operation) in ANALYSIS, this includes things like the Weyl scalars, all calculations must be local to a cell / vertex / face
  2. ordering of refinement levels is irrelevant
  3. in the same Cactus iteration a scheduled function may be called on different refinement levels in EVOL than in ANALYSIS.

Tasks

We will meet Mondays 9am US Central (Chicago) time via Zoom for a work meeting.

  • verify and possible correct how active_levels is used
  • verify that SYNC happen in the manner expected (once per group)
  • add restrict outside of ODESolver step
    • design correct condition for this restrict
  • add interpolation in time for interpolator
    • possibly support using RK coeffs for this interpolation -> requires tight coupling between interpolator (driver) and time stepper
Add a custom footer
Add a custom sidebar
Clone this wiki locally