Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Catch Up Steps Not Working Properly #10609

Closed
friedmud opened this issue Jan 25, 2018 · 1 comment
Closed

Catch Up Steps Not Working Properly #10609

friedmud opened this issue Jan 25, 2018 · 1 comment
Assignees
@friedmud
Labels
C: Framework P: normal A defect affecting operation with a low possibility of significantly affects. T: defect An anomaly, which is anything that deviates from expectations.

Comments

@friedmud
Copy link
Contributor

Rationale

Catch Up Steps in TransientMultiApp are not working properly. They are taking more iterations than necessary and are causing individual apps to go out of sync.

Description

Here is a simple test. A Master has two sub-apps. One of the sub-apps is set to fail the second timestep. It should then take two (exactly TWO!) catch-up steps to catch up to the Master and the other Sub. Instead it takes three!!

Time Step  0, time = 0
                dt = 0

Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  1, time = 0.2
                dt = 0.2
 0 Nonlinear |R| = 3.316625e+00
      0 Linear |R| = 3.316625e+00
      1 Linear |R| = 7.030567e-02
      2 Linear |R| = 1.191402e-03
      3 Linear |R| = 2.803504e-05
 1 Nonlinear |R| = 2.803509e-05
      0 Linear |R| = 2.803509e-05
      1 Linear |R| = 1.246573e-06
      2 Linear |R| = 1.215561e-08
      3 Linear |R| = 1.967839e-10
 2 Nonlinear |R| = 1.972189e-10
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  1, time = 0.2
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 3.316625e+00
sub_app0:       0 Linear |R| = 3.316625e+00
sub_app0:       1 Linear |R| = 7.030567e-02
sub_app0:       2 Linear |R| = 1.191402e-03
sub_app0:       3 Linear |R| = 2.803504e-05
sub_app0:  1 Nonlinear |R| = 2.803509e-05
sub_app0:       0 Linear |R| = 2.803509e-05
sub_app0:       1 Linear |R| = 1.246573e-06
sub_app0:       2 Linear |R| = 1.215561e-08
sub_app0:       3 Linear |R| = 1.967839e-10
sub_app0:  2 Nonlinear |R| = 1.972189e-10
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  1, time = 0.2
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 3.316625e+00
sub_app1:       0 Linear |R| = 3.316625e+00
sub_app1:       1 Linear |R| = 7.030567e-02
sub_app1:       2 Linear |R| = 1.191402e-03
sub_app1:       3 Linear |R| = 2.803504e-05
sub_app1:  1 Nonlinear |R| = 2.803509e-05
sub_app1:       0 Linear |R| = 2.803509e-05
sub_app1:       1 Linear |R| = 1.246573e-06
sub_app1:       2 Linear |R| = 1.215561e-08
sub_app1:       3 Linear |R| = 1.967839e-10
sub_app1:  2 Nonlinear |R| = 1.972189e-10
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END


No Transfers on TIMESTEP_END To MultiApps


Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  2, time = 0.4
                dt = 0.2
 0 Nonlinear |R| = 1.958977e-01
      0 Linear |R| = 1.958977e-01
      1 Linear |R| = 1.397686e-02
      2 Linear |R| = 2.932666e-04
      3 Linear |R| = 4.728870e-06
      4 Linear |R| = 6.153870e-08
 1 Nonlinear |R| = 6.297277e-08
      0 Linear |R| = 6.297277e-08
      1 Linear |R| = 1.376346e-09
      2 Linear |R| = 1.625438e-11
      3 Linear |R| = 4.474381e-13
 2 Nonlinear |R| = 4.474956e-13
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  2, time = 0.4
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 1.958977e-01
sub_app0:       0 Linear |R| = 1.958977e-01
sub_app0:       1 Linear |R| = 1.397686e-02
sub_app0:       2 Linear |R| = 2.932666e-04
sub_app0:       3 Linear |R| = 4.728870e-06
sub_app0:       4 Linear |R| = 6.153870e-08
sub_app0:  1 Nonlinear |R| = 6.297277e-08
sub_app0:       0 Linear |R| = 6.297277e-08
sub_app0:       1 Linear |R| = 1.376346e-09
sub_app0:       2 Linear |R| = 1.625438e-11
sub_app0:       3 Linear |R| = 4.474381e-13
sub_app0:  2 Nonlinear |R| = 4.474956e-13
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  2, time = 0.4
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 1.958977e-01
sub_app1:       0 Linear |R| = 1.958977e-01
sub_app1:       1 Linear |R| = 1.397686e-02
sub_app1:       2 Linear |R| = 2.932666e-04
sub_app1:       3 Linear |R| = 4.728870e-06
sub_app1:       4 Linear |R| = 6.153870e-08
sub_app1:  1 Nonlinear |R| = 6.297277e-08
sub_app1:       0 Linear |R| = 6.297277e-08
sub_app1:       1 Linear |R| = 1.376346e-09
sub_app1:       2 Linear |R| = 1.625438e-11
sub_app1:       3 Linear |R| = 4.474381e-13
sub_app1:  2 Nonlinear |R| = 4.474956e-13
sub_app1:  Solve Did NOT Converge!




*** Warning ***
sub_app1 failed to converge!



Starting Catch Up!
Solving sub_app catch up step 0
sub_app1: 
sub_app1: Time Step  2, time = 0.3
sub_app1:                 dt = 0.1
sub_app1:  0 Nonlinear |R| = 1.958977e-01
sub_app1:       0 Linear |R| = 1.958977e-01
sub_app1:       1 Linear |R| = 1.161043e-02
sub_app1:       2 Linear |R| = 1.583221e-04
sub_app1:       3 Linear |R| = 2.569333e-06
sub_app1:       4 Linear |R| = 3.704563e-08
sub_app1:  1 Nonlinear |R| = 4.009467e-08
sub_app1:       0 Linear |R| = 4.009467e-08
sub_app1:       1 Linear |R| = 7.407040e-10
sub_app1:       2 Linear |R| = 1.102053e-11
sub_app1:       3 Linear |R| = 2.778348e-13
sub_app1:  2 Nonlinear |R| = 2.778871e-13
sub_app1:  Solve Converged!
Solving sub_app catch up step 1
sub_app1: 
sub_app1: Time Step  3, time = 0.4
sub_app1:                 dt = 0.1
sub_app1:  0 Nonlinear |R| = 7.522512e-02
sub_app1:       0 Linear |R| = 7.522512e-02
sub_app1:       1 Linear |R| = 5.238928e-03
sub_app1:       2 Linear |R| = 7.699752e-05
sub_app1:       3 Linear |R| = 9.726709e-07
sub_app1:       4 Linear |R| = 1.812178e-08
sub_app1:  1 Nonlinear |R| = 1.907928e-08
sub_app1:       0 Linear |R| = 1.907928e-08
sub_app1:       1 Linear |R| = 4.545916e-10
sub_app1:       2 Linear |R| = 6.288271e-12
sub_app1:       3 Linear |R| = 1.341262e-13
sub_app1:  2 Nonlinear |R| = 1.340724e-13
sub_app1:  Solve Converged!
Solving sub_app catch up step 2
sub_app1: 
sub_app1: Time Step  4, time = 0.5
sub_app1:                 dt = 0.1
sub_app1:  0 Nonlinear |R| = 3.656925e-02
sub_app1:       0 Linear |R| = 3.656925e-02
sub_app1:       1 Linear |R| = 2.614478e-03
sub_app1:       2 Linear |R| = 3.808703e-05
sub_app1:       3 Linear |R| = 4.636082e-07
sub_app1:       4 Linear |R| = 8.683308e-09
sub_app1:  1 Nonlinear |R| = 9.742874e-09
sub_app1:       0 Linear |R| = 9.742874e-09
sub_app1:       1 Linear |R| = 2.184415e-10
sub_app1:       2 Linear |R| = 2.889074e-12
sub_app1:       3 Linear |R| = 6.572418e-14
sub_app1:  2 Nonlinear |R| = 6.576686e-14
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END


No Transfers on TIMESTEP_END To MultiApps


Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  3, time = 0.6
                dt = 0.2
 0 Nonlinear |R| = 4.963758e-02
      0 Linear |R| = 4.963758e-02
      1 Linear |R| = 4.294719e-03
      2 Linear |R| = 8.631213e-05
      3 Linear |R| = 1.088588e-06
      4 Linear |R| = 2.018785e-08
 1 Nonlinear |R| = 2.279516e-08
      0 Linear |R| = 2.279516e-08
      1 Linear |R| = 5.030274e-10
      2 Linear |R| = 8.656245e-12
      3 Linear |R| = 2.641904e-13
      4 Linear |R| = 3.001497e-15
 2 Nonlinear |R| = 3.249808e-15
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  3, time = 0.6
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 4.963758e-02
sub_app0:       0 Linear |R| = 4.963758e-02
sub_app0:       1 Linear |R| = 4.294719e-03
sub_app0:       2 Linear |R| = 8.631213e-05
sub_app0:       3 Linear |R| = 1.088588e-06
sub_app0:       4 Linear |R| = 2.018785e-08
sub_app0:  1 Nonlinear |R| = 2.279516e-08
sub_app0:       0 Linear |R| = 2.279516e-08
sub_app0:       1 Linear |R| = 5.030274e-10
sub_app0:       2 Linear |R| = 8.656245e-12
sub_app0:       3 Linear |R| = 2.641904e-13
sub_app0:       4 Linear |R| = 3.001497e-15
sub_app0:  2 Nonlinear |R| = 3.249808e-15
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  5, time = 0.7
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 1.824928e-02
sub_app1:       0 Linear |R| = 1.824928e-02
sub_app1:       1 Linear |R| = 1.623867e-03
sub_app1:       2 Linear |R| = 3.028496e-05
sub_app1:       3 Linear |R| = 3.834879e-07
sub_app1:       4 Linear |R| = 7.244099e-09
sub_app1:  1 Nonlinear |R| = 8.515781e-09
sub_app1:       0 Linear |R| = 8.515781e-09
sub_app1:       1 Linear |R| = 1.988662e-10
sub_app1:       2 Linear |R| = 3.067191e-12
sub_app1:       3 Linear |R| = 9.048956e-14
sub_app1:       4 Linear |R| = 1.183486e-15
sub_app1:  2 Nonlinear |R| = 1.651119e-15
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END

Notice how for the final timestep the two sub-apps are actually at two different times! This is really not good!

Impact

This may have been impacting every MultiApp solve that uses catch-up steps... actually generating incorrect / weird solutions.
@friedmud friedmud added C: Framework T: defect An anomaly, which is anything that deviates from expectations. P: normal A defect affecting operation with a low possibility of significantly affects. labels Jan 25, 2018
@friedmud friedmud self-assigned this Jan 25, 2018
friedmud added a commit to friedmud/moose that referenced this issue Jan 25, 2018
@friedmud
Fix catch up steps refs idaholab#10609
99d835e
@friedmud friedmud mentioned this issue Jan 25, 2018
Merged
@friedmud
Copy link
Contributor Author

With #10610 this same simple test now looks correct. I have added a test to make sure it stays this way!

Time Step  0, time = 0
                dt = 0

Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  1, time = 0.2
                dt = 0.2
 0 Nonlinear |R| = 3.316625e+00
      0 Linear |R| = 3.316625e+00
      1 Linear |R| = 7.030567e-02
      2 Linear |R| = 1.191402e-03
      3 Linear |R| = 2.803504e-05
 1 Nonlinear |R| = 2.803509e-05
      0 Linear |R| = 2.803509e-05
      1 Linear |R| = 1.246573e-06
      2 Linear |R| = 1.215561e-08
      3 Linear |R| = 1.967839e-10
 2 Nonlinear |R| = 1.972189e-10
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  1, time = 0.2
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 3.316625e+00
sub_app0:       0 Linear |R| = 3.316625e+00
sub_app0:       1 Linear |R| = 7.030567e-02
sub_app0:       2 Linear |R| = 1.191402e-03
sub_app0:       3 Linear |R| = 2.803504e-05
sub_app0:  1 Nonlinear |R| = 2.803509e-05
sub_app0:       0 Linear |R| = 2.803509e-05
sub_app0:       1 Linear |R| = 1.246573e-06
sub_app0:       2 Linear |R| = 1.215561e-08
sub_app0:       3 Linear |R| = 1.967839e-10
sub_app0:  2 Nonlinear |R| = 1.972189e-10
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  1, time = 0.2
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 3.316625e+00
sub_app1:       0 Linear |R| = 3.316625e+00
sub_app1:       1 Linear |R| = 7.030567e-02
sub_app1:       2 Linear |R| = 1.191402e-03
sub_app1:       3 Linear |R| = 2.803504e-05
sub_app1:  1 Nonlinear |R| = 2.803509e-05
sub_app1:       0 Linear |R| = 2.803509e-05
sub_app1:       1 Linear |R| = 1.246573e-06
sub_app1:       2 Linear |R| = 1.215561e-08
sub_app1:       3 Linear |R| = 1.967839e-10
sub_app1:  2 Nonlinear |R| = 1.972189e-10
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END


No Transfers on TIMESTEP_END To MultiApps


Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  2, time = 0.4
                dt = 0.2
 0 Nonlinear |R| = 1.958977e-01
      0 Linear |R| = 1.958977e-01
      1 Linear |R| = 1.397686e-02
      2 Linear |R| = 2.932666e-04
      3 Linear |R| = 4.728870e-06
      4 Linear |R| = 6.153870e-08
 1 Nonlinear |R| = 6.297277e-08
      0 Linear |R| = 6.297277e-08
      1 Linear |R| = 1.376346e-09
      2 Linear |R| = 1.625438e-11
      3 Linear |R| = 4.474381e-13
 2 Nonlinear |R| = 4.474956e-13
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  2, time = 0.4
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 1.958977e-01
sub_app0:       0 Linear |R| = 1.958977e-01
sub_app0:       1 Linear |R| = 1.397686e-02
sub_app0:       2 Linear |R| = 2.932666e-04
sub_app0:       3 Linear |R| = 4.728870e-06
sub_app0:       4 Linear |R| = 6.153870e-08
sub_app0:  1 Nonlinear |R| = 6.297277e-08
sub_app0:       0 Linear |R| = 6.297277e-08
sub_app0:       1 Linear |R| = 1.376346e-09
sub_app0:       2 Linear |R| = 1.625438e-11
sub_app0:       3 Linear |R| = 4.474381e-13
sub_app0:  2 Nonlinear |R| = 4.474956e-13
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  2, time = 0.4
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 1.958977e-01
sub_app1:       0 Linear |R| = 1.958977e-01
sub_app1:       1 Linear |R| = 1.397686e-02
sub_app1:       2 Linear |R| = 2.932666e-04
sub_app1:       3 Linear |R| = 4.728870e-06
sub_app1:       4 Linear |R| = 6.153870e-08
sub_app1:  1 Nonlinear |R| = 6.297277e-08
sub_app1:       0 Linear |R| = 6.297277e-08
sub_app1:       1 Linear |R| = 1.376346e-09
sub_app1:       2 Linear |R| = 1.625438e-11
sub_app1:       3 Linear |R| = 4.474381e-13
sub_app1:  2 Nonlinear |R| = 4.474956e-13
sub_app1:  Solve Did NOT Converge!




*** Warning ***
sub_app1 failed to converge!



Starting Catch Up!
Solving sub_app catch up step 0
sub_app1: 
sub_app1: Time Step  2, time = 0.3
sub_app1:                 dt = 0.1
sub_app1:  0 Nonlinear |R| = 1.958977e-01
sub_app1:       0 Linear |R| = 1.958977e-01
sub_app1:       1 Linear |R| = 1.161043e-02
sub_app1:       2 Linear |R| = 1.583221e-04
sub_app1:       3 Linear |R| = 2.569333e-06
sub_app1:       4 Linear |R| = 3.704563e-08
sub_app1:  1 Nonlinear |R| = 4.009467e-08
sub_app1:       0 Linear |R| = 4.009467e-08
sub_app1:       1 Linear |R| = 7.407040e-10
sub_app1:       2 Linear |R| = 1.102053e-11
sub_app1:       3 Linear |R| = 2.778348e-13
sub_app1:  2 Nonlinear |R| = 2.778871e-13
sub_app1:  Solve Converged!
Solving sub_app catch up step 1
sub_app1: 
sub_app1: Time Step  3, time = 0.4
sub_app1:                 dt = 0.1
sub_app1:  0 Nonlinear |R| = 7.522512e-02
sub_app1:       0 Linear |R| = 7.522512e-02
sub_app1:       1 Linear |R| = 5.238928e-03
sub_app1:       2 Linear |R| = 7.699752e-05
sub_app1:       3 Linear |R| = 9.726709e-07
sub_app1:       4 Linear |R| = 1.812178e-08
sub_app1:  1 Nonlinear |R| = 1.907928e-08
sub_app1:       0 Linear |R| = 1.907928e-08
sub_app1:       1 Linear |R| = 4.545916e-10
sub_app1:       2 Linear |R| = 6.288271e-12
sub_app1:       3 Linear |R| = 1.341262e-13
sub_app1:  2 Nonlinear |R| = 1.340724e-13
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END


No Transfers on TIMESTEP_END To MultiApps


Backing Up MultiApps
Waiting For Other Processors To Finish
Jobs complete: 1/1
Finished Backing Up MultiApps


Time Step  3, time = 0.6
                dt = 0.2
 0 Nonlinear |R| = 4.963758e-02
      0 Linear |R| = 4.963758e-02
      1 Linear |R| = 4.294719e-03
      2 Linear |R| = 8.631213e-05
      3 Linear |R| = 1.088588e-06
      4 Linear |R| = 2.018785e-08
 1 Nonlinear |R| = 2.279516e-08
      0 Linear |R| = 2.279516e-08
      1 Linear |R| = 5.030274e-10
      2 Linear |R| = 8.656245e-12
      3 Linear |R| = 2.641904e-13
      4 Linear |R| = 3.001497e-15
 2 Nonlinear |R| = 3.249808e-15
 Solve Converged!

No Transfers on TIMESTEP_END To MultiApps


Executing MultiApps on TIMESTEP_END
Solving MultiApp sub_app
Solving Normal Step!
sub_app0: 
sub_app0: Time Step  3, time = 0.6
sub_app0:                 dt = 0.2
sub_app0:  0 Nonlinear |R| = 4.963758e-02
sub_app0:       0 Linear |R| = 4.963758e-02
sub_app0:       1 Linear |R| = 4.294719e-03
sub_app0:       2 Linear |R| = 8.631213e-05
sub_app0:       3 Linear |R| = 1.088588e-06
sub_app0:       4 Linear |R| = 2.018785e-08
sub_app0:  1 Nonlinear |R| = 2.279516e-08
sub_app0:       0 Linear |R| = 2.279516e-08
sub_app0:       1 Linear |R| = 5.030274e-10
sub_app0:       2 Linear |R| = 8.656245e-12
sub_app0:       3 Linear |R| = 2.641904e-13
sub_app0:       4 Linear |R| = 3.001497e-15
sub_app0:  2 Nonlinear |R| = 3.249808e-15
sub_app0:  Solve Converged!
Solving Normal Step!
sub_app1: 
sub_app1: Time Step  4, time = 0.6
sub_app1:                 dt = 0.2
sub_app1:  0 Nonlinear |R| = 3.656925e-02
sub_app1:       0 Linear |R| = 3.656925e-02
sub_app1:       1 Linear |R| = 3.227364e-03
sub_app1:       2 Linear |R| = 6.210491e-05
sub_app1:       3 Linear |R| = 7.795711e-07
sub_app1:       4 Linear |R| = 1.476959e-08
sub_app1:  1 Nonlinear |R| = 1.697712e-08
sub_app1:       0 Linear |R| = 1.697712e-08
sub_app1:       1 Linear |R| = 4.204287e-10
sub_app1:       2 Linear |R| = 6.318350e-12
sub_app1:       3 Linear |R| = 1.924445e-13
sub_app1:       4 Linear |R| = 2.286338e-15
sub_app1:  2 Nonlinear |R| = 2.578240e-15
sub_app1:  Solve Converged!
Successfully Solved MultiApp sub_app.
Jobs complete: 1/1
Waiting For Other Processors To Finish
Finished Executing MultiApps on TIMESTEP_END

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@friedmud friedmud

Labels
C: Framework P: normal A defect affecting operation with a low possibility of significantly affects. T: defect An anomaly, which is anything that deviates from expectations.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@friedmud