Adds LIGHT super-cycling capability

[pwolfram]

This merge provides the capability to do LIGHT super-cycling.

An example use case is

namelist changes

config_dt = '00:10:00'
config_AM_lagrPartTrack_compute_interval = '0000_01:00:00'

example particle file changes:

dtParticle = 3600 # time step for particles in s
timeIntegration = 4 #use RK4

[pwolfram]

Testing

A spun-up 32km SOMA test case was used with two cases over 30 days of simulation:

1. 300s particle timestep at 600s compute time step (sub-cycled) using RK2
2. 3600s particle timestep at 1 hr compute interval (super-cycled) using RK4

The error is greatest near regions of strongly varying flow, e.g., in initial regions near the shelf as well as in the bifurcation points of the jet; in general error is reasonably small and Results are reasonable and the computational step speedup is approximately a factor of 6 for the compute step.

Speedup for G RRS18to6 (on grizzly)

This provides a drastic speedup in the compute step of LIGHT, e.g., for speed up of a factor of 6 relative to use of RK2 with dtParticle=300, as previously used. Results are consistent with expected cost corresponding to computation frequency, time step frequency, and choice of Runge Kutta method.

Speed up can be tuned relative to accuracy via choice of dtParticle, timeIntegration, and config_am_lagrparttrack_compute_interval.

This capability will drastically reduce the computational cost of LIGHT's compute step in the 18to6, as suggests a 6 hr time step with a three-day computational frequency using RK4 would provide roughly a factor of 120X speedup relative to computing particles at the native 6-min timestep frequencies.

[pwolfram]

@vanroekel, files from this branch will be needed for application of LIGHT to the 18to6.

[pwolfram]

I've confirmed that the LIGHT and nightly test suites pass relative to ocean/develop.

[mark-petersen]

Passes nightly regression suite on grizzly with gnu/opt and intel/debug. All bfb with previous commit, as expected.

@pwolfram Does this need any corresponding changes in E3SM, such as any change in namelist flag settings?

[pwolfram]

@mark-petersen, we will need to change namelist and streams files for E3SM once we have the performance fully sorted out. I'm currently testing an approach to reduce restart cost. I don't think that effort should hold up this PR.

[vanroekel]

@mark-petersen and @pwolfram do the potential name list changes imply that LIGHT will be on by default in E3SM runs? If so, I think this requires a much broader discussion than here. I’m pretty uncomfortable with LIGHT on by default for E3SM.

[pwolfram]

@vanroekel, I think there quite a few challenges that would need surmounted before it could be on by default. I'm also not comfortable myself with this notion until I can demonstrate enhanced performance for IO and then there are other software engineering tasks that would also likely need to be completed first too. I'm afraid we are on a tangent here that is not pertinent to this PR.

[mark-petersen]

I get it. The take-home message is that we can merge this PR and not have any repercussions in E3SM for standard configurations. Correct?

[pwolfram]

Correct

…

________________________________ From: Mark Petersen <notifications@github.com> Sent: Wednesday, January 10, 2018 5:37:31 PM To: MPAS-Dev/MPAS Cc: Phillip J. Wolfram; Mention Subject: Re: [MPAS-Dev/MPAS] Adds LIGHT super-cycling capability (#1484) I get it. The take-home message is that we can merge this PR and not have any repercussions in E3SM for standard configurations. Correct? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1484 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEGMrZ2tPWnsZ2bUd6Fl7t4fxE45a8aoks5tJVfLgaJpZM4RYX3->.

[mark-petersen]

This PR looks fine. I will merge it after E3SM-Project/E3SM#1971 is in ACME, probably later today.

[mark-petersen]

@vanroekel mentioned that this PR works on grizzly but had a problem on other machines. @pwolfram, I'd like to postpone this merge until we can confirm that it works on machines like theta and titan.

[mark-petersen]

This will be included in the next batch of small fixes for MPAS in E3SM, most likely on Feb 8.