Moist BC wave + slab + MPI

[LenkaNovak]

• will address stage 2 and 3 of Moist Baroclinic Wave terraplanet with basic MPI #39

TODO

• automate copying in ClimaAtmos driver + writing wrappers
• implement MPI init in coupler
• couple slab
• clean up code (e.g., "the same Space type but not the same instance" error) Enable broadcasting with the same spaces and different instances #45 - found a work around, so not needed
• merge CA PRs for BCs bring back external boundary conditons ClimaAtmos.jl#257
• merge CC PR for spatially varying BC specification
• merge CC PR for TempestRemap online application #CC604
• include the new (ClimaSimulations-excluding) ClimaAtmos interface
• implement correct surface drag
• conservation checks

• peformance checks
  • ctrl setup: h_elem = 4, z_elem = 10, nq = 5, time_end = 1e4, dt = 1e2 (all time quoted in secs)
  • MacOS:
    • atmos only solve!: 77.5s (as in ClimaAtmos) [no output save]
    • atmos only step!: 90.7s [output saved at every dt]
    • coupled step!: 90.9s [output saved at every dt]
    • coupled step!: 74,4s [no output save]
  • HPC cluster with MPI:

• physical checks (crashes at day 17d, even without the additional heating blob, but good enough for now, drag also checked)

- comparison between runs of different drag coefficient values (Cd) looks reasonable (NB: drag values are difficult to check due to a field type conversion issue)

- (comparison with literature of the uncouple case is in ClimaAtmos)

The clean-up PR will include

• introduce coupler-specific interface abstractions Coupler-specific Interface #44
• substitute the latest version of ClimaAtmos interface (TC.jl, limiters, perf improvements etc)
• buildkite hook
• find in literature / produce using existing models coupled cases to compare

[LenkaNovak]

@sriharshakandala nice job on ClimaComms.jl implementation! 🚀 I was wondering what's the neatest way of tracking sim time if we want to add performance checks? Do you use @btime?

[sriharshakandala]

@sriharshakandala nice job on ClimaComms.jl implementation! 🚀 I was wondering what's the neatest way of tracking sim time if we want to add performance checks? Do you use @btime?

For measuring total simulation times (order of a few seconds or greater), generally speaking, @elapsed should suffice! https://github.com/CliMA/ClimaCore.jl/blob/main/examples/hybrid/driver.jl#L199

@btime is better for microbenchmarking (small pieces of code which take a few ns/micro secs)
@btime is also impractical for measuring full simulation times as it runs the function multiple times to gather statistics!

[LenkaNovak]

@sriharshakandala nice job on ClimaComms.jl implementation! 🚀 I was wondering what's the neatest way of tracking sim time if we want to add performance checks? Do you use @btime?

For measuring total simulation times (order of a few seconds or greater), generally speaking, @elapsed should suffice! https://github.com/CliMA/ClimaCore.jl/blob/main/examples/hybrid/driver.jl#L199

@btime is better for microbenchmarking (small pieces of code which take a few ns/micro secs) @btime is also impractical for measuring full simulation times as it runs the function multiple times to gather statistics!

Sounds good, thank you!

[szy21]

Suggested change

	@. Yₜ.c.uₕ += ᶜdivᵥ(ᶠK_E * ᶠinterp(ᶜρ) * ᶠgradᵥ(Y.c.uₕ / ᶜρ))
	@. Yₜ.c.uₕ += ᶜdivᵥ(ᶠK_E * ᶠgradᵥ(Y.c.uₕ))

[LenkaNovak]

bors r+

[bors]

Build succeeded:

• ci 1.7 - macOS-latest
• ci 1.7 - ubuntu-latest
• ci 1.7 - windows-latest
• docbuild
• format