Revamp Buildkite CI tests

[LenkaNovak]

We want our current buildkite tests to reflect the latest AMIP advancements. In particular, we want to test:

3 tiers of runs

• shortruns (regular CI): unit tests, MSE tables, performance, short simulations (wall: ~20-30mins)
• midruns: run manually if MSE breaks, longer + higher resolution than shortruns, (wall: ~2h) testing the following:
  • diagnostic / prognostic EDMF (when available)
  • topography
  • insolation types
  • radiation types
  • albedo types
  • surface flux types
• longruns: benchmarks, longer simulations testing the extent of our capabilities (100s days) (wall: ~6h)

To Do

1. add MSE tables to shortruns
2. split current CI into 3 pipelines described above
   a. set up infrastructure for short and longruns to be run automatically, midruns manually
3. go through revamped CI, make sure runs are meaningful tests, not duplicated, test both physical results and performance, etc

Questions

• which quantities do we want to add MSE tables for?
  • water conservation, energy conservation?
• how to run buildkite pipeline manually only?