revamp longruns buildkite pipeline

[LenkaNovak]

We need to revamp our longruns to reflect the current stable targets in ClimaAtmos. The current target in ClimaAtmos is longrun_aquaplanet_rhoe_equilmoist_nz63_0M_55km_rs35km_clearsky_tvinsolation (though the specifications of this may change due to unphysical features - see below). For now we use this current target to setup our testing pipeline.

Components

• integration hierarchy - testing that atmos surface initialization is consistent with the coupler surface for coarse aquaplanet, and includes conservation checks (this can be eventually be moved to the second tier CI as part of Revamp Buildkite CI tests #406 ). The runs involve
  • slabplanet_aqua_atmos_sf_nocouple (atmos fluxes at every RK stage, prescribed SSTs from atmos)
  • slabplanet_aqua_atmos_sf_couple (atmos fluxes at every RK stage, prescribed SSTs from coupler)
  • slabplanet_aqua_coupler_sf (coupler fluxes at every coupling step, prescribed SSTs from coupler)
  • slabplanet_aqua_coupler_sf_evolve_ocn (coupler fluxes at every coupling step, slab ocean)
  • slabplanet_terra (coupler fluxes at every coupling step, bucket surface)
  • slabplanet_coupler_sf_evolve_ocn (coupler fluxes at every coupling step, slab ocean, bucket surface)
• current target tests: idealized surface - for long-run reports and like for like atmos comparison
  • standalone: trigger ClimaAtmos driver with target config from the coupler (coupler does not step this)
  • slabplanet_aqua_new_target_nocouple (atmos fluxes at every RK stage, prescribed SSTs from atmos, stepped by coupler)
  • slabplanet_aqua_new_target (coupler fluxes at every coupling step, prescribed SSTs from coupler)
  • slabplanet_aqua_new_target_evolve_ocn (coupler fluxes at every coupling step, slab ocean)
  • slabplanet_new_target (coupler fluxes at every coupling step, prescribed idealized SSTs from coupler, evolving bucket, target resolution)
  • slabplanet_new_target_evolve_ocn (coupler fluxes at every coupling step, slab ocean, evolving bucket, target resolution)
• current target tests: AMIP surface - current state of AMIP for long-run reports
  • amip_new_target (AMIP with no topography)
  • amip_new_target_topo (full AMIP)
• enable evolving ocean from .yml file
• enable toml file specification from the coupler
• add infrastructure in target dir for the atmos standalone run
• make surface default init conditions equivalent to atmos

Tasks

• implement the above components
• introduce default initial conditions that are consistent with those in ClimaAtmos (note, this made the eEisenman sea ice component less stable. For now t_end for this case was decreased, but this will be investigated further as part of Enable less severe initial conditions #494 )
• stabilize coupled long runs (this was achieved by increasing the thermal capacity, but still yields unphysical circulations. This will be investigated in Phase 2: Stabilizing long runs #485)

QA

• see builds in the longrun pipeline