Use ClimaAtmos new config file interface #395

valeriabarra · 2023-08-17T18:35:11Z

Purpose

This PR updates ClimaAtmos to v0.16.0 and its new interface for configuration files (rather than CLI options), and keeps the existing functionality for the Coupler to still use CLI options.

This new interface reduces the maintenance burden and code duplication that we used to have (we used to manually copy/paste ClimaAtmos' default CLI options). Now, if any default in ClimaAtmos will change in the future, the Coupler won't need to overwrite them. They will be reflected directly in the Coupler.

Closes #388

To-do

Update all buildkite regular and longrun pipeline jobs
Complete tests in interactive mode
Test & Debug

Content

Updated to latest ClimaAtmos release (v0.16.0)
Got rid of ClimaAtmos parsed_args table (keeping the only one entry that exists now, --config_file)
Renamed "AMIP - modular, Float32 test" -> "AMIP - modular, Float64 test", since we were using Float64 in that test
Cleaned up flame.jl and flame_diff.jl scripts (they were referring to an old job_id target_amip_n32_shortrun that we don't use anymore)

Remap and MPI trouble shooting (issues that came up after cluster upgrade)

mpi hdf5 circ dependency circular dependencies throwing wrong library errors
- need to use the JuliaProject.toml and specify JULIA_LOAD_PATH (soon to be resolved with the TempestRemap new release, but for now we need to load packages in a specific order)
bus errors, race conditions when writing new regrid files
- tests exhibit different behaviour with different modules (solution above)
- apply barriers were needed
overloaded ApplyOfflineMap remapping
- apply the correct const comms_ctx = ClimaComms.context(ClimaComms.CPUSingleThreaded())
coupler AMIP race conditions
- ensure regrid directory contains the run-specific run_name
tempest remap file error
- TR has a character limit, so the regrid directory path shouldn't be too long.

Review checklist

I have:

followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/
followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/
followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy
checked that this PR does not duplicate an open PR.

In the Content, I have included

relevant unit tests, and integration tests,
appropriate docstrings on all functions, structs, and modules, and included relevant documentation.

-->

I have read and checked the items on the review checklist.

valeriabarra · 2023-08-23T18:22:09Z

bors try

bors · 2023-08-23T18:55:12Z

try

Build failed:

buildkite/climacoupler-ci

valeriabarra · 2023-08-29T01:20:08Z

CI passes (I just rebased onto main and squashed all commits to simplify the review process) and this PR is finally ready for review! Thanks!

nefrathenrici

I'm not sure about some of the perf changes, but the atmos interface changes all look good to me! Thanks for dealing with the new config flow

LenkaNovak

Good job on this, @valeriabarra ! I have a few clarification comments and suggestions to re-enable interactive runs.

LenkaNovak · 2023-08-29T15:44:44Z

.buildkite/pipeline.yml

@@ -9,6 +9,9 @@ env:
  BUILDKITE_BRANCH: "${BUILDKITE_BRANCH}"
  JULIA_MAX_NUM_PRECOMPILE_FILES: 100
  GKSwstype: 100
+  CONFIG_PATH: "config/model_configs"
+  PERF_CONFIG_PATH: "config/perf_configs"
+  MPI_CONFIG_PATH: "config/mpi_configs"


It could be clearer to call this mpi_unit_test_configs since some of the integration test configs also use MPI.

I had followed the same name convention they have in Atmos. Do we care? Or we don't mind having them different?
For other cases (e.g., ClimaLSM, I had seen we tend to follow the same name conventions)

perf/flame_diff.jl

experiments/AMIP/modular/coupler_driver_modular.jl

experiments/AMIP/modular/components/atmosphere/climaatmos_init.jl

experiments/AMIP/modular/coupler_driver_modular.jl

valeriabarra · 2023-08-29T20:59:42Z

Good job on this, @valeriabarra ! I have a few clarification comments and suggestions to re-enable interactive runs.

Interesting, thanks! I had tested it interactively (in fact, it was one of the checkboxes in the PR description). Let me take a closer look. Perhaps that was some time before I made some latest changes. Thanks for noticing

valeriabarra · 2023-08-30T01:49:56Z

bors try

bors · 2023-08-30T02:21:41Z

try

Build failed:

buildkite/climacoupler-ci

valeriabarra · 2023-09-01T02:02:21Z

Hi all, thanks for the review. I believed I replied/addressed all comments and CI passes. Thanks!

LenkaNovak

Hi @valeriabarra , thanks for the revisions! Look good, but I'm noticing that the flame graphs are not being generated on Buildkite anymore. I'm also getting some errors with TemperstRemap when running interactively. I should have some time to debug/merge this next week. Does that work for you?

valeriabarra · 2023-09-02T18:12:43Z

Hi @valeriabarra , thanks for the revisions! Look good, but I'm noticing that the flame graphs are not being generated on Buildkite anymore. I'm also getting some errors with TemperstRemap when running interactively. I should have some time to debug/merge this next week. Does that work for you?

Oh, so sorry for not noticing that. They were fixed before the revision. I must have missed something. Of course, if you can, please. Otherwise let me know if I can help. Thanks a lot!

CI: Renamed AMIP - modular, Float32 test -> AMIP - modular, Float64 test Apply reviewer's comments Use different merge order in interactive/non-interactive mode flame test modification trigger perf configs using model configs Update to ClimaAtmos v0.16.0: Use config file interface CI: Renamed AMIP - modular, Float32 test -> AMIP - modular, Float64 test flame test modification trigger perf configs using model configs rm perf LSM dep Update to ClimaAtmos v0.16.0: Use config file interface CI: Renamed AMIP - modular, Float32 test -> AMIP - modular, Float64 test Apply reviewer's comments Use different merge order in interactive/non-interactive mode flame test modification trigger perf configs using model configs rm ClimaLSM deps of sub folders fix rebase Buildkite name fix spec run_name hdf5 mpi fix try fix perf runs revert MPI load try MPI specs revert revert mpi write debug - barrier try try CC env vars try CC env vars srun CA16 BK setup flame fix j1.9 Manifest v1.9 local pass add JuliaProject.toml revert Projects runs serially, try JuliaProject, J1.8.5 srun for mpi jobs add srun for amip rm depot path rm CLIMACOMMS_CONTEXT test JULIA_DEPOT_PATH race in mpi tests race in mpi tests race in mpi tests race in mpi tests race in mpi tests show pid race in mpi tests show pid device race in mpi tests race in AMIP cleanup cleanup add notes to local mpi tests add notes to local mpi tests GKSwstype: 100

LenkaNovak · 2023-09-21T00:21:10Z

bors r+

bors · 2023-09-21T02:08:09Z

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

valeriabarra added enhancement New feature or request 🍃 leaf Issue coupled to a PR Dependencies 🔗 labels Aug 17, 2023

valeriabarra self-assigned this Aug 17, 2023

valeriabarra marked this pull request as draft August 17, 2023 18:35

valeriabarra force-pushed the valeria/test-atmos-toml-config-files branch 3 times, most recently from 85f9435 to 7588afb Compare August 21, 2023 19:58

valeriabarra closed this Aug 21, 2023

valeriabarra reopened this Aug 21, 2023

valeriabarra force-pushed the valeria/test-atmos-toml-config-files branch 5 times, most recently from ed70bd0 to bf44bc6 Compare August 23, 2023 02:59

valeriabarra mentioned this pull request Aug 23, 2023

CompatHelper: bump compat for ClimaAtmos to 0.16, (keep existing compat) #397

Closed

bors bot added a commit that referenced this pull request Aug 23, 2023

Try #395:

751ba51

valeriabarra force-pushed the valeria/test-atmos-toml-config-files branch 2 times, most recently from c1601d6 to 90075f9 Compare August 24, 2023 17:29

valeriabarra marked this pull request as ready for review August 24, 2023 17:33

valeriabarra force-pushed the valeria/test-atmos-toml-config-files branch 4 times, most recently from 17e1ef6 to 95b5959 Compare August 28, 2023 19:07

valeriabarra mentioned this pull request Aug 29, 2023

CompatHelper: bump compat for ClimaAtmos to 0.16, (keep existing compat) #411

Closed

valeriabarra force-pushed the valeria/test-atmos-toml-config-files branch 2 times, most recently from ec99c75 to 5504640 Compare August 29, 2023 01:18

valeriabarra requested a review from LenkaNovak August 29, 2023 01:19

valeriabarra requested a review from nefrathenrici August 29, 2023 01:19

nefrathenrici approved these changes Aug 29, 2023

View reviewed changes

LenkaNovak requested changes Aug 29, 2023

View reviewed changes

bors bot added a commit that referenced this pull request Aug 30, 2023

Try #395:

f748d2f

LenkaNovak mentioned this pull request Sep 1, 2023

Separate atmos and config files #416

Closed

LenkaNovak reviewed Sep 1, 2023

View reviewed changes

LenkaNovak approved these changes Sep 18, 2023

View reviewed changes

LenkaNovak force-pushed the valeria/test-atmos-toml-config-files branch 9 times, most recently from b943ea3 to d072248 Compare September 20, 2023 21:15

LenkaNovak force-pushed the valeria/test-atmos-toml-config-files branch from d072248 to 45f86f6 Compare September 20, 2023 22:19

bors bot merged commit 5f678b9 into main Sep 21, 2023
10 checks passed

bors bot deleted the valeria/test-atmos-toml-config-files branch September 21, 2023 02:08

juliasloan25 mentioned this pull request Sep 21, 2023

CompatHelper: bump compat for ClimaAtmos to 0.16, (keep existing compat) #413

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ClimaAtmos new config file interface #395

Use ClimaAtmos new config file interface #395

valeriabarra commented Aug 17, 2023 •

edited by LenkaNovak

Loading

valeriabarra commented Aug 23, 2023

bors bot commented Aug 23, 2023

valeriabarra commented Aug 29, 2023

nefrathenrici left a comment

LenkaNovak left a comment

LenkaNovak Aug 29, 2023

valeriabarra Aug 29, 2023

valeriabarra commented Aug 29, 2023

valeriabarra commented Aug 30, 2023

bors bot commented Aug 30, 2023

valeriabarra commented Sep 1, 2023

LenkaNovak left a comment

valeriabarra commented Sep 2, 2023

LenkaNovak commented Sep 21, 2023

bors bot commented Sep 21, 2023

Use ClimaAtmos new config file interface #395

Use ClimaAtmos new config file interface #395

Conversation

valeriabarra commented Aug 17, 2023 • edited by LenkaNovak Loading

Purpose

To-do

Content

Remap and MPI trouble shooting (issues that came up after cluster upgrade)

valeriabarra commented Aug 23, 2023

bors bot commented Aug 23, 2023

try

valeriabarra commented Aug 29, 2023

nefrathenrici left a comment

Choose a reason for hiding this comment

LenkaNovak left a comment

Choose a reason for hiding this comment

LenkaNovak Aug 29, 2023

Choose a reason for hiding this comment

valeriabarra Aug 29, 2023

Choose a reason for hiding this comment

valeriabarra commented Aug 29, 2023

valeriabarra commented Aug 30, 2023

bors bot commented Aug 30, 2023

try

valeriabarra commented Sep 1, 2023

LenkaNovak left a comment

Choose a reason for hiding this comment

valeriabarra commented Sep 2, 2023

LenkaNovak commented Sep 21, 2023

bors bot commented Sep 21, 2023

valeriabarra commented Aug 17, 2023 •

edited by LenkaNovak

Loading