Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add target resolution AMIP Buildkite drivers #143

Merged
merged 1 commit into from
Nov 1, 2022
Merged

Add target resolution AMIP Buildkite drivers #143

merged 1 commit into from
Nov 1, 2022

Conversation

LenkaNovak
Copy link
Collaborator

@LenkaNovak LenkaNovak commented Oct 25, 2022 •
edited
Loading

Purpose

Add high resolution AMIP Buildkite.

To-do

Buildkite drivers:

  • short run - with the final target setup (to be tested and required to pass for integration)
  • long run - not required to pass, but showing the current best stable simulation.

Components

  • update all packages (when ClimaAtmos is up to date with ClimaCore), remove dependencies on specific branches, enable z0 spec, enable land snow, enable observed albedo
  • MPI segmentation errors
    • - memory issues - compute in scrartch
    • - Heisenbug with artifacts
      • Julia update solved this issue
        • note Fix race condition with artifacts #145 will additionally ensure only one Artifacts download per pipeline to avoid further race conditions - these are the reasons why some of the buildkite runs were re-run in the tests below.
  • sea ice and Bucket surface flux specification - ensuring that no negative moisture fluxes (this will need to be revisited to be more realistic)

Next steps:

  • add AMIP tutorial
  • performance testing
  • further improvements / tuning / testing of AMIP parameterizations

@LenkaNovak LenkaNovak changed the title Add 100day & high resolution AMIP Buildkite Add target resolution AMIP Buildkite drivers Oct 25, 2022
@LenkaNovak LenkaNovak force-pushed the mpi-buildkite-longrun branch 2 times, most recently from 5007428 to 19aa808 Compare October 25, 2022 03:05
@LenkaNovak LenkaNovak mentioned this pull request Oct 28, 2022
Merged
@LenkaNovak LenkaNovak force-pushed the mpi-buildkite-longrun branch 4 times, most recently from c9be4e2 to 6fc0ef5 Compare October 31, 2022 17:00
@LenkaNovak LenkaNovak requested a review from szy21 October 31, 2022 17:53
@LenkaNovak LenkaNovak marked this pull request as ready for review October 31, 2022 17:53
szy21
szy21 approved these changes Oct 31, 2022
View reviewed changes
Copy link
Member

@szy21 szy21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks! I left only one comment.

experiments/AMIP/moist_mpi_earth/coupler_driver.jl Outdated
land_mask = land_mask,
date0 = date0,
mono = mono_surface,
)
update_midmonth_data!(date0, SIC_info)
SIC_init = interpolate_midmonth_to_daily(date0, SIC_info)
ice_mask = get_ice_mask.(SIC_init, mono_surface)
ice_mask = get_ice_mask.(SIC_init, mono_surface) .* FT(0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this on purpose?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review! Yeah, this was done to let the system spin up ice-free for the first 15 days. The mask then gets updated with the prescribed SIC thereafter. That said, after the latest changes in surface fluxes, this may not make that much of a difference. I can take a look. :)

@LenkaNovak LenkaNovak mentioned this pull request Nov 1, 2022
Merged
@LenkaNovak
bring back energy conservation
1b318ef 
format

amip plots

debug amip instability

addit

rebase clean

rm tmp output

buildkite spec

stop extra ts update

display amip plot

flip axis

ncep plots

format

ncep sfc data fix

automate ncep plot date

generalize plot function

format

reinit conservation check last

Conflicts:
	experiments/AMIP/moist_mpi_earth/coupler_driver.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/amip_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/ncep_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/variable_definer.jl

mpi-enabled

fix

Conflicts:
	experiments/AMIP/moist_mpi_earth/Manifest.toml
	experiments/AMIP/moist_mpi_earth/cli_options.jl

bring back energy conservation

format

amip plots

debug amip instability

addit

rebase clean

rm tmp output

buildkite spec

stop extra ts update

display amip plot

flip axis

ncep plots

format

ncep sfc data fix

automate ncep plot date

generalize plot function

format

reinit conservation check last

Conflicts:
	experiments/AMIP/moist_mpi_earth/coupler_driver.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/amip_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/ncep_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/variable_definer.jl

mpi-enabled

fix

Conflicts:
	experiments/AMIP/moist_mpi_earth/Manifest.toml
	experiments/AMIP/moist_mpi_earth/cli_options.jl

bring back energy conservation

format

amip plots

debug amip instability

addit

rebase clean

rm tmp output

buildkite spec

stop extra ts update

display amip plot

flip axis

ncep plots

format

ncep sfc data fix

automate ncep plot date

generalize plot function

format

reinit conservation check last

Conflicts:
	experiments/AMIP/moist_mpi_earth/coupler_driver.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/amip_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/ncep_visualizer.jl
	experiments/AMIP/moist_mpi_earth/coupler_utils/variable_definer.jl

mpi-enabled

fix

name fix

mpi race - wip

wip

Co-authored-by: Sriharsha Kandala <sriharsha.kvs@gmail.com>

Co-authored-by: Simone Byrne

masks ok but NaNs (wip)

Cd Ch vals

rebased and MPI tested (pending physical validation)

REGRID_DIR for times

unstable at 4d - unrelared to mpi (wip)

mesh fix

re-add regridding for postprocessor (AMIP runs)

Co-authored-by: Zhaoyi Shen <szy21@users.noreply.github.com>
Co-authored-by: Valeria Barra <valeriabarra@users.noreply.github.com>

re-add regridding for postprocessor (AMIP runs)

Co-authored-by: Zhaoyi Shen <szy21@users.noreply.github.com>
Co-authored-by: Valeria Barra <valeriabarra@users.noreply.github.com>

format

buildkite race fix

format

format

mpi fix

unit test

unit test

MPI race

MPI race in bc reader

format

format

delete file

format

save output from single core amip

update unit test

call buildkite artifacts

long high res run

pipelines

format

format

format

checkpoint

checkpoint

adjusted albedo

update for longruns

fix

pipeline

Pkg updates

bug fix

mpi dup error

diags update

up pipeline

up buildkite julia

julia in man

unit test fix

cleanup

long run pipeline flags

pipline fix

pipline fix

rm ice-free spin up
@LenkaNovak
Copy link
Collaborator Author

bors r+

@LenkaNovak LenkaNovak mentioned this pull request Nov 1, 2022
Closed
27 tasks
@bors
Copy link
Contributor

bors bot commented Nov 1, 2022

Build succeeded:

@bors bors bot merged commit fef78da into main Nov 1, 2022
@bors bors bot deleted the mpi-buildkite-longrun branch November 1, 2022 15:58
@LenkaNovak LenkaNovak mentioned this pull request Nov 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@szy21 szy21 szy21 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@LenkaNovak @szy21