unify buildkite pipelines

[juliasloan25]

Purpose

remove some environment variables from our buildkite pipelines, and unify standard and longrun pipelines

closes #441

---

• I have read and checked the items on the review checklist.

[simonbyrne]

Looks good to me, but might be a good idea to try running it

[juliasloan25]

see longrun here: https://buildkite.com/clima/climacoupler-longruns/builds/212

[simonbyrne]

You're hitting CliMA/slurm-buildkite#47

This is applied to all jobs, but some other jobs also specify slurm_mem_per_cpu, which causes conflicts

[juliasloan25]

Ahh okay, thank you

[juliasloan25]

Do you think it may be similarly problematic to specify both slurm_ntasks and slurm_tasks_per_node? e.g. here

[simonbyrne]

Yes, I think that can do weird things.

[juliasloan25]

@LenkaNovak it looks like these changes will fix the longrun issues we had that prevented the coupler reports from being generated :)

[LenkaNovak]

Awesome! Thanks for pursuing this @juliasloan25 and @simonbyrne for your help! Once this is merged we can finalize #456 and start profiling the DYAMOND run with @sriharshakandala 👏

[LenkaNovak]

Maybe just a clarification, @simonbyrne , wouldn't removing slurm_tasks_per_node: X make our weak scaling in these long runs less accurate? We noticed in the past that testing across nodes caused a large variability in walltime.

[simonbyrne]

Looking at the docs:

--ntasks-per-node=
Request that ntasks be invoked on each node. If used with the --ntasks option, the --ntasks option will take precedence and the --ntasks-per-node will be treated as a maximum count of tasks per node. Meant to be used with the --nodes option. This is related to --cpus-per-task=ncpus, but does not require knowledge of the actual number of cpus on each node. In some cases, it is more convenient to be able to request that no more than a specific number of tasks be invoked on each node. Examples of this include submitting a hybrid MPI/OpenMP app where only one MPI "task/rank" should be assigned to each node while allowing the OpenMP portion to utilize all of the parallelism present in the node, or submitting a single setup/cleanup/monitoring job to each node of a pre-existing allocation as one step in a larger job script.

What I would generally suggest is using either just slurm_ntasks, or slurm_ntasks_per_node + slurm_nodes.

[juliasloan25]

I can it change to slurm_ntasks_per_node + slurm_nodes if we want to be more specific! Let me know which you prefer @LenkaNovak

Also, as of this PR our target AMIP longrun fails due to numerical instability. It's hard to tell which PR introduced this, since the last time the longruns successfully ran was Sept 17, and between then and now the target AMIP run was failing due to HDF5 errors. It could have been the updates to ClimaAtmos v0.16.0 or v0.16.1.

[LenkaNovak]

slurm_ntasks_per_node + slurm_nodes

Sounds good. Thanks for the pointer! @juliasloan25 Would it be possible to do this, i.e. specify slurm_ntasks_per_node + slurm_nodes? I've just tested it here. The precise setup is a trade off between wait time for resource allocation and getting the specific setup we need, but it would be good to have a more trackable / comparable configurations for the AMIP run set. Otherwise happy to merge this.

[juliasloan25]

bors r+

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• buildkite/climacoupler-ci
• ci 1.9 - macOS-latest
• ci 1.9 - ubuntu-latest
• ci 1.9 - windows-latest
• docbuild
• format