MPI AMIP FINE memory issues

[juliasloan25]

Recently, the MPI AMIP FINE test in CI has begun failing nondeterministically, initially here. The error suggests that this is due to a lack of memory.

For now, this issue is being addressed by increasing the memory allocation for this test (see PR #195), but we should find and fix the root cause of this problem.

One task that may help enforce reproducibility is adding compat entries in the Project.toml files in this repo.

ClimaCore #1113 has resolved the memory usage issues. #279 removes the extra memory allocations we had added to the buildkite driver as a workaround.

Fixed by #279.

[LenkaNovak]

@charleskawczynski would you be able to help with this please? Does ClimaAtmos suffer from similar issues?

[charleskawczynski]

I think I'd recommend first updating to the latest versions. If this job is failing consistently, then maybe change it to a soft fail. I have some ideas about what might help, I'll chat with @simonbyrne tomorrow

[juliasloan25]

The memory issues seem to have originated in ClimaCore PR#978. ClimaCore has relevant issues #948 and #1064, which note the issue there. Hopefully once those are addressed, the memory usage issues in the coupler will be resolved as well.

[LenkaNovak]

Thanks for the update, @juliasloan25 ! :)