Convergence tests pipeline takes several hours to finish

[ali-ramadhan]

Most of the convergence tests run in a reasonable amount of time except for a few:

For example: https://buildkite.com/clima/oceananigans-validation-experiments/builds/137

• CPU Gaussian advection-diffusion: ~2:40 hours
• GPU Gaussian advection-diffusion: ~1:45 hours
• CPU forced flow free-slip: ~8 hours

I think for the Gaussian advection-diffusion tests we can just reduce the number of iterations. Some tests run for 26214 iterations.

Not sure why the CPU forced flow free-slip is so slow though. It's a 2D simulation. The CPU forced flow fixed-slip is similar I think and takes only ~20 minutes.

[glwagner]

Uhh weird. I guess we have to proof read those scripts. The free-slip and fixed-slip experiments just differ by boundary conditions I think.

[glwagner]

I'm closing this issue because I'm judging that it's not of current, timely relevance to Oceananigans development. If you would like to make it a higher priority or if you think the issue was closed in error please feel free to re-open.