SOwISC12to60E2r02: Update spin-up config

[xylar]

More gradual decrease of Rayleigh damping and increase of time step was needed (still over 20 days total) to prevent blow-up in the new configuration.

[xylar]

I forgot that this hadn't yet been committed to #518 when I merged that PR :-(

[xylar]

I'm testing this now on LANL IC at:

/lustre/scratch4/turquoise/xylar/spin_up_SOwISC12to60kmL64E3SMv2rev02/ocean/global_ocean/SO60to10wISC/spin_up

[xylar]

@mark-petersen and @kristin-hoch, I'm having a lot of trouble with spinning up this mesh. I think I copied exactly a set of time steps (barotorpic and baroclinic), Rayleigh damping coefficients etc. that worked for me with the same setup (but before the recent merge of this mesh to ocean/develop). But I've rerun and I simply can't get the Rayleigh damping to or below 2e-5. I've tried smaller time steps (both barotropic and baroclinic) and longer spin-ups with higher Rayleigh damping (40 days in my latest attempt). In my latest attempt, I stepped the Rayleigh damping down every 5 days from 1e-4 to 2e-5 in steps of 1e-5. The CFl is well below 0.1 for almost the whole time, before suddenly spiking some time after the move to 2e-5. Here's the CFL before the spike (x axis is in hours):

My attempts are in:

/lustre/scratch4/turquoise/xylar/spin_up_SOwISC12to60kmL64E3SMv2rev02/ocean/global_ocean/SO60to10wISC/

but I'm more looking for some general advice and intuition.

In the meantime, I'm downloading VTK files from the last restart to see if they give me any insight.

[mark-petersen]

Thanks. I've struggled with spin-up before too. The normal solution is to run longer at the early stages (short time step, high Rayleigh), but usually a few days is enough, and 40 is way overboard.

We've changed the vertical grid, and this also has ice shelves. Check the thickness of the thinnest cell, especially the partial bottom cells. Also, see if the blow-up happens below ice shelves, in a PBC, a tilted cell, or what. It could also be an odd-ball shaped cell in the horizontal. You could plot kineticEnergyCell when it spikes in the globa stats, and see where it is high.

[kristin-hoch]

Xylar,
One thing that's tripped me up more than once is the conversion factors in the mesh initialization. For whatever reason, some of the meshes have depth in m, some in cm. If the mesh is in cm, you have to set
config_global_ocean_depth_conversion_factor = 0.01 and config_global_ocean_tracer_depth_conversion_factor = 0.01

[mark-petersen]

@kristin-hoch great point, thanks for pointing that out. I'll double check that flag in COMPASS as well.

[xylar]

@kristin-hoch, that would have certainly been a possible (and embarrassing) mistake but I've checked that. Those conversion factors only apply to the 60-layer mesh taken from POP. We're now using 64 layers and the default conversion factor of 1.0

[xylar]

Okay, I finally was able to take the time to fix this today. It turned out to be a single-cell ice shelf with an abrupt drop in bottomDepth compared to its neighbors. For now, I'm doing a bit more global smoothing of the bathymetry to get rid of this. In the next mesh iteration, we could consider doing more smoothing near ice shelves than elsewhere in the mesh if we want to avoid smoothing elsewhere.

[xylar]

I'm not sure the spin-up needs to be divided into 4 steps at this point, but I also don't think it does any harm, so I'd like to leave it that way.

@mark-petersen, could you take a look and make sure this makes sense to you? It is ready to merge when the timing is right. Once that happens, I'll redo the init and spin_up runs so we have the makings of an E3SM initial condition.

[mark-petersen]

@xylar this all looks good. If you had a successful spin-up, I'm happy to merge it. I don't mind the extra spin-up steps, or the extra smoothing.

[xylar]

@mark-petersen, yep, my spin-up was successful so go ahead and merge, please!