Improve model start-up time

[nichannah]

The data assimilation team needs to run many very short ensemble members of ACCESS-OM2 (0.1). The long start-up time of the model accounts for a very significant amount of the overall runtime. Investigate how this can be improved.

Russ' email says ...

The main issue at the start appears to be the setting up of OASIS. It's pretty much a consistent 140 second process measured from the MOM side. This is the same for your IAF runs.

OASIS init 138.232003 138.331698 138.280318 0.026412 0.011 1 0 4357
Initialization 168.702646 168.802362 168.751017 0.026371 0.014 0 0 4357
(Ocean initialization) 28.757189 30.121281 29.393740 0.164157 0.002 11 0 4357

Termination is about 130 seconds for Pavel's runs. And the integration time is about 6 minutes.

CICE timing doesn't really capture all of the initialisation time at the moment but I think it is faster. It looks like it might be taking a minute or so to write its restarts which is slow and seems much longer than the IAF runs.

Timer 12: ReadWrite 70.70 seconds
Timer 14: History 24.63 seconds

I'm not sure in Nic's PIO changes will help that. I think it's only for the history component.

[russfiedler]

@nichannah identified that it appears that the bulk of the startup cost in CICE is due to the broadcasting of 2D arrays in ice_grid.F90 https://github.com/COSIMA/cice5/blob/master/source/ice_grid.F90#L203-L204