horiz_interp_conserve_mod:no latitude index found #83
Comments
|
Hi @eocene. I would imagine that the new nodes are using a different version of MPI, or something like that. So that I can help more, could you tell us what kind of model are you running? I.e. are you using grey radiation / RRTM / held-suarez etc? I would say that your error here is almost certainly not a problem with the interpolation routine, but is a symptom of some other problem. ( I would also advise against doing too much digging in the interpolation routine. It's very long and not that easy to read!). |
|
Hi Stephen, thank you very much indeed for the (quick) response! I've checked the MPI versions and apparently they are all the same. I absolutely agree about the symptom-not-cause diagnosis. For completeness, I have run various test_cases and mysteriously they all fail differently. All these have been recompiled again, and tried with various PE numbers etc. (Another thing I've tried is increasings "num_iters" in horiz_interp_conserve.F90 based on reading the code/docs, but to no avail). I'm sure that there is something amiss/stupid/negligent on my end, but since the root cause seems to be a bit obscure at the moment any hints really would be very much appreciated! Thank you very much indeed. axisymmetric fails exactly like my personal configuration: Held-Suarez fails on a segmentation fault: and Realistic-Continents fails on 'regularize: Failure to converge' |
|
I obtained a similar error using the realistic-continents case. The result being a crash with message :"regularize: Failure to converge". |
|
Strange - @eocene did you end up getting a handle on this problem? |
|
@AlexAudette - do you also find other test cases to be failing, or is it just the realistic continents one? |
|
@sit23 So far it is only with the realistic continents. I am able to run my simulation at T42 using the era_land_T42.nc land mask file, but when I create my own at T85, I get the same error as eocene. |
|
@AlexAudette OK - that's a slightly different problem, which we have encountered ourselves. The background is that when you put data like topography into the spectral dynamical core, the spikiness of the data and the finite number of Fourier modes means that you form Gibbs ripples etc in the topography. To help counter this, the model automatically smooths the incoming topography, which reduces the size of the ripples. The degree of the smoothing is controlled by the parameter |
|
@sit23 Thanks for you answer. I tried reducing the ocean_topog_smoothing parameter from 0.8 to 0.05 by increments of 0.2, but still no success, I still get the same error : 2020-03-10 09:28:42,216 - isca - INFO - process running as 110162 |
|
OK - could you try setting it to 0? That should turn off the regularisation, and we can see if it runs then or not. You could also try increasing the parameter, just in case I've mis-remembered the way you need to go! |
|
So it runs now with the parameter set to 0, thank you very much. I tried as well to increase the parameter to 0.96 and still crashed at the place. I will keep an eye out for truncation effects. Thanks again! |
|
OK - you will probably find that the gibbs ripples are significant without any smoothing. You'll particularly see it in the vertical velocity and the precipitation. When we've run with topography at T85, we have managed to run the smoothing, but I can't quite lay my hands on the smoothing parameter we used. I'll let you know if I find it. We are working on alternatives to this smoothing algorithm, which should be available soon. |
|
Just found it - looks like I tried 0.85 for the smoothing parameter and it worked with T85 topography. |
|
Interesting, I just tried with this same value and it still fails to regularize. Did you do anything special with your topography file? |
|
Well, you're welcome to try the T85 topography file that I used and see if it works for you. You can find it here: |
dennissergeev
added
idealised
Hi all,
I have been running Isca on a machine that has recently had new nodes installed. Before the new nodes all was fine. Now, when running I get a fatal error returned from all PEs like the one below. It's hard to report to sys admin without specific request (I suspect something wasn't done when installing the new nodes but could easily be wrong). Before I go digging into this interpolation module that is triggering the error, I just wanted to check with you whether you had seen this before and/or had an idea what might be the trigger.
Thank you very much indeed for any possible help in advance,
2019-01-28 10:23:49,905 - isca - DEBUG - FATAL from PE 0: horiz_interp_conserve_mod:no latitude index found: n,sph= 1 NaN 2019-01-28 10:23:49,905 - isca - DEBUG - 2019-01-28 10:23:49,905 - isca - DEBUG - application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0The text was updated successfully, but these errors were encountered: