Simulations crash because of NetCDF module

[mmammes]

Hello everyone,

I have an issue with restarts of my simulations, they keep crashing right at the start, always on the same date.
I'm on Levante using AWI-CM3 v3.2 and I have daily output. After my simulation crashed for the first time, I restarted it but it crashed again. The error line and the one before in the log file ( /work/ab0246/a270254/output/lig_003/failed_23500309-23500309_6/log/lig_003_awicm3_compute_10903577.log ) read as follows:

3:  reading restart PARALLEL for hsnow at /work/ab0246/a270254/output//lig_003/run_23500309-23500309/work/fesom.2350.ice.restart/hsnow.nc
3:  error in line          261 io_netcdf_file_module.F90 NetCDF: Unknown file format

I tried to restart from the day before in case the file was corrupted but this still gave an error (log file: /work/ab0246/a270254/output/lig_003/run_23500308-23500308/log/lig_003_awicm3_compute_10928425.log ):

512:  reading restart PARALLEL for hnode at /work/ab0246/a270254/output//lig_003/run_23500308-23500308/work/fesom.2350.oce.restart/hnode.nc
512:  error in line          261 io_netcdf_file_module.F90 NetCDF: HDF error

Please let me know if you have any ideas what could cause this problem.

All the best, Mareike

[fernandadialzira]

Hi Mareike!

I also had this issue with one simulation.

I have seen something on the matter in FESOM/fesom2#533, but it did not give me much of a clue on how to solve this.

the log of my error is at /work/ab0246/a270179/runtime/awicm3-v3.1/v31-e280-3900-768-001/run_18500101-18501231/log/v31-e280-3900-768-001_awicm3_compute_10806456.log

reading restart PARALLEL for w_expl at /work/ab0246/a270179/runtime//awicm3-v3.1//v31-e280-3900-768-001/run_18500101-18501231/work/fesom.3900.oce.restart/w_expl.nc
 128:  error in line          254 io_netcdf_file_module.F90 No such file or directory
 128: 1
 130:  reading restart PARALLEL for w at /work/ab0246/a270179/runtime//awicm3-v3.1//v31-e280-3900-768-001/run_18500101-18501231/work/fesom.3900.oce.restart/w.nc
 129:  error in line          254 io_netcdf_file_module.F90 No such file or directory
   1:  error in line          254 io_netcdf_file_module.F90 No such file or directory
 386:  reading restart PARALLEL for w_impl at /work/ab0246/a270179/runtime//awicm3-v3.1//v31-e280-3900-768-001/run_18500101-18501231/work/fesom.3900.oce.restart/w_impl.nc
 129: 1
   1: 1
   2:  error in line          254 io_netcdf_file_module.F90 No such file or directory
   2: 1
   2:  error in line          101 async_threads_module.F90

[mandresm]

Can this be a problem with conda? I would try deactivating conda if you have it activated and also commenting the conda lines in the .bashrc if there are any and restarting the shell.

If this doesn't solve it, please let me know.

[mandresm]

I see, no idea about this, seems very bound to the FESOM source code. @JanStreffing, do you know anything about this?

[JanStreffing]

/work/ab0246/a270254/output//lig_003/run_23500309-23500309/work/fesom.2350.ice.restart/hsnow.nc

Does not exist.

[JanStreffing]

That explains the error, but then the question is why does it not exist :-)

[JanStreffing]

That would be because run_23500309-23500309 was renamed to failed_23500309-23500309

[fernandadialzira]

She renamed the folder after the crash to see if that could have been due to a hardware problem while copying the restart files. Yet, whenever she would try a restart, it would fail on the same day. We tried the workaround of changing the clock to the previous day and restart, and still it would crash. I had the same issue with a branchoff simulation complaining about files that did exist. Maybe we can meet on Wednesday about it?