LIS DA crashes

[sunnydays99]

Dear community,

I have been trying to understand why my data assimilation runs keeps on quitting but to no avail. In the retrospective runmode it stops after the first 3 hour and I receive a SIGSEGV: Segmentation fault - invalid memory reference error. Changing the attributes or perturbation files did not help yield to any other output.
If anyone knows where I should look for to fix the error I would gladly appreciate any advice.
Thank you for your time

The error message follows:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x14c8e0023960 in ???
#1 0x14c8e0022ac5 in ???
#2 0x14c8dfc4251f in ???
at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#0 0x153c06a23960 in ???
#1 0x153c06a22ac5 in ???
#2 0x153c0664251f in ???
at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#0 0x153d4bc23960 in ???
#1 0x153d4bc22ac5 in ???
#2 0x153d4b84251f in ???
at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#0 0x14f56c423960 in ???
#1 0x14f56c422ac5 in ???
#2 0x14f56c04251f in ???
at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#3 0x55e4cde5e2d3 in clsmf25_gettwspred_
at ../surfacemodels/land/clsm.f2.5/da_tws/clsmf25_gettwspred.F90:55
#4 0x55e4ceb6173c in lsmdagetobspred_
at ../core/LIS_lsm_FTable.c:1399
#5 0x55e4cdf6303f in _enksgrace_mod_MOD_enksgrace_increments
at ../dataassim/algorithm/enksgrace/enksgrace_Mod.F90:291
#6 0x55e4ceb5bbe3 in computeincrements
at ../core/LIS_dataassim_FTable.c:436
#7 0x55e4cd76766b in _lis_dataassimmod_MOD_lis_dataassim_run
at ../core/LIS_dataAssimMod.F90:517
#8 0x55e4cea24ddf in retrospective_runmod_MOD_lis_run_retrospective
at ../runmodes/retrospective/retrospective_runMod.F90:203
#9 0x55e4ceb6922d in lisrun
at ../core/LIS_runmode_FTable.c:202
#10 0x55e4ce2118aa in lisdrv
at ../offline/lisdrv.F90:70
#11 0x55e4ce2119b4 in main
at ../offline/lisdrv.F90:57
#3 0x56333af512d3 in clsmf25_gettwspred
at ../surfacemodels/land/clsm.f2.5/da_tws/clsmf25_gettwspred.F90:55
#4 0x56333bc5473c in lsmdagetobspred
at ../core/LIS_lsm_FTable.c:1399
#5 0x56333b05603f in _enksgrace_mod_MOD_enksgrace_increments
at ../dataassim/algorithm/enksgrace/enksgrace_Mod.F90:291
#6 0x56333bc4ebe3 in computeincrements
at ../core/LIS_dataassim_FTable.c:436
#7 0x56333a85a66b in _lis_dataassimmod_MOD_lis_dataassim_run
at ../core/LIS_dataAssimMod.F90:517
#8 0x56333bb17ddf in retrospective_runmod_MOD_lis_run_retrospective
at ../runmodes/retrospective/retrospective_runMod.F90:203
#9 0x56333bc5c22d in lisrun
at ../core/LIS_runmode_FTable.c:202
#10 0x56333b3048aa in lisdrv
at ../offline/lisdrv.F90:70
#11 0x56333b3049b4 in main
at ../offline/lisdrv.F90:57
#3 0x5604c10302d3 in clsmf25_gettwspred
at ../surfacemodels/land/clsm.f2.5/da_tws/clsmf25_gettwspred.F90:55
#4 0x5604c1d3373c in lsmdagetobspred
at ../core/LIS_lsm_FTable.c:1399
#5 0x5604c113503f in _enksgrace_mod_MOD_enksgrace_increments
at ../dataassim/algorithm/enksgrace/enksgrace_Mod.F90:291
#6 0x5604c1d2dbe3 in computeincrements
at ../core/LIS_dataassim_FTable.c:436
#7 0x5604c093966b in _lis_dataassimmod_MOD_lis_dataassim_run
at ../core/LIS_dataAssimMod.F90:517
#8 0x5604c1bf6ddf in retrospective_runmod_MOD_lis_run_retrospective
at ../runmodes/retrospective/retrospective_runMod.F90:203
#9 0x5604c1d3b22d in lisrun
at ../core/LIS_runmode_FTable.c:202
#10 0x5604c13e38aa in lisdrv
at ../offline/lisdrv.F90:70
#11 0x5604c13e39b4 in main
at ../offline/lisdrv.F90:57
#3 0x5638f69622d3 in clsmf25_gettwspred
at ../surfacemodels/land/clsm.f2.5/da_tws/clsmf25_gettwspred.F90:55
#4 0x5638f766573c in lsmdagetobspred
at ../core/LIS_lsm_FTable.c:1399
#5 0x5638f6a6703f in _enksgrace_mod_MOD_enksgrace_increments
at ../dataassim/algorithm/enksgrace/enksgrace_Mod.F90:291
#6 0x5638f765fbe3 in computeincrements
at ../core/LIS_dataassim_FTable.c:436
#7 0x5638f626b66b in __lis_dataassimmod_MOD_lis_dataassim_run
at ../core/LIS_dataAssimMod.F90:517
#8 0x5638f7528ddf in _retrospective_runmod_MOD_lis_run_retrospective
at ../runmodes/retrospective/retrospective_runMod.F90:203
#9 0x5638f766d22d in lisrun
at ../core/LIS_runmode_FTable.c:202
#10 0x5638f6d158aa in lisdrv
at ../offline/lisdrv.F90:70
#11 0x5638f6d159b4 in main
at ../offline/lisdrv.F90:57

lis.config_clsm_graceda.txt
forcing_pert_attribs.txt
forcing_attribs.txt
clsm_tws_pertattribs.txt
clsm_tws_attribs.txt
grace_tws_pertattribs.txt
grace_tws_attribs.txt

[dmocko]

Hi @sunnydays99

Are there any further messages that might be helpful at the end of any of your lislog files?

Can you diagnose if you have enough memory available on your machine to run this configuration?

[sunnydays99]

I have around 75GB of storage left and the error persists if I try reduce the interval to a month instead of a full year

Here is the one for the log file, with the exception of a Error in nf90_get_att: GRIDTYPE warning nothing else shows up.
log_graceda.0000.txt

[dmocko]

Thanks - but I mean memory, not available disk space storage.

Our job queue system gives a report of the amount of memory used during the simulation, e.g.:

  Max Real Mem Used (on any node) : 41.68G
  Max Pct Memory Used             : 3.74%

[sunnydays99]

Okay I see, I am running LIS within my own virtual machine, not sure how to check for the specific memory usage outside from what the system monitor reports. In total my VM has 40GB of memory and during the run around each LIS executable uses around 2GB of virtual memory and 200MB of actual memory storage (less then 7% of the available amount).

[dmocko]

Hi @TimLahmers @KathyNie

Can you please help try to figure out this issue?

@sunnydays99 - what branch are you using?

[sunnydays99]

@KathyNie These are not begin created, as the run stops beforehand.
But when I changed the Data assimilation output interval for diagnostics: from 1da to 12hr and the same for the perturbation restart interval , I managed to output the both directories EnKF and DAPERT after which the same error in the return code message showed up and the run aborted. I tried to use ncview to visualize the ensemble spread and it appear as if all 3 are zeros.
I will attach the EnKF output below, maybe that's the culprit?
LIS_DA_EnKF_200301011200_spread.a01.d01.nc.txt

[KathyNie]

Interesting! I'm not sure if that is the cause, but let's try not output the ensemble spread and see if that helps pass this crash point:

"Data assimilation output ensemble spread: 0"

[sunnydays99]

Okay I see, I tried a run without outputting the ensemble spread without any change to the crash point unfortunately

[KathyNie]

Ok, I will run your case on my side and take a close look. Will get back to you if I have any questions or if I find any bugs.

[KathyNie]

@sunnydays99 Could you also share those files in your INPUT directory such as the CLSM related parameter files?

[sunnydays99]

thank you so much for taking a closer look at the error message.The clsm parameter files is as follows :
Uploading lis_input.gdas.clsm.nc.txt…

[KathyNie]

Hi, it seems that it is not fully uploaded. Would you mind upload it again?

[KathyNie]

Hi @sunnydays99 I'm also curious to learn more about your research question regarding CLSM-based GRACE-DA. Additionally it would be helpful if you could identify your organization. That would help us have a better context on how we can help. Currently our GRACE-DA practices are primarily centered around the Noah-MP land surface model and I just learned that when working with CLSM, you need to contact the corresponding team to get a set of parameter specifically to your configuration requirements. If those parameters are not configured correctly, it could potentially impact the capability with GRACE-DA.

[sunnydays99]

@KathyNie thank you for your message. I tried to set up GRACE-DA for Noah-MP instead and that is running without issues. It might have been the missing parameters for CLSM that were impacting the performance.