Data Assimilation testcase error

[Gayathridevi123]

Hi,

I was trying out the LSM data assimilation (DA) experiment example (step 6) as given on the LISF website and ended up with an error:-

[ERR] ESMF_DistGridCreate failed: obsgridDG Stopping.
[ERR] endrun is being called

Please find the attached files.
file.zip

Please kindly let me know how to solve this error.

Thank you,
Gayathri

[bmcandr]

Hi @Gayathridevi123, in a previous discussion you mentioned that you are using LIS 7.3.1. Is this correct? If so, this is a known error when running LISF version 7.3.1 (or earlier) with ESMF v8. See #630 for an explanation and a link to the pull request that fixed this issue.

The solution involves downgrading to ESMF 7.1.0 or upgrading to LISF 7.3.2.

[Gayathridevi123]

Hi...

I upgraded to the master branch. And for the data assimilation test case (step6) now got error:-

LIS: H5MM.c:165: H5MM__sanity_check_block: Assertion `0 == memcmp(block->b + 8 + block->u.info.size,H5MM_block_tail_guard_s,8)' failed.

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0x2aaaaf1d2c3d in ???
#1  0x2aaaaf1d1e83 in ???
#2  0x327b63250f in ???
#3  0x327b632495 in ???
#4  0x327b633c74 in ???
#5  0x327b62b60d in ???
#6  0x327b62b6cf in ???
#7  0x2aaaae5daff2 in H5MM__sanity_check_block
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5MM.c:165
#8  0x2aaaae5dafff in H5MM__sanity_check
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5MM.c:188
#9  0x2aaaae5db15e in H5MM_xfree
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5MM.c:538
#10  0x2aaaae537e68 in H5FL__reg_gc_list
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5FL.c:544
#11  0x2aaaae538905 in H5FL_reg_free
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5FL.c:382
#12  0x2aaaae6ab345 in H5S_close
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5S.c:478
#13  0x2aaaae47119e in H5D__free_chunk_info
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5Dchunk.c:1505
#14  0x2aaaae6ddf14 in H5SL_release_common
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5SL.c:896
#15  0x2aaaae6f36eb in H5SL_free
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5SL.c:2305
#16  0x2aaaae47106e in H5D__chunk_io_term
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5Dchunk.c:2868
#17  0x2aaaae4a7d9b in H5D__read
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5Dio.c:571
#18  0x2aaaae7e8211 in H5VL__native_dataset_read
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5VLnative_dataset.c:166
#19  0x2aaaae7c3239 in H5VL__dataset_read
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5VLcallback.c:2045
#20  0x2aaaae7cda04 in H5VL_dataset_read
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5VLcallback.c:2079
#21  0x2aaaae4a6665 in H5Dread
        at /home/Gayathri/software2/hdf5-1.12.0/src/H5Dio.c:191
#22  0x2aaaadf2bcc0 in h5dread_f_c
        at /home/Gayathri/software2/hdf5-1.12.0/fortran/src/H5Df.c:1243
#23  0x2aaaadf20432 in __h5_gen_MOD_h5dread_rkind_4_rank_2
        at /home/Gayathri/software2/hdf5-1.12.0/fortran/src/H5_gen.F90:3034
#24  0x16fd35d in read_nasasmap_data_
        at ../dataassim/obs/NASA_SMAPsm/read_NASASMAPsm.F90:1320
#25  0x17075bc in read_nasasmapsm_
        at ../dataassim/obs/NASA_SMAPsm/read_NASASMAPsm.F90:350
#26  0x60957b in readdaobs_
        at ../core/LIS_dataassim_FTable.c:840
#27  0x5b96c9 in __lis_daobservationsmod_MOD_lis_readdaobservations
        at ../core/LIS_DAobservationsMod.F90:1362
#28  0x184047c in __retrospective_runmod_MOD_lis_run_retrospective
        at ../runmodes/retrospective/retrospective_runMod.F90:200
#29  0x75e91e in lisrun_
        at ../core/LIS_runmode_FTable.c:202
#30  0x10b11b1 in lisdrv
        at ../offline/lisdrv.F90:70
#31  0x10b1297 in main
        at ../offline/lisdrv.F90:57
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node nesac-hpc exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Is it because of the HDF5 library?

In the first test case I changed the "Max snow albedo data source: Barlage_Native" to "NCEP_LIS" as it was giving error- HDF4 required to process Barlage max snow albedo!. Now getting an error in the data assimilation step.

Files are attached.
files.zip

Thank you
Gayathri

[bmcandr]

I am not an expert in this area, but I suspect the issue may be memory related and not necessarily due to the HDF5 library. This is because your log file shows that LIS successfully read at least 10 SMAP files successfully.

The error appears to have occurred in this function in the HDF5 library in H5MM.c which is a module that contains memory management functions:

/*-------------------------------------------------------------------------
 * Function:    H5MM__sanity_check_block
 *
 * Purpose:     Check a block wrapper around a buffer to validate it.

---

Regarding switching your max albedo source... Yes, we should probably indicate in the Public Testcase Walkthrough that LIS must be built with the optional HDF libraries to run the testcases. Thank you for bringing this to our attention.

[Gayathridevi123]

Hi,

I tried "ulimit -s unlimited" to have an unlimited stack size but still got the same error.

[sudarsan3010]

@Gayathridevi123 I am facing the same issues. How did you resolve it?!

[Gayathridevi123]

Hi @sudarsan3010,

Sorry, I'm not exactly remembering now as I tried long back ago...I think I reinstalled HDF5 library and tried with the recent version of LISF and like that it worked for me. In my case HDF5 library had some issues.