segfault when cmorizing nwdFracLut variable

[etiennesky]

This variable is in cmip6-output-control-files/CMIP/EC-EARTH-CC/cmip6-experiment-CMIP-historical/cmip6-data-request-varlist-CMIP-historical-EC-EARTH-CC.json .

I don't have access to the logfiles, only in the past the problem went away by removing the entry in the .json file.

[etiennesky]

Hi @nierad @treerink I cannot set labels, nor assignees to this issue...

[nierad]

Hi @etiennesky!
Can you send me a nwdFracLut_monthly output file please. Maybe there is sth weird in it...

[etiennesky]

Hi @nierad you can find the file here ftp://bscesftp.bsc.es/etourign/nwdFracLut_monthly.out.gz

here is the log of the cmorization job with segfault when cmorizing nwdFracLut_monthly

2019-05-27 18:23:59 INFO:ece2cmor3.lpjg2cmor: Processing file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/nep_monthly.out
2019-05-27 18:24:00 INFO:ece2cmor3.lpjg2cmor: Creating lpjg netcdf file for variable nep for year 1979
2019-05-27 18:24:04 INFO:ece2cmor3.lpjg2cmor: CMORizing variable nep in table Emon form nep in file nep.out...
2019-05-27 18:24:05 INFO:ece2cmor3.lpjg2cmor: CMOR closed file /gpfs/scratch/cns54/cns54230/a1tf/19790101/fc00/runtime/cmor_outputs/cmor_1/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r1i1p1f1/Emon/nep/gr/v20190510/nep_Emon_EC-Earth3_historical_r1i1p1f1_gr_197901-197912.nc
2019-05-27 18:24:05 INFO:ece2cmor3.lpjg2cmor: Uncompressing file /gpfs/scratch/cns54/cns54230/a1tf/19790101/fc00/runtime/output/lpjg/001/fLuc_monthly.out.gz to temporary file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/fLuc_monthly.out
2019-05-27 18:24:05 INFO:ece2cmor3.lpjg2cmor: Processing file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/fLuc_monthly.out
2019-05-27 18:24:05 INFO:ece2cmor3.lpjg2cmor: Creating lpjg netcdf file for variable fLuc for year 1979
2019-05-27 18:24:10 INFO:ece2cmor3.lpjg2cmor: CMORizing variable fLuc in table Emon form fLuc in file fLuc.out...
2019-05-27 18:24:10 INFO:ece2cmor3.lpjg2cmor: CMOR closed file /gpfs/scratch/cns54/cns54230/a1tf/19790101/fc00/runtime/cmor_outputs/cmor_1/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r1i1p1f1/Emon/fLuc/gr/v20190510/fLuc_Emon_EC-Earth3_historical_r1i1p1f1_gr_197901-197912.nc
2019-05-27 18:24:10 INFO:ece2cmor3.lpjg2cmor: Uncompressing file /gpfs/scratch/cns54/cns54230/a1tf/19790101/fc00/runtime/output/lpjg/001/nwdFracLut_monthly.out.gz to temporary file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/nwdFracLut_monthly.out
2019-05-27 18:24:10 INFO:ece2cmor3.lpjg2cmor: Processing file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/nwdFracLut_monthly.out
2019-05-27 18:24:13 INFO:ece2cmor3.lpjg2cmor: Creating lpjg netcdf file for variable nwdFracLut for year 1979
2019-05-27 18:24:21 INFO:ece2cmor3.lpjg2cmor: Creating singleton axis for typenwd using file /gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/ece2cmor/ece2cmor3/veg-fc00-197901/nwdFracLut_monthly.out...
2019-05-27 18:24:21 INFO:ece2cmor3.lpjg2cmor: CMORizing variable nwdFracLut in table Emon form nwdFracLut in file nwdFracLut.out...
/gpfs/scratch/cns54/cns54230/a1tf/auto-ecearth3/plugins/cmorization.sh: line 255: 173043 Segmentation fault      ./ece2cmor.py --lpjg --exp $EXPID --conf metadata-conf-veg-${START_date}-${chunk}-${MEMBER}.json --tmpdir $tmpdir --vars ```

[treerink]

To prevent a cmorising crash in v1.2.0 nwdFracLut has been taken off from the json data request files.

[treerink]

As I understand from a mail conversation at 23 sept 2019 this nwdFracLut can be cmorised without a segfault. However, the cmorised output is still under debate in #503 so it is not ready for publishing. Note also that earlier this variable has been taken off from the json data request file by excepting it in a latest stage in drq2varlist.

Closing this issue because the segfault seems not to occur longer. Please reopen again, if the segfault occurs still at any platform.

[etiennesky]

Hi I still get the segfault on my platform, but I am using 1.2.0.

[treerink]

For this 1.2.0 or 1.2.1 should not matter for this. However I recommend to recreate your ece2cmor3 environment using the latest master.

[zklaus]

To be fair, the instance I found was done with

processed by ece2cmor vv1.1.0, git rev. 032f6287076b212e5c49922af94a0ddecb191a16

according to the global attributes.

[etiennesky]

hum maybe this is a regression then?

[treerink]

If not something platform related, some commit between the 4th and the 20th of June ...