t.rast.import.netcdf: Create STRDS from netcdf-cf data

[ninsbl]

No description provided.

[veroandreo]

I added some minor grammar fixes.

Testing with one of the manual page examples, I get:

./t.rast.import.netcdf.py output=SeNorge semantic_labels=semantic_labels.conf memory=2048 nprocs=2 start_time="2020-08-01" end_time="2021-08-01" input=https://thredds.met.no/thredds/fileServer/senorge/seNorge_2018/Archive/seNorge2018_2020.nc
ERROR 1: CPLCreateUserFaultMapping(): syscall(__NR_userfaultfd) failed
Traceback (most recent call last):
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1202, in <module>
    sys.exit(main())
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1030, in main
    inputs_dict = [
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1031, in <listcomp>
    parse_netcdf(
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 734, in parse_netcdf
    ncdf_metadata = ncdf.GetMetadata()
AttributeError: 'NoneType' object has no attribute 'GetMetadata'

Is this perhaps something on my end?

[veroandreo]

Add a link to the standards perhaps?

[ninsbl]

Link to the cf-convention added in 387c75c

[ninsbl]

Is this perhaps something on my end?

Yes, I think so. In fact I ran into the same issue myself. It is the combination of GDAL version and Linux Kernel: https://www.mail-archive.com/gdal-dev@lists.osgeo.org/msg37419.html
For double checking: what is your GDAL version (gdalinfo --version)?

Anyway, I added a note in the manual about it. I could add a check for the GDAL version (and OS) as well, but Fedora 36 ships GDAL 3.4.3 and also Ubunutu 20.04 ships newer versions of GDAL, so the issue should disappear on Linux...

[veroandreo]

Is this perhaps something on my end?

Yes, I think so. In fact I ran into the same issue myself. It is the combination of GDAL version and Linux Kernel: https://www.mail-archive.com/gdal-dev@lists.osgeo.org/msg37419.html For double checking: what is your GDAL version (gdalinfo --version)?

Anyway, I added a note in the manual about it. I could add a check for the GDAL version (and OS) as well, but Fedora 36 ships GDAL 3.4.3 and also Ubunutu 20.04 ships newer versions of GDAL, so the issue should disappear on Linux...

I updated to Fedora 36 and have GDAL 3.4.3 now. I ran the same example than before in the NC location and now it seems it's doing something... This is what I get:

ERROR: Projection of datasetERROR:  does notProjection  of appeardataset to match  doescurrent  notlocation.
 
       appearLocation  toPROJ_INFO  match is:current
        name:location. 
Lambert
       Location  PROJ_INFO Conformalis: 
       Conicname:
        proj:Lambert  lccConformal
        datum:Conic nad83
       a: 
       6378137.0proj:
       es:  lcc0.006694380022900787
       
       lat_1:datum:  36.16666666666666nad83
       
       lat_2:a:  34.33333333333334
       6378137.0lat_0:
        es:33.75 
       0.006694380022900787lon_0:
        lat_1:-79 
       x_0: 36.16666666666666609601.22
       
       y_0: 0
       lat_2: no_defs: defined
       init:34.33333333333334 EPSG:3358
       
lat_0:
        Dataset33.75 PROJ_INFO is:
       
       lon_0:name:  -79
       unnamed
       ellps: grs80
       x_0:proj:  utm609601.22
       zone:
        y_0: 33
       0no_defs:
        no_defs:defined defined

       
       init:Difference  in: proj
EPSG:3358
       In
 
       caseDataset  ofPROJ_INFO  nois: significant
        name: unnameddifferences
        ellps:in grs80 the
        proj:projection utm 
       definitions,
       use zone:the  -o33 flag
        no_defs:to  definedignore
 
       themDifference  andin:  useproj 
current
        Inlocation  casedefinition. 
       ofConsider  nogenerating  significanta  differencesnew  inlocation  thefrom  projectionthe  definitions,input
        usedataset  theusing  -othe 
       flag 'location'to ignore them and use current  parameter.
location definition.
       Consider generating a new location from the input dataset using the
       'location' parameter.
Exception in thread Thread-7 (_handle_results):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.10/multiprocessing/pool.py", line 579, in _handle_results
    task = get()
  File "/usr/lib64/python3.10/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: CalledModuleError.__init__() missing 3 required positional arguments: 'module', 'code', and 'returncode'

I guess the problem is the region definition and CRS, but it's curious how it got printed...

[veroandreo]

Next, I tried in a 4326 location, region Argentina and got this:

ERROR: Unable to execute transaction:
--#############################################################################
-- This SQL script generates the table in which all registered GRASS_MAP
maps
-- are listed in.
--
-- This table will be created for each STDS.
--
-- Author: Soeren Gebbert soerengebbert <at> googlemail <dot> com
--#############################################################################

-- SPACETIME_NAME is a placeholder for specific stds name (SQL compliant):
name_mapset
-- GRASS_MAP is a placeholder for specific map type: raster, raster3d or
vector

-- This table stores the.ids of the GRASS_MAP maps registered in the
current spacetime GRASS_MAP table
CREATE TABLE raster_map_register_4da62fbf8ec447e7a78345c58fcfd98e (
id VARCHAR NOT NULL, -- This column is a primary key storing the registered
GRASS_MAP map.ids
PRIMARY KEY (id)
);
CREATE INDEX raster_map_register_4da62fbf8ec447e7a78345c58fcfd98e_index ON
raster_map_register_4da62fbf8ec447e7a78345c58fcfd98e (id);INSERT INTO
strds_base ( mapset ,id ,name ,creator ,creation_time ,temporal_type
,semantic_type ,modification_time ) VALUES ('pruebas' ,'SeNorge@pruebas'
,'SeNorge' ,'veroandreo' ,'2022-10-04 23:24:28.080690' ,'absolute' ,'mean'
,'2022-10-04 23:24:28.080698') ;
INSERT INTO strds_absolute_time ( id ,start_time ,end_time ,granularity
,map_time ) VALUES ('SeNorge@pruebas' ,NULL ,NULL ,NULL ,NULL) ;
INSERT INTO strds_spatial_extent ( id ,north ,south ,east ,west ,top
,bottom ,proj ) VALUES ('SeNorge@pruebas' ,NULL ,NULL ,NULL ,NULL ,NULL
,NULL ,'XY') ;
INSERT INTO strds_metadata ( id ,title ,description ,command
,number_of_maps ,min_max ,max_max ,min_min ,max_min ,nsres_min ,nsres_max
,ewres_min ,ewres_max ,aggregation_type ,number_of_semantic_labels
,raster_register ) VALUES ('SeNorge@pruebas' ,'seNorge_2018, tn: daily
minimum temperature (18UTC-18UTC), time: min, version: 22.09, seNorge_2018,
tg: daily mean temperature (06UTC-06UTC), time: mean, version: 2
Traceback (most recent call last):
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1204, in <module>
    sys.exit(main())
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1120, in main
    tgis.open_new_stds(
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/open_stds.py", line 213, in open_new_stds
    sp.insert(dbif)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 430, in insert
    dbif.execute_transaction(statement)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/core.py", line 1247, in execute_transaction
    return self.connections[mapset].execute_transaction(statement)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/core.py", line 1580, in execute_transaction
    self.cursor.executescript(statement)
sqlite3.OperationalError: table strds_metadata has no column named number_of_semantic_labels

I get it that Norwegian climate data won't be available for Argentina ;) but perhaps something can be done with the error that gets printed?

[veroandreo]

Ok, now I set the region to Norway in a 4326 location, created the semantic label conf file, but it still produces error:

./t.rast.import.netcdf.py output=SeNorge semantic_labels=semantic_labels.conf memory=2048 nprocs=2 start_time="2020-08-01" end_time="2021-08-01" input=https://thredds.met.no/thredds/fileServer/senorge/seNorge_2018/Archive/seNorge2018_2020.nc
WARNING: Overwriting space time raster dataset <SeNorge> and unregistering
         all maps
ERROR: Unable to execute :
SELECT id , title , description , command , number_of_maps , min_max ,
max_max , min_min , max_min , nsres_min , nsres_max , ewres_min , ewres_max
, aggregation_type , number_of_semantic_labels , raster_register FROM
strds_metadata WHERE id = 'SeNorge@pruebas';
Traceback (most recent call last):
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1204, in <module>
    sys.exit(main())
  File "/home/veroandreo/software/grass-addons/src/temporal/t.rast.import.netcdf/./t.rast.import.netcdf.py", line 1120, in main
    tgis.open_new_stds(
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/open_stds.py", line 198, in open_new_stds
    sp.delete(dbif)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 2156, in delete
    self.metadata.select(dbif)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/base.py", line 392, in select
    dbif.execute(sql, mapset=mapset)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/core.py", line 1201, in execute
    return self.connections[mapset].execute(statement, args)
  File "/home/veroandreo/software/grass-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/temporal/core.py", line 1540, in execute
    self.cursor.execute(statement)
sqlite3.OperationalError: no such column: number_of_semantic_labels

[ninsbl]

Thanks @veroandreo for bearing with me here.

In the code, the GRASS version is checked to figure out if semantic labels can be used. Your test shows, that this is not safe.
I guess you have a TGIS DB version < 3 and are running GRASS 8. So I need to catch this differently...

When you got the projection errors, did you use the version from the latest commit (6e3f80c)?
I experienced the same when importing (r.in.gdal) instead of linking (r.external) and tried to fix exactly that in 3e8c060.

The region setting should only have an effect if you use the r-flag...

Norwegian data is out of bounds of the NC CRS, which is why reprojection may fail in that case anyway. Not sure how to catch this though...

Anyway thanks for continuous tries and feedback!

[ninsbl]

I take the liberty to merge this one now. If there are issues left that I did not catch, please open a bugreport...

[neteler]

Now online:
https://grass.osgeo.org/grass8/manuals/addons/t.rast.import.netcdf.html