Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename hydrolib converted dataset varname/dims/vars #527

Closed
19 tasks done
veenstrajelmer opened this issue Sep 21, 2023 · 1 comment · Fixed by #567
Closed
19 tasks done

Rename hydrolib converted dataset varname/dims/vars #527

veenstrajelmer opened this issue Sep 21, 2023 · 1 comment · Fixed by #567

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Sep 21, 2023 •
edited

As described in Deltares/HYDROLIB-core#553

  • dfmt plipointsDataset to locationDataset >> won't do
  • rename dimension plipoints to locations (not location) >> node
  • rename vars to location_x, location_y, and location_name >> lon, lat, station_id
  • set station_id and station_names as coordinates if present as data_vars: ds = ds.set_coords(['station_id','station_names'])
  • move internally from pd.DataFrame to GeoDataFrame with Linestring (not Points) >> we need points for xyn and xyz files
  • feed gdf with points/linestrings to interp_regularnc_to_plipointsDataset (instead of concatenated pandas dataframe)
  • consider renaming plipointsDataset_to_ForcingModel, interpolate_tide_to_plipoints, interp_regularnc_to_plipoints, interp_uds_to_plipoints and interp_hisnc_to_plipoints (also raise deprecationwarnings) >> won't do
  • maybe wait for Align FM ncbnd and dfmt data_interp dataset conventions #541
  • fix cmems in modelbuilder notebook, due to depths not passed in open_dataset_extra(), new testcase test_open_dataset_extra_correctdepths()
  • remove duplicate lat/latitude and lon/longitude from data_interp
  • check if ncbnd nodedim can be renamed without fm crash, might not be read. If so, rename to location or so
  • test ncbnd file from modelbuilder/interpolate_nc_to_bc in fm run, easier by converting fews nc files from C01 model from JIRA issue to bc and back with new ForcingModel_to_plipointsDataset >> run succeeds, but we need dims in order time-node-z otherwise results will be different: https://issuetracker.deltares.nl/browse/UNST-7402
  • include maybe_convert_fews_to_dfmt in plipointsDataset_to_ForcingModel and ForcingModel_to_plipointsDataset? Also put dimension transpose in that function?
  • dataset from dfmt.forcinglike_to_Dataset() has var attrs locationname and name (are equal), prevent this
  • prevent hardcoded renaming of z/node to depth/location in maybe_convert_fews_to_dfmt?
  • fix duplication percentage by defining cmems dummy dataset (pytest fixture or dfmt.download?) With that, also fix TODO in test_interpolate_grid2bnd.py
  • make tide interp testcase available on github with gtsm_opendap instead of fes2014
  • run entire testbank
  • solve code smells?

Follow up issue: #590
@veenstrajelmer veenstrajelmer mentioned this issue Jul 6, 2023
Open
@veenstrajelmer
Copy link
Collaborator Author

veenstrajelmer commented Oct 3, 2023 •
edited

Convert to FM/FEWS dataset instead, as mentioned in UNST-6187. Example file: p:\dflowfm\maintenance\JIRA\06000-06999\06187\C01\salinity_DCSM-FM_OB_all.nc

<xarray.Dataset>
Dimensions:        (time: 14, node: 206, z: 50)
Coordinates:
  * time           (time) datetime64[ns] 2023-08-12T06:00:00 ... 2023-08-15T1...
    lat            (node) float64 ...
    lon            (node) float64 ...
  * z              (z) float32 -5.728e+03 -5.275e+03 -4.833e+03 ... -1.54 -0.49
Dimensions without coordinates: node
Data variables:
    station_id     (node) |S64 ...
    station_names  (node) |S255 ...
    so             (time, node, z) float32 ...
Attributes:
    Conventions:                  CF-1.6
    title:                        Data
    institution:                  Deltares
    source:                       Export NETCDF-CF_ZLAYERS from Delft-FEWS
    history:                      2023-08-16 15:17:20 GMT: exported from Delf...
    references:                   http://www.delft-fews.com
    Metadata_Conventions:         Unidata Dataset Discovery v1.0
    summary:                      Data exported from Delft-FEWS
    date_created:                 2023-08-16 15:17:20 GMT
    fews_implementation_version:  2023.01
    fews_patch_number:            122941
    fews_build_number:            121175

Although it would still be convenient to slightly adjust the dataset like this. Mainly since so would then be the only date_vars:

<xarray.Dataset>
Dimensions:     (time: 14, node: 206, z: 50)
Coordinates:
  * time        (time) datetime64[ns] 2023-08-12T06:00:00 ... 2023-08-15T12:0...
    lat         (node) float64 ...
    lon         (node) float64 ...
    station_id  (node) |S64 ...
  * z           (z) float32 -5.728e+03 -5.275e+03 -4.833e+03 ... -1.54 -0.49
Dimensions without coordinates: node
Data variables:
    so          (time, node, z) float32 ...
Attributes:
    Conventions:                  CF-1.6
    title:                        Data
    institution:                  Deltares
    source:                       Export NETCDF-CF_ZLAYERS from Delft-FEWS
    history:                      2023-08-16 15:17:20 GMT: exported from Delf...
    references:                   http://www.delft-fews.com
    Metadata_Conventions:         Unidata Dataset Discovery v1.0
    summary:                      Data exported from Delft-FEWS
    date_created:                 2023-08-16 15:17:20 GMT
    fews_implementation_version:  2023.01
    fews_patch_number:            122941
    fews_build_number:            121175

@xldeltares xldeltares mentioned this issue Oct 6, 2023
Open
13 tasks
@veenstrajelmer veenstrajelmer linked a pull request Oct 6, 2023 that will close this issue
527 rename hydrolib converted dataset varnamedimsvars 1 #567
Merged
This was referenced Oct 13, 2023
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

527 rename hydrolib converted dataset varnamedimsvars 1 Deltares/dfm_tools
1 participant
@veenstrajelmer