NETCDF writing and reading issues

[ShinyFabio]

Hi,
I was trying to use your package to downscale the ERA5 dew point temperature. I need a monthly product for 2020 and 2022 hence my code just for January 2020 is the following:

Extent <- raster::extent(c(7,13,44,46)) # the extent of Po Valley
output_dir = "C:/Users/fabio/Desktop/KrigR"

Pipe_Krig <- krigR(
  Variable = "2m_dew_point_temperature",
  Type = "reanalysis",
  DataSet = "era5-land",
  Dir = output_dir,
  DateStart = "2020-01-01",
  DateStop = "2020-01-31",
  TResolution = "month",   #Temporal resolution of final product.
  TStep = 1,   ##1 because I want just one product per month
  FUN = "mean",
  Extent = Extent,
  API_User = API_User,
  API_Key = API_Key,
  Target_res = .05,
  nmax = Inf   #Controls local kriging. Number of nearest observations to be used kriging of each observation.
)

But that's the log:

Staging 1 download(s).
  |=================================================================================================================================| 100%
Checking for known data issues.
Error in R_nc4_open: No such file or directory
Error in ncdf4::nc_open(filename, readunlim = FALSE, suppress_dimvals = TRUE) : 
  Error in nc_open trying to open file C:/Users/fabio/Desktop/KrigR/0001_dew_point_temperature_2020-01-01_2020-01-31_month.nc (return_on_error= FALSE )

So I tried to use first the download function and I had another error:

Error in warn_or_error(error_msg, error = fail_is_error) : 
  Data transfer failed with permanent client error: the request you have submitted is not valid.
Reason given: Ambiguous : 2m_dew_point_temperature could be 2M TEMPERATURE ANOMALY OF AT MOST -1K or 2M TEMPERATURE ANOMALY OF AT MOST -2K; No request.

I don't understand the problem. For the variable name, I checked the API code from here. https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land?tab=form

[ErikKusch]

Hi @ShinyFabio, thanks for bringing this to my attention. I believe this may have to do with KrigR still relying on "old" raster package functionality - resolving this is at the top of my lengthy development goal list.

For the time being, could you let me know your package versions? You may have to return to earlier versions for this to be fixed.

To address your second part of your question, the correct variable name is "2m_dewpoint_temperature" (notice there is no underscore between dew and point). No idea how this download call worked in the first place.

Cheers,
Erik

[ShinyFabio]

Oh, you're right I added another underscore! Unfortunately, it still doesn't work. Now both KrigR() and download_ERA() functions return this error:

Error in ncdf4::ncvar_def(varname, varunit, list(xdim, ydim, zdim), NAflag,  : 
  unused argument (sources = "C:\\Users\\fabio\\Desktop\\KrigR\\0001_2m_temperature_2020-01-01_2020-01-31_month.nc")

My KrigR version is 0.1.2 (I installed it three days ago).

[ErikKusch]

Please provide the versions of all loaded packages.

[ShinyFabio]

Sure. Here's my sessionInfo

R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=Italian_Italy.utf8  LC_CTYPE=Italian_Italy.utf8    LC_MONETARY=Italian_Italy.utf8 LC_NUMERIC=C                  
[5] LC_TIME=Italian_Italy.utf8    

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] KrigR_0.1.2       httr_1.4.5        stars_0.6-0       abind_1.4-5       fasterize_1.0.4   sf_1.0-11         lubridate_1.9.2  
 [8] automap_1.1-9     doSNOW_1.0.20     snow_0.4-4        doParallel_1.0.17 iterators_1.0.14  foreach_1.5.2     rgdal_1.6-5      
[15] raster_3.6-20     sp_1.6-0          stringr_1.5.0     keyring_1.3.1     ecmwfr_1.5.0      ncdf4_1.21       

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10        lattice_0.20-45    FNN_1.1.3.1        class_7.3-20       zoo_1.8-11         assertthat_0.2.1   gstat_2.1-0       
 [8] utf8_1.2.3         R6_2.5.1           R.matlab_3.7.0     plyr_1.8.8         e1071_1.7-13       ggplot2_3.4.1      pillar_1.8.1      
[15] rlang_1.1.0        curl_5.0.0         rstudioapi_0.14    R.utils_2.12.2     R.oo_1.25.0        munsell_0.5.0      proxy_0.4-27      
[22] compiler_4.2.2     pkgconfig_2.0.3    tidyselect_1.2.0   tibble_3.2.0       intervals_0.15.3   codetools_0.2-18   reshape_0.8.9     
[29] fansi_1.0.4        spacetime_1.2-8    dplyr_1.1.0        R.methodsS3_1.8.2  grid_4.2.2         jsonlite_1.8.4     lwgeom_0.2-11     
[36] gtable_0.3.1       lifecycle_1.0.3    DBI_1.1.3          magrittr_2.0.3     scales_1.2.1       units_0.8-1        KernSmooth_2.23-20
[43] cli_3.6.0          stringi_1.7.12     cachem_1.0.7       xts_0.13.0         generics_0.1.3     vctrs_0.5.2        tools_4.2.2       
[50] glue_1.6.2         fastmap_1.1.1      timechange_0.2.0   colorspace_2.1-0   terra_1.7-18       classInt_0.4-9     memoise_2.0.1 

[ErikKusch]

Right- thank you for that. As the issue traces back to the saving of a NETCDF file, this must be an issue with the new version of the ncdf4 package.

Can you try to install a previous version of ncdf4? Version 1.19 works for me (as seen in my environment below). Obviously, this suggestion is a bandaid fix and I will continue to investigate solutions.

R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] KrigR_0.1.2       httr_1.4.2        stars_0.5-5       abind_1.4-5       fasterize_1.0.3  
 [6] sf_1.0-5          lubridate_1.8.0   automap_1.0-14    doSNOW_1.0.19     snow_0.4-4       
[11] doParallel_1.0.16 iterators_1.0.13  foreach_1.5.1     rgdal_1.5-28      raster_3.5-11    
[16] sp_1.4-6          stringr_1.4.0     keyring_1.3.0     ecmwfr_1.3.0      ncdf4_1.19       

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10        lattice_0.20-41    FNN_1.1.3          prettyunits_1.1.1  class_7.3-17      
 [6] ps_1.6.0           zoo_1.8-9          utf8_1.2.2         assertthat_0.2.1   gstat_2.0-8       
[11] rprojroot_2.0.2    sdm_1.1-8          R6_2.5.1           plyr_1.8.6         e1071_1.7-9       
[16] pillar_1.6.4       rlang_0.4.10       curl_4.3.2         callr_3.7.0        desc_1.4.0        
[21] devtools_2.4.3     proxy_0.4-26       compiler_4.0.2     pkgconfig_2.0.3    pkgbuild_1.3.1    
[26] tidyselect_1.1.1   tibble_3.1.6       intervals_0.15.2   codetools_0.2-16   fansi_0.5.0       
[31] reshape_0.8.8      spacetime_1.2-5    crayon_1.4.2       dplyr_1.0.7        withr_2.4.3       
[36] grid_4.0.2         lwgeom_0.2-8       lifecycle_1.0.1    DBI_1.1.2          magrittr_2.0.1    
[41] units_0.7-2        KernSmooth_2.23-17 cli_3.1.0          stringi_1.7.6      cachem_1.0.6      
[46] fs_1.5.2           remotes_2.4.2      testthat_3.1.1     vctrs_0.3.8        ellipsis_0.3.2    
[51] xts_0.12.1         generics_0.1.1     tools_4.0.2        glue_1.4.2         purrr_0.3.4       
[56] processx_3.5.2     pkgload_1.2.4      fastmap_1.1.0      terra_1.7-18       sessioninfo_1.2.2 
[61] classInt_0.4-3     memoise_2.0.1      usethis_2.1.5    

[ShinyFabio]

Hi, I tried to install another version of ncdf4 but I could install only v1.21 (the actual version) and v1.20 (but KrigR doesn't work with this version too). I tested up to the v1.16 but they don't install (some errors during compilation). I tried also downloading the tar.gz and install locally but same.

[ErikKusch]

Hi - that's a bummer. As this is a netcdf/ncdf4 issue, I am afraid there is no way to resolve this directly in KrigR. However, I can build a workaround by allowing a user selection of output file-type.

Unfortunately, KrigR is currently not my highest priority (I just started a new position this week). Please bear with me as implementation of this fix may take me a week or so. Feel free to remind if a fix isn't up within seven days from now.

Cheers,
Erik

[ShinyFabio]

I've just spent few hours trying to help you. Your code returns the error when running the following line:

writeRaster(x = Era5_ras, filename = file.path(Dir, FileName), overwrite = TRUE, format="CDF", varname = Variable)
I tried to use the writeRaster() from {terra} package instead of {raster} package but no luck. The error comes only with CDF format. Even if I simply load the .nc file with raster() / stack() function and then trying to save it with writeRaster() I have the error.

After a while I've found another function inside the terra package called writeCDF() that write as NetCDF a SpatRaster or SpatRasterDataset. With this function the error disappears. I tested on my computer the fix and it works. The only difference is that krigR requires the FileName (can't be null). I'm sending to you a pull request.

However this is a workaround. The error comes from the terra/raster/ncdf4 packages. I opened an issue on the raster github page.

rspatial/raster#308

[ErikKusch]

Hi,

thank you very much for going the extra mile to resolving this issue! I really appreciate the help. I have made the necessary changes and this issue should no longer persist.

[ShinyFabio]

Hi, I just want to inform you that the issue with writeRaster() should be fixed. Have a look here:

rspatial/raster#308 (comment)