selectDWD returns outdated URLs

[mottensmann]

Queries for historical weather records using selectDWD returns URLs that cannot be opened. In the past those URLs worked but seem outdated now. I am not able to figure out where these URLs are generated but perhaps the example can help in solving the issue.

##  example
library(rdwd)
library(magrittr)
stations <- c("Belm", "Salzuflen, Bad", "Bielefeld-Deppendorf")
url <- lapply(stations, selectDWD, res = 'daily', var = 'kl', per = 'h')  
url

All URLs end with 20171231 and do not exist as shown below for the first.

[[1]]
[1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/tageswerte_KL_00342_20101201_20171231_hist.zip"
[[2]]
[1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/tageswerte_KL_04371_19350101_20171231_hist.zip"
[[3]]
[1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/tageswerte_KL_07106_20060901_20171231_hist.zip"

dataDWD(url[[1]], quiet = T)

Error in download.file(url = file[i], destfile = outfile[i], quiet = TRUE, :
cannot open URL 'ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/tageswerte_KL_00342_20101201_20171231_hist.zip'
In addition: Warning message:
In download.file(url = file[i], destfile = outfile[i], quiet = TRUE, :
InternetOpenUrl failed: ''

The correct URLs end with 20181231 and dataDWD can handle those

out <- lapply(url, stringr::str_replace, "20171231", "20181231") %>% 
  lapply(., dataDWD, quiet = T)
head(out[[1]])

STATIONS_ID MESS_DATUM QN_3 FX FM QN_4 RSK RSKF SDK SHK_TAG NM VPM PM TMK UPM TXK TNK TGK eor
1 342 2010-12-01 10 NA NA 3 0.0 4 3.0 NA NA NA NA -7.0 NA -4.9 -9.1 -9.2 eor
2 342 2010-12-02 10 7.0 NA 3 0.5 4 0.0 0 7.7 3.1 996.5 -7.4 90 -6.5 -8.6 -8.2 eor
3 342 2010-12-03 10 7.6 NA 3 0.1 4 0.0 1 6.9 3.4 998.0 -5.6 85 -3.9 -7.5 -9.7 eor
4 342 2010-12-04 10 12.0 NA 3 4.7 4 3.6 1 5.0 4.1 1000.5 -3.2 84 -0.8 -4.9 -6.6 eor
5 342 2010-12-05 10 13.1 NA 3 2.5 4 0.0 3 7.8 6.2 989.3 0.7 97 1.4 -1.2 -1.1 eor
6 342 2010-12-06 10 5.5 NA 3 0.1 4 0.0 1 7.9 5.8 989.7 0.2 95 1.6 -4.5 -6.3 eor

From session.info()

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] magrittr_1.5 rdwd_0.11.0

[brry]

DWD has just released the new version of the historical data including 2018. rdwd 0.11.9 on github already reflects the changes. I'm working on some other changes and hope to push to CRAN soon. Thanks for filing this issue!

[brry]

By the way, selectDWD can handle vectorized inputs. It takes less code and execution time is slightly smaller:

url <- selectDWD(stations, res='daily', var='kl', per='h') 

[brry]

The new version of rdwd is now on CRAN.