Skip to content

Commit

Permalink
dataDWD: force can now be number of hours after which to re-download
Browse files Browse the repository at this point in the history
  • Loading branch information
brry committed Oct 26, 2019
1 parent 2540714 commit f45b080
Show file tree
Hide file tree
Showing 4 changed files with 14 additions and 8 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: rdwd
Title: Select and Download Climate Data from 'DWD' (German Weather Service)
Version: 1.1.34
Version: 1.1.35
Date: 2019-10-26
Depends: R(>= 2.10)
SystemRequirements: Pandoc (>= 1.12.3), pandoc-citeproc
Expand Down
2 changes: 0 additions & 2 deletions NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ rdwd ToDo-list / wishlist / issues
- readDWD.data: figure out reason for fread trouble (reported in emails by some users)
- readDWD.meta: better analasis of fwf widths needed
- indexFTP: read tree with data.table/vroom
- dataDWD: change/allow force to be number of hours after which to re-download
- function for index updates
- selectDWD("Berlin", res="monthly", var="kl", per="h", exactmatch = F) should only return stations with data, with a single warning only


Expand Down
14 changes: 10 additions & 4 deletions R/dataDWD.R
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@
#' Created if not existent. DEFAULT: "DWDdata" at current \code{\link{getwd}()}
#' @param force Logical (vector): always download, even if the file already exists in \code{dir}?
#' Use NA to force re-downloading files older than 24 hours.
#' If FALSE, it is still read (or name returned). DEFAULT: FALSE
#' Use a numerical value to force after that amount of hours.
#' Note you might want to set \code{overwrite=TRUE} as well.
#' If FALSE, the file is still read (or name returned). DEFAULT: FALSE
#' @param overwrite Logical (vector): if force=TRUE, overwrite the existing file
#' rather than append "_1"/"_2" etc to the filename? DEFAULT: FALSE
#' @param sleep Number. If not 0, a random number of seconds between 0 and
Expand Down Expand Up @@ -150,10 +152,14 @@ outfile <- gsub(paste0(base,"/"), "", file)
outfile <- gsub("/", "_", outfile)

# force=NA management
force_old <- difftime(Sys.time(), file.mtime(outfile), units="h") > 24
if(is.null(force)) stop("force cannot be NULL. Must be TRUE, FALSE, NA or a number.")
force <- rep(force, length=length(outfile)) # recycle vector

This comment has been minimized.

Copy link
@phit0

phit0 Jan 4, 2024

Hi! For the following use of dataDWD this does not seem to work as expected.

urls <- rdwd::selectDWD(
      res = "daily",
      id = c(15200, 15190),
      var = "kl",
      per = "hr"
    )
rdwd::dataDWD(
      urls,
      sleep = 1,
      read = FALSE,
      force = c(365*24, 6),
      dir = tempdir())

While debugging I found after line 154 force is

[1] 8760    6 8760    6

while outfile yields

[1] "daily_kl_historical_tageswerte_KL_15190_20180701_20221231_hist.zip"
[2] "daily_kl_historical_tageswerte_KL_15200_20130601_20221231_hist.zip"
[3] "daily_kl_recent_tageswerte_KL_15190_akt.zip"                       
[4] "daily_kl_recent_tageswerte_KL_15200_akt.zip" 

This results in only half of my _hist files being updated and only half of my _akt files being updated.

This comment has been minimized.

Copy link
@brry

brry Jan 4, 2024

Author Owner

Good catch - I'll have a more detailed look as soon as I can.
For now, you could use force = rep(c(365*24, 6), each=2) if I'm not mistaken...

This comment has been minimized.

Copy link
@phit0

phit0 Mar 8, 2024

force = c(rep(365, length(grep("hist", urls))), rep(6, length(grep("akt", urls)))), was my solution :)

force[is.na(force) & force_old] <- TRUE # force if old
force[is.na(force)] <- FALSE # otherwise don't
fT <- sapply(force, isTRUE)
fF <- sapply(force, isFALSE)
if(any(fT)) force[fT] <- 0
if(any(fF)) force[fF] <- Inf
force[is.na(force)] <- 24
force <- difftime(Sys.time(), file.mtime(outfile), units="h") > force

dontdownload <- file.exists(outfile) & !force
if( any(dontdownload) & !quiet )
Expand Down
4 changes: 3 additions & 1 deletion man/dataDWD.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit f45b080

Please sign in to comment.