with_tz silently gives wrong results for many timezones

[apiskors]

I'm not sure if this should be considered a bug or a feature request:

The output from with_tz() seems un-trustworthy, because while SOME timezones work correctly, many others SILENTLY return incorrect results! When wrong, it appears that the time is actually still in UTC but is incorrectly labeled as being whatever timezone the user requested. If the timezone is unknown and thus will give wrong results, shouldn't an error or warning be thrown?

I get the same results with either lubridate_1.5.6.tar.gz or lubridate_1.6.0.tar.gz from CRAN, installed from source. Simple example:

$ grep DESCRIPTION /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
$ uname -a
Linux pinky 3.2.0-57-generic #87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ /usr/local/pkg/R-3.2-branch-20160718/bin/R --vanilla --no-restore --no-save

R version 3.2.5 Patched (2016-05-05 r70929) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu/x86_64 (64-bit)

> require("lubridate"); options("width"=100)
*** output flushed ***
> xx <- as.POSIXct(c(1471233699 ,1471234300 ,1471234901) ,origin="1970-01-01")
> yy <- ymd_hms(c("2016-08-15 04:01:39 UTC" ,"2016-08-15 04:11:40 UTC" ,"2016-08-15 04:21:41 UTC"))
> as.numeric(xx)
[1] 1471233699 1471234300 1471234901
> identical(as.numeric(xx) ,as.numeric(yy))
[1] TRUE

> with_tz(xx ,"UTC")  # Ok.
[1] "2016-08-15 04:01:39 UTC" "2016-08-15 04:11:40 UTC" "2016-08-15 04:21:41 UTC"
> with_tz(xx ,"EST")  # Ok.
[1] "2016-08-14 23:01:39 EST" "2016-08-14 23:11:40 EST" "2016-08-14 23:21:41 EST"
> with_tz(xx ,"EDT")  # Wrong!
[1] "2016-08-15 04:01:39 EDT" "2016-08-15 04:11:40 EDT" "2016-08-15 04:21:41 EDT"
> with_tz(xx ,"CST")  # Wrong!
[1] "2016-08-15 04:01:39 CST" "2016-08-15 04:11:40 CST" "2016-08-15 04:21:41 CST"
> with_tz(xx ,"CDT")  # Wrong!
[1] "2016-08-15 04:01:39 CDT" "2016-08-15 04:11:40 CDT" "2016-08-15 04:21:41 CDT"
> with_tz(xx ,"-1000")  # Wrong!
[1] "2016-08-15 04:01:39" "2016-08-15 04:11:40" "2016-08-15 04:21:41"
> with_tz(xx ,"+1000")  # Wrong!
[1] "2016-08-15 04:01:39" "2016-08-15 04:11:40" "2016-08-15 04:21:41"
> with_tz(xx ,"America/New_York")     # Ok.
[1] "2016-08-15 00:01:39 EDT" "2016-08-15 00:11:40 EDT" "2016-08-15 00:21:41 EDT"
> with_tz(xx ,"America/Chicago")      # Ok.
[1] "2016-08-14 23:01:39 CDT" "2016-08-14 23:11:40 CDT" "2016-08-14 23:21:41 CDT"
> with_tz(xx ,"America/Los_Angeles")  # Ok.
[1] "2016-08-14 21:01:39 PDT" "2016-08-14 21:11:40 PDT" "2016-08-14 21:21:41 PDT"

> zz <- c("UTC" ,"EST" ,"EDT" ,"CST" ,"CDT" ,"America/New_York" ,"America/Chicago" ,"America/Los_Angeles")
> rbind(zz ,(zz %in% OlsonNames()))
   [,1]   [,2]   [,3]    [,4]    [,5]    [,6]               [,7]              [,8]
zz "UTC"  "EST"  "EDT"   "CST"   "CDT"   "America/New_York" "America/Chicago" "America/Los_Angeles"
   "TRUE" "TRUE" "FALSE" "FALSE" "FALSE" "TRUE"             "TRUE"            "TRUE"

[vspinu]

You provided an answer to your cuestion. EDT, CST, CDT are not valid time zones on your system, which, for you, match with Olson's data base:

> zz <- c("UTC" ,"EST" ,"EDT" ,"CST" ,"CDT" ,"America/New_York" ,"America/Chicago" ,"America/Los_Angeles")
> cbind(zz ,(zz %in% OlsonNames()))
     zz                           
[1,] "UTC"                 "TRUE" 
[2,] "EST"                 "TRUE" 
[3,] "EDT"                 "FALSE"
[4,] "CST"                 "FALSE"
[5,] "CDT"                 "FALSE"
[6,] "America/New_York"    "TRUE" 
[7,] "America/Chicago"     "TRUE" 
[8,] "America/Los_Angeles" "TRUE" 

You can do with_tz(.., tzone="blabla") and it will still work.

I wouldn't know a good solution for this as it's OS dependent. Valid timezones on some OSes need not be in Olson names. Even those in OlsonNames databases might not be valid. It's quite a mess. See R's docs for OlsonNames.

One solution would be to introduce a warning if tz is not in OlsonNames with the possibility to suppress it with an option.

with_tz(xx ,"-1000") # Wrong!
[ 1] "2016-08-15 04:01:39" "2016-08-15 04:11:40" "2016-08-15 04:21:41"
with_tz(xx ,"+1000") # Wrong!

+1000 should be specified as GMT-1 (negative for est).

[vspinu]

I made some progress on this for force_tz only, an error is now thrown if tz is not recognized. withi_tz is still in the air.