Inconsistent Datetime microsecond output in Rcpp

[driftwoods]

I would like to generate Datetime objects with microsecond precision using Rcpp. However I got inconsistent Datetime outputs (in the microsecond part) in R, even though the printout within Rcpp code shows accurate result. To demonstrate this issue, let's save the following Rcpp code into a file called ./test_Rcpp_datetime.cpp.

#include <ctime>
#include <chrono>
#include <Rcpp.h>
using namespace Rcpp;
using namespace std;

// [[Rcpp::export]]
Datetime RcppConvertMicrosecsToDatetime(int idate,
                                        int year,
                                        int month,
                                        int day,
                                        int hour,
                                        int min,
                                        int secs,
                                        int usecs) {
  tm time;
  time.tm_year = year - 1900;
  time.tm_mon = month - 1;
  time.tm_mday = day;
  time.tm_hour = hour;
  time.tm_min = min;
  time.tm_sec = secs;
  time.tm_isdst = 0; // this is important
  
  time_t ttime = mktime(&time);
  chrono::system_clock::time_point tp = chrono::system_clock::from_time_t(ttime);
  tp += chrono::microseconds(usecs);
  auto duration = tp.time_since_epoch(); 
  auto micros = chrono::duration_cast<chrono::microseconds>(duration).count();
  double seconds = micros / 1.0e6;
  Rcpp::Rcout << "micro seconds since epoch = " << micros << endl;
  Rcpp::Rcout << std::setprecision(18) << "seconds since epoch = " << seconds << endl;
  Datetime dt = seconds;
  Rcpp::Rcout << "Datetime.getMicroSeconds() = " << dt.getMicroSeconds() << endl;
  
  return dt; 
}

/*** R
options(digits.secs=6)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834198)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834199)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834200)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834201)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834202)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834203)
RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834204)
*/

In R condole, we can do Rcpp::sourceCpp('./test_Rcpp_datetime.cpp'), below is the output of running. Rcpp internal microseconds values look good to me (with proper round of decimals). But the microsecond value in R is weird, it doesn't do round properly and neither does floor properly. How can I generate Datetime object correctly in Rcpp? Thanks.

> options(digits.secs=6)

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834198)
micro seconds since epoch = 1515879001834198
seconds since epoch = 1515879001.834198
Datetime.getMicroSeconds() = 834198
[1] "2018-01-13 15:30:01.834197 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834199)
micro seconds since epoch = 1515879001834199
seconds since epoch = 1515879001.83419895
Datetime.getMicroSeconds() = 834199
[1] "2018-01-13 15:30:01.834198 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834200)
micro seconds since epoch = 1515879001834200
seconds since epoch = 1515879001.83419991
Datetime.getMicroSeconds() = 834200
[1] "2018-01-13 15:30:01.8341 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834201)
micro seconds since epoch = 1515879001834201
seconds since epoch = 1515879001.8342011
Datetime.getMicroSeconds() = 834201
[1] "2018-01-13 15:30:01.834201 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834202)
micro seconds since epoch = 1515879001834202
seconds since epoch = 1515879001.83420205
Datetime.getMicroSeconds() = 834202
[1] "2018-01-13 15:30:01.834202 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834203)
micro seconds since epoch = 1515879001834203
seconds since epoch = 1515879001.834203
Datetime.getMicroSeconds() = 834203
[1] "2018-01-13 15:30:01.834203 CST"

> RcppConvertMicrosecsToDatetime(idate=20180113, year=2018, month=1, day=13, hour=15, min=30, secs=1, usecs=834204)
micro seconds since epoch = 1515879001834204
seconds since epoch = 1515879001.83420396
Datetime.getMicroSeconds() = 834204
[1] "2018-01-13 15:30:01.834203 CST"

[eddelbuettel]

Without looking too too closely I think the answer may be "you don't".

R's Datetime (ie POSIXct) is implemented as fractional seconds since the epoch, using 53 bits of the 64 bits of the double type. The 'almost microsecond' precision is well documented at the R side using the same type. In essence, I think you are doing a lossy conversion here. So you may want to stick with std::chrono if you need the precision.

One alternative is to remain with higher resolution. Our nanotime package on CRAN uses bit64::integer to represent nanosecond increments. I use that at work too where we use similar int64_t based nano-second time. (And there are examples at the Rcpp Gallery site showing conversions...)

[eddelbuettel]

You can also look at my RcppCCTZ package for another approach -- also using higher-precision representation at the C++ level. But once you get to POSIXct the same issue re-appears. That is an R constraint.

[driftwoods]

Thanks for your explanation, Dirk. It's kind of a disappointment to me that R doesn't use 64 bits in POSIXct objects. This is just another bummer after I learned that R doesn't have a 64 bits integer type.

[eddelbuettel]

The int64 issue bites a little, but I am having good luck with the nanotime package we wrote, sitting on top of the bit64 package and its integer64 implementation. Well supported inside data.table, which is my favourite container.

That R only has 32-bit and 64-bit floats is a reflection of when it was created, and what it's focus is. With the design it is hard to add to new atomistic types at the core. So we make do with user-added type. And where it really matter -- in C++ code we call -- we have the proper types we need.