ENH: add origin to to_datetime

[jreback]

closes #11276
closes #11745

superseded #11470

[jreback]

@jorisvandenbossche @chris-b1

ok, rebase and updated original PR. pls have a look.

IIRC there were some more test cases needed, but dont' really remember

[chris-b1]

'epoch' seems non-descriptive - in a sense, aren't all origins epochs? Maybe default should be 'unix' like @shoyer had suggested?

[jreback]

Unix time (also known as POSIX time or epoch time) ?

https://en.wikipedia.org/wiki/Unix_time

[chris-b1]

Certainly the unix epoch date is implied by "epoch time" but it's also a general term, that could refer to epoch timekeeping in general.

https://en.wikipedia.org/wiki/Epoch_(reference_date)#Computing

[chris-b1]

Not critical, but could expand to other semi-common origins @bashtage mentions here.
#11470 (comment)

[jreback]

hmm, maybe, can always add if people really want this.

[codecov]

Codecov Report

Merging #15828 into master will decrease coverage by 0.03%.
The diff coverage is 91.11%.

@@            Coverage Diff             @@
##           master   #15828      +/-   ##
==========================================
- Coverage   90.98%   90.95%   -0.04%     
==========================================
  Files         143      143              
  Lines       49449    49464      +15     
==========================================
- Hits        44993    44991       -2     
- Misses       4456     4473      +17

Flag	Coverage Δ
#multiple	?
#single	?

Impacted Files	Coverage Δ
pandas/tseries/tdi.py	90.37% <100%> (+0.03%)	⬆️
pandas/tseries/tools.py	85.2% <90.24%> (+0.26%)	⬆️
pandas/io/gbq.py	25% <0%> (-58.34%)	⬇️
pandas/io/packers.py	87.57% <0%> (-0.94%)	⬇️
pandas/core/common.py	90.96% <0%> (-0.34%)	⬇️
pandas/io/html.py	84.48% <0%> (-0.33%)	⬇️
pandas/core/series.py	94.86% <0%> (-0.11%)	⬇️
pandas/core/frame.py	97.56% <0%> (-0.1%)	⬇️
pandas/compat/numpy/function.py	94.19% <0%> (-0.04%)	⬇️
pandas/core/indexing.py	94.08% <0%> (-0.03%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cc021...ebb4acd. Read the comment docs.

[jreback]

ok changed epoch -> unix and updated docs.

[jorisvandenbossche]

The problem is that this does not work ... type explanation should be on one line (for good html docs) (ref numpy/numpydoc#87)

[jreback]

so pretty much make this shorter then?

[jorisvandenbossche]

blank line between the two lines above

[jorisvandenbossche]

We should add a todo item to solve this

[jreback]

what part, OutOfBounds dates generally?

[jorisvandenbossche]

No, I mean for origin specifically (by circumventing the Timestamp creation), to have something like pd.to_datetime(.., origin='0001-01-01') working if the end timestamp is not out of bounds.

Eg by using dateutil.parser.parse(origin), but then we still need to convert that to the correct offset

[jorisvandenbossche]

As I commented in the original PR (#11470 (comment)), I think it can solve some corner cases to do the origin handling before the actual parsing (as you do for the 'julian' case).

A bit an artificial example, but to show one of the problems:

In [11]: pd.to_datetime(200*365, unit='D')
Out[11]: Timestamp('2169-11-13 00:00:00')

In [12]: pd.to_datetime(200*365, unit='D', origin='1870-01-01')
Out[12]: Timestamp('2069-11-13 00:00:00')

In [13]: pd.to_datetime(300*365, unit='D', origin='1870-01-01')
...
OutOfBoundsDatetime: cannot convert input with unit 'D'

So the last one is actually not an OutOfBounds datetime, but just because we first parse the number as epoch, it raises this error. If we first subtract the correct value from the numeric argument, and then parse as epoch, the above will work fine.

[jreback]

fixed! this was tricky.

[jorisvandenbossche]

Some cases where something goes wrong:

• This should raise an error (if the origin is first processed and the argument adapted for it, then this will raise an error):

  In [2]: pd.to_datetime("2005-01-01", origin="1960-01-01")
  Out[2]: Timestamp('1995-01-01 00:00:00')
  

• Not sure how this can return 12:00 as the hour? (EDIT -> this is probably just the definition? "starting at noon")

  In [4]: pd.to_datetime(2456658, origin='julian', unit='D')
  Out[4]: Timestamp('2013-12-31 12:00:00')
  

• The error message in case of out of bounds here:

  In [3]: pd.to_datetime(1, origin="julian", unit='D')
  ....
  OutOfBoundsDatetime: cannot convert input -2440586.5 with the unit 'D'
  

  is a bit unfortunate (the "input -2440586.5"). But that is of course the consequence of processing the origin first and adapting the arg based on that, before passing to the actual parsing methods ... (so that's a clear drawback of this approach I suggested also for the other origins). Not sure whether this could easily be solved ..

[jorisvandenbossche]

let's use 'reference date' instead of 'offset' here as well (as you did in the docstring)

[jreback]

Here are the results with new push.

In [1]: pd.to_datetime(200*365, unit='D')
Out[1]: Timestamp('2169-11-13 00:00:00')

In [2]: pd.to_datetime(200*365, unit='D', origin='1870-01-01')
Out[2]: Timestamp('2069-11-13 00:00:00')

In [3]: pd.to_datetime(300*365, unit='D', origin='1870-01-01')
Out[3]: Timestamp('2169-10-20 00:00:00')

In [4]: pd.to_datetime("2005-01-01", origin="1960-01-01")
ValueError: '2005-01-01' is not compatible with origin='1960-01-01'; it must be numeric with a unit specified 

In [5]: pd.to_datetime(2456658, origin='julian', unit='D')
Out[5]: Timestamp('2013-12-31 12:00:00')

In [6]: pd.to_datetime(1, origin="julian", unit='D')
OutOfBoundsDatetime: 1 is Out of Bounds for origin='julian'

note [1], [2] (were correct before)
[5] seems fine as well.

[jorisvandenbossche]

I would leave this sentence in, it is a good explanation of what an origin is.

[jorisvandenbossche]

New changes look good!

[jorisvandenbossche]

You can have the same problem with a custom defined origin (not unix of julian)

[jreback]

yes, but that is handled below (this is only for the julian section). I do raise the same error (w/o the julian reference)

[jreback]

    elif origin not in ['unix', 'julian']:

        # arg must be a numeric
        original = arg
        if not ((is_scalar(arg) and (is_integer(arg) or is_float(arg))) or
                is_numeric_dtype(np.asarray(arg))):
            raise ValueError(
                "'{arg}' is not compatible with origin='{origin}'; "
                "it must be numeric with a unit specified ".format(
                    arg=arg,
                    origin=origin))

        # we are going to offset back to unix / epoch time
        try:
            offset = tslib.Timestamp(origin) - tslib.Timestamp(0)
        except tslib.OutOfBoundsDatetime:
            raise tslib.OutOfBoundsDatetime(
                "origin {} is Out of Bounds".format(origin))
        except ValueError:
            raise ValueError("origin {} cannot be converted "
                             "to a Timestamp".format(origin))

[jorisvandenbossche]

Wouldn't it be simpler to check if it is a list and then convert to array? (or check that is is not a scalar/series/index)

Then the arg = arg + offset does just what you want, and the conversion back as below is not needed

[jreback]

@jorisvandenbossche make those changes and merged. thanks for the review.

[jorisvandenbossche]

And thanks for picking up the PR!