DatetimeIndex construction with periods and closed

[Adirio]

Code Sample, a copy-pastable example if possible

import pandas as pd

dti1 = pd.DatetimeIndex(start="2018-10-16", periods=12, freq="H", closed="right")
print(dti1)

dti2 = pd.DatetimeIndex(end="2018-10-16", periods=12, freq="H", closed="left")
print(dti2)

Outputs:

DatetimeIndex(['2018-10-16 01:00:00', '2018-10-16 02:00:00',
               '2018-10-16 03:00:00', '2018-10-16 04:00:00',
               '2018-10-16 05:00:00', '2018-10-16 06:00:00',
               '2018-10-16 07:00:00', '2018-10-16 08:00:00',
               '2018-10-16 09:00:00', '2018-10-16 10:00:00',
               '2018-10-16 11:00:00'],
              dtype='datetime64[ns]', freq='H')
DatetimeIndex(['2018-10-15 13:00:00', '2018-10-15 14:00:00',
               '2018-10-15 15:00:00', '2018-10-15 16:00:00',
               '2018-10-15 17:00:00', '2018-10-15 18:00:00',
               '2018-10-15 19:00:00', '2018-10-15 20:00:00',
               '2018-10-15 21:00:00', '2018-10-15 22:00:00',
               '2018-10-15 23:00:00'],
              dtype='datetime64[ns]', freq='H')

Problem description

When explicitly asking for a number of periods, an index of that size is expected. By using closed with the matching boundary specified (start for closed="right" and end for closed="left") we get one less than the requested periods.

Probably this also happens with other functions/methods that use the closed argument and specify the number of items desired.

Expected Output

DatetimeIndex(['2018-10-16 01:00:00', '2018-10-16 02:00:00',
               '2018-10-16 03:00:00', '2018-10-16 04:00:00',
               '2018-10-16 05:00:00', '2018-10-16 06:00:00',
               '2018-10-16 07:00:00', '2018-10-16 08:00:00',
               '2018-10-16 09:00:00', '2018-10-16 10:00:00',
               '2018-10-16 11:00:00', '2018-10-16 12:00:00'],
              dtype='datetime64[ns]', freq='H')
DatetimeIndex(['2018-10-15 12:00:00', '2018-10-15 13:00:00',
               '2018-10-15 14:00:00', '2018-10-15 15:00:00',
               '2018-10-15 16:00:00', '2018-10-15 17:00:00',
               '2018-10-15 18:00:00', '2018-10-15 19:00:00',
               '2018-10-15 20:00:00', '2018-10-15 21:00:00',
               '2018-10-15 22:00:00', '2018-10-15 23:00:00'],
              dtype='datetime64[ns]', freq='H')

Output of pd.show_versions() (short version)

python: 3.6.4.final.0

pandas: 0.23.4

[mroeschke]

I'd argue that this is the correct behavior, albeit not documented clearly:

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.date_range.html

The order of operations is essentially: Create the dates from start, end, periods, and/or freq and then use the closed argument to truncate the created dates if None isn't specified.

# start would be '2018-10-16 00:00:00',
# end would be '2018-10-16 11:00:00'
In [13]: pd.DatetimeIndex(start="2018-10-16", periods=12, freq="H", closed=None)
Out[13]:
DatetimeIndex(['2018-10-16 00:00:00', '2018-10-16 01:00:00',
               '2018-10-16 02:00:00', '2018-10-16 03:00:00',
               '2018-10-16 04:00:00', '2018-10-16 05:00:00',
               '2018-10-16 06:00:00', '2018-10-16 07:00:00',
               '2018-10-16 08:00:00', '2018-10-16 09:00:00',
               '2018-10-16 10:00:00', '2018-10-16 11:00:00'],
              dtype='datetime64[ns]', freq='H')

# This looks like a bug, '2018-10-16 11:00:00' shouldn't be included
In [14]: pd.DatetimeIndex(start="2018-10-16", periods=12, freq="H", closed='left')
Out[14]:
DatetimeIndex(['2018-10-16 00:00:00', '2018-10-16 01:00:00',
               '2018-10-16 02:00:00', '2018-10-16 03:00:00',
               '2018-10-16 04:00:00', '2018-10-16 05:00:00',
               '2018-10-16 06:00:00', '2018-10-16 07:00:00',
               '2018-10-16 08:00:00', '2018-10-16 09:00:00',
               '2018-10-16 10:00:00', '2018-10-16 11:00:00'],
              dtype='datetime64[ns]', freq='H')

In [15]: pd.DatetimeIndex(start="2018-10-16", periods=12, freq="H", closed='right')
Out[15]:
DatetimeIndex(['2018-10-16 01:00:00', '2018-10-16 02:00:00',
               '2018-10-16 03:00:00', '2018-10-16 04:00:00',
               '2018-10-16 05:00:00', '2018-10-16 06:00:00',
               '2018-10-16 07:00:00', '2018-10-16 08:00:00',
               '2018-10-16 09:00:00', '2018-10-16 10:00:00',
               '2018-10-16 11:00:00'],
              dtype='datetime64[ns]', freq='H')

[Adirio]

I do not agree. If you specify periods instead of end is because you want that size. The same should apply if you specify end and periods. You are considering that by specifying periods and only one of the boundaries the user wants to give you the two boundaries (that's what you are doing by saying it should end at '2018-10-16 11:00:00' while that is not true). The user just specified the start and the periods, why would you return one less period that specified?

The easy way to do this is always computing one period more than needed and then truncating to the number of periods specified. Only one is needed as it can not be opened on both sides (this bug happens when one of the boundaries is not given, so there is nothing to open or close in that side).

EDIT: I edited the Issue to include the end and closed="left" mirror bug.

[Adirio]

I found a check in the code that is suposed to prevent using closed with one of the boundaries not defined:

raise ValueError("Closed has to be None if not both of start and end are defined")

In the master branch this check is moved to here, but is still the same.

The problem with this error is that it never gets raised because the check is bugged, it checks for both to be None (and) instead of any of them (or).

Now the issue changes. The question now is, why would you disallow to use closed if one of the boundaries is not defined? I will close this Issue and start a new one proposing to change this behaviour after I make a PR solving the and/or bug mentioned.