New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: date_range not working as it intuitively should when specifying start, end, and periods #20808

Closed
onnoeberhard opened this Issue Apr 24, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@onnoeberhard
Contributor

onnoeberhard commented Apr 24, 2018

Code Sample, a copy-pastable example if possible

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.date_range(start, end, periods=1000)    # Intuitively a linearly spaced time series
Traceback (most recent call last):

  File "<ipython-input-69-2304a28824c6>", line 1, in <module>
    pd.date_range(start, end, periods=1000)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 2057, in date_range
    closed=closed, **kwargs)

  File "E:\Anaconda3\lib\site-packages\pandas\util\_decorators.py", line 118, in wrapper
    return func(*args, **kwargs)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 324, in __new__
    ambiguous=ambiguous)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 421, in _generate
    raise ValueError('Of the three parameters: start, end, and '

ValueError: Of the three parameters: start, end, and periods, exactly two must be specified

Problem description

I need a DatetimeIndex object to later use as index in a Series. DatetimeIndex should start at start, end at end and have a fixed number of elements (1000). Intuitively, this should work with pd.date_range, but it doesn't, and I haven't found a good explanation about why this is the case.
I have found a workaround on Stackoverflow (https://stackoverflow.com/questions/25796030/how-can-i-use-pandas-date-range-to-obtain-a-time-series-with-n-specified-perio) that does work:

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.to_datetime(np.linspace(start.value, end.value, 1000))
DatetimeIndex(['2008-01-02 07:51:37.999476992',
               '2008-01-02 07:51:45.206684160',
               '2008-01-02 07:51:52.413891328',
               '2008-01-02 07:51:59.621098496',
               '2008-01-02 07:52:06.828305920',
               '2008-01-02 07:52:14.035513088',
               '2008-01-02 07:52:21.242720256',
               '2008-01-02 07:52:28.449927424',
               '2008-01-02 07:52:35.657134592',
               '2008-01-02 07:52:42.864341760',
               ...
               '2008-01-02 09:50:33.134612224',
               '2008-01-02 09:50:40.341819392',
               '2008-01-02 09:50:47.549026560',
               '2008-01-02 09:50:54.756233728',
               '2008-01-02 09:51:01.963440896',
               '2008-01-02 09:51:09.170648064',
               '2008-01-02 09:51:16.377855488',
               '2008-01-02 09:51:23.585062656',
               '2008-01-02 09:51:30.792269824',
               '2008-01-02 09:51:37.999476992'],
              dtype='datetime64[ns]', length=1000, freq=None)

Expected Output

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.date_range(start, end, periods=1000)
DatetimeIndex(['2008-01-02 07:51:37.999476992',
               '2008-01-02 07:51:45.206684160',
               '2008-01-02 07:51:52.413891328',
               '2008-01-02 07:51:59.621098496',
               '2008-01-02 07:52:06.828305920',
               '2008-01-02 07:52:14.035513088',
               '2008-01-02 07:52:21.242720256',
               '2008-01-02 07:52:28.449927424',
               '2008-01-02 07:52:35.657134592',
               '2008-01-02 07:52:42.864341760',
               ...
               '2008-01-02 09:50:33.134612224',
               '2008-01-02 09:50:40.341819392',
               '2008-01-02 09:50:47.549026560',
               '2008-01-02 09:50:54.756233728',
               '2008-01-02 09:51:01.963440896',
               '2008-01-02 09:51:09.170648064',
               '2008-01-02 09:51:16.377855488',
               '2008-01-02 09:51:23.585062656',
               '2008-01-02 09:51:30.792269824',
               '2008-01-02 09:51:37.999476992'],
              dtype='datetime64[ns]', length=1000, freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 44 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.22.0
pytest: 3.3.2
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.2
scipy: 1.0.1
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@jreback

This comment has been minimized.

Contributor

jreback commented Apr 24, 2018

I suppose if freq is NOT specified then you could accept all three and give a linspace repr. What breaks if you do that?

If freq is specified then this should for sure raise. The idea of the error is simply to be helpful in telling you that you may have overspecified the args.

@chris-b1

This comment has been minimized.

Contributor

chris-b1 commented Apr 24, 2018

Your workaround from SO is a pretty reasonable solution, not sure we should support this. I definitely would not want to change the default behavior, but suppose could with with something like freq=None or freq='interpolate'

@onnoeberhard

This comment has been minimized.

Contributor

onnoeberhard commented Apr 26, 2018

I agree to definetely don't change the default behaviour. As I see, the default freq is 'D', so this:

pd.date_range(start, end, periods=1000)

would not work, because it is the same as

pd.date_range(start, end, periods=1000, freq='D')

which really should not work. However, if the user explicitly sets freq=None, the linspace behaviour would be practical:

pd.date_range(start, end, periods=1000, freq=None)

I suggest a simple change in the date_range function like so:

def date_range(start=None, end=None, periods=None, freq='D', tz=None,
               normalize=False, name=None, closed=None, **kwargs):
    """
    ...
    """
    # Return a linearly spaced DatetimeIndex if `freq` is not set, but `start`, `end`, and `periods` are
    if start and end and periods and not freq:
        di = tools.to_datetime(np.linspace(start, end, periods), **kwargs)
        if name:
            di.name = name
        return di

    return DatetimeIndex(start=start, end=end, periods=periods,
                         freq=freq, tz=tz, normalize=normalize, name=name,
                         closed=closed, **kwargs)

together with appropriate changes in the docstring and test functions. This would provide the desired behaviour, while not changing anything else.

I am not yet a contributor, so I cannot implement this myself (unless I am made one 🙃)

Because the change is really just a convenience feature of the date_range function, I don't think it would be wise to implement this directly in the DatetimeIndex constructor, or in another function, like bdate_range.

@TomAugspurger

This comment has been minimized.

Contributor

TomAugspurger commented Apr 27, 2018

FWI, we could change the default freq to None, and document that it's 'D' when only two of star, end, and freq are specified. That way pd.date_range(start, end, periods=100) will work.

I am not yet a contributor, so I cannot implement this myself (unless I am made one 🙃)

Anyone can make a pull request: http://pandas-docs.github.io/pandas-docs-travis/contributing.html

@onnoeberhard

This comment has been minimized.

Contributor

onnoeberhard commented Apr 27, 2018

That's a good idea, I like it much better.

Anyone can make a pull request

Oh okay I didn't know that (never done this before), then I am going to try to implement it :)

@jreback jreback added this to the Next Major Release milestone Apr 30, 2018

@jreback jreback changed the title from `date_range` not working as it intuitively should when specifying start, end, and periods to ENH: date_range not working as it intuitively should when specifying start, end, and periods Apr 30, 2018

@jreback jreback modified the milestones: Next Major Release, 0.23.0 May 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment