df.interpolate limit_direction= does not work without limit=x setting #16282

Closed
WBare opened this Issue May 8, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@WBare
Contributor

WBare commented May 8, 2017

the interpolate limit_direction setting should work even without a limit=x setting.

When interpolate is called without a limit=x setting, it always (apparently) uses the limit_direction setting default of 'forward' even when it is explicitly set to 'backward' or 'both'

The code below will show 3 DataFrames.

  • The first shows the initial before interpolation.
  • The second shows the interpolation done correctly in the backward direction.
  • The third shows the dataframe done incorrectly in the 'forward' direction even though the limit_direction='backward' was used.

The docs do not mention a linkage between the two settings, but if this is expected behavior (I hope not), it would be oh so nice to raise an exception like the existing error for a bad limit_direction setting.

Many thanks for all your work!

Example Code:

import pandas as pd
import numpy as np

dfMain = pd.DataFrame({
    'a': [0, 1, np.NAN, 3, 4],
    'b': [np.NaN, 1, np.NaN, 3, 4],
    'c': [0 , 1, 2, 3, np.NaN]})

print("initial df")
print(dfMain)

print("backwards with limit=1 (correct result):")
print(dfMain.interpolate(limit_direction='backward', limit=1))

print("backwards no longer works when limit is removed (it is going forward):")
print(dfMain.interpolate(limit_direction='backward'))

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.0.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-75-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 34.4.1
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
boto: 2.45.0
pandas_datareader: 0.2.1
None
initial df

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger May 8, 2017

Contributor

Thanks for the report, want to take a look at where it's going wrong? Maybe somewhere around

def interpolate(self, method='pad', axis=0, index=None, values=None,

Contributor

TomAugspurger commented May 8, 2017

Thanks for the report, want to take a look at where it's going wrong? Maybe somewhere around

def interpolate(self, method='pad', axis=0, index=None, values=None,

@TomAugspurger TomAugspurger added this to the 0.20.2 milestone May 8, 2017

@WBare

This comment has been minimized.

Show comment
Hide comment
@WBare

WBare May 9, 2017

Contributor

I've got a fix for this and the associated enhancement for limit_direction='inside' GH16284.

The issue was inside missing.py interpolate_1d

I have tested myself, and I'm running the test suite now, but since I'm not familiar with the code base, someone really should carefully review the code. It is actually fairly simple, but I don't know if there are other dependencies.

Contributor

WBare commented May 9, 2017

I've got a fix for this and the associated enhancement for limit_direction='inside' GH16284.

The issue was inside missing.py interpolate_1d

I have tested myself, and I'm running the test suite now, but since I'm not familiar with the code base, someone really should carefully review the code. It is actually fairly simple, but I don't know if there are other dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment