Predict fails for data frame

[matthew-brett]

For the following script:

import pandas as pd
import statsmodels.formula.api as smf

df = pd.DataFrame([[3, 0.030], [10, 0.060], [20, 0.120]], columns=['BSA', 'Absorbance'])

model = smf.ols('Absorbance ~ BSA', data=df)
fit = model.fit()
fit.predict([0.25])

I get an error ending

PatsyError: Error evaluating factor: NameError: name 'BSA' is not defined
    Absorbance ~ BSA
                 ^^^

Is this expected?

In [3]: statsmodels.__version__
Out[3]: '0.8.0.dev0+ad70c62'

[josef-pkt]

Yes, that is expected.

The default in predict when using formulas is to go through patsy. patsy needs to be able to identify which variable the data corresponds to. So the exog needs to be a DataFrame or something dict like
e.g.

>>> fit.predict({'BSA': [0.25]})
0    0.012568
dtype: float64

otherwise, it is possible to provide the full predict design matrix and avoid patsy with transform=False (no categorical or similar transformations are done in this case)

>>> fit.predict([1, 0.25], transform=False)
array([ 0.01256849])

There were some problems in the last year after we changed to returning a pandas Series or DataFrame where we missed some possible cases of input types. (I'm using current master, but we still have one open PR to allow other possible argument types for exog in predict.)

[josef-pkt]

However, it looks like patsy doesn't like Series

>>> fit.predict(df.mean(0))
raises an exception in patsy

>>> fit.predict(df.mean(0).to_frame().T)
0    0.07
dtype: float64

>>> fit.predict([1, 11.], transform=False)
array([ 0.07])

and this looks like a bug coming from the new indexing of the predicted return
(we predict correctly but have the wrong index, I guess)

>>> fit.predict(pd.Series([1, 11.]), transform=False)
0    0.07
1    0.07
dtype: float64

[josef-pkt]

(Life is much simpler in a pure numpy world)

[matthew-brett]

It is possible to give a more obvious error message?

[josef-pkt]

I'm not sure or I doubt it's practical.

"NameError: name 'BSA' is not defined" sounds pretty descriptive to me.
The error is raised by patsy, so we don't have direct control over the exception message, short of catching the error and rewriting exception messages.

[matthew-brett]

Well I can say that the error message was confusing enough for me that I did a bug report...

I had the implicit hope that, in this case, the machinery could guess what I meant, but if not, it would be nice to know why (as in something like "we need to know what RHS values you passed in order to do the prediction". Can you not check that the input is a data frame etc?

[hellsanjeev]

I have a similar issue, though I have used a dataframe -

import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt

calo=pd.read_csv("/Users/Sanjeev/Desktop/Excel R Assignments/Simple Linear Regression/calories_consumed.csv")

import statsmodels.formula.api as smf
model = smf.ols("wt~cal",data=calo).fit()

pred = model.predict(pd.DataFrame(calo['wt']))

The Error is -

pred = model.predict(pd.DataFrame(calo['wt']))
Traceback (most recent call last):

  File "<ipython-input-53-4fcbf1ee1921>", line 1, in <module>
    pred = model.predict(pd.DataFrame(calo['wt']))

  File "/anaconda3/lib/python3.6/site-packages/statsmodels/base/model.py", line 837, in predict
    exog = dmatrix(design_info, exog, return_type="dataframe")

  File "/anaconda3/lib/python3.6/site-packages/patsy/highlevel.py", line 291, in dmatrix
    NA_action, return_type)

  File "/anaconda3/lib/python3.6/site-packages/patsy/highlevel.py", line 169, in _do_highlevel_design
    return_type=return_type)

  File "/anaconda3/lib/python3.6/site-packages/patsy/build.py", line 888, in build_design_matrices
    value, is_NA = _eval_factor(factor_info, data, NA_action)

  File "/anaconda3/lib/python3.6/site-packages/patsy/build.py", line 63, in _eval_factor
    result = factor.eval(factor_info.state, data)

  File "/anaconda3/lib/python3.6/site-packages/patsy/eval.py", line 566, in eval
    data)

  File "/anaconda3/lib/python3.6/site-packages/patsy/eval.py", line 551, in _eval
    inner_namespace=inner_namespace)

  File "/anaconda3/lib/python3.6/site-packages/patsy/compat.py", line 43, in call_and_wrap_exc
    exec("raise new_exc from e")

  File "<string>", line 1, in <module>

PatsyError: Error evaluating factor: NameError: name 'cal' is not defined
    wt~cal
       ^^^`

Would be really thankful if u can help.

[josef-pkt]

@hellsanjeev You need to have columns in the dataframe with the same names as those used in the formula, in this case there is no "cal" column in the DataFrame used for prediction.

patsy uses the column names to identify which data corresponds to the terms in the formula both for creating the model and for prediction.