WeibullAFTFitter does not ignore the index of the DataFrame and errors (NaNs were detected in the dataset.)

[joshlk]

Show by example:

from lifelines import WeibullAFTFitter
import pandas as pd

data = pd.DataFrame({'duration': np.arange(1, 11), 'event': np.ones(10), 'x0': np.arange(1, 11)})

regr = WeibullAFTFitter()
regr.fit(data, duration_col='duration', event_col='event')

That works fine. But if I take a row subset of the data so the DataFrame index isn't arange:

sub_data = data[[True, False]*5]
regr = WeibullAFTFitter()
regr.fit(sub_data, duration_col='duration', event_col='event')

It produces the following error even though there is no inf or nan values:

TypeError: NaNs were detected in the dataset. Try using pd.isnull to find the problematic values.

It seems like the function is not ignoring the DataFrame index.

Versions:

• Lifelines: 0.25.8
• Pandas: 1.1.5

[CamDavidsonPilon]

Hi @joshlk, yup looks like a bug. I think the source is formulaic however. Ex:

from formulaic import Formula

design_info = Formula("1")

df = pd.DataFrame(np.arange(5), index=[0, 2, 4, 6, 8])

print(design_info.get_model_matrix(df))

I'll make an issue with them to resolve this. For now, you can either downgrade a version of lifelines, say 0.25.7, (we recently swapped to Formulaic), or do a .reset_index(drop=True) prior to fitting:

sub_data = data[[True, False]*5].reset_index(drop=True)
regr = WeibullAFTFitter()
regr.fit(sub_data, duration_col='duration', event_col='event')

[CamDavidsonPilon]

👋 Formulaic just released 2.2 - try an upgrade locally pip install formulaic==0.2.2 to fix this.