Provide Initial Guess to Regression Fitters

[bacalfa]

It would be great to be able to provide an initial guess point (warm-start) to the regression fitters, such as WeibullAFTFitter. I'm referring to this line:

lifelines/lifelines/fitters/__init__.py

Line 1018 in d9d3f9f

init_values = np.zeros((n_params,))

I've been comparing this particular fitter to R's survreg, and for some datasets, their solutions don't agree at all. I'd like to provide the same initial values to both codes and hopefully get the same solution.

[CamDavidsonPilon]

Yikes, I'm not pleased that surveg and lifelines give different results. Are you able to share an example dataset here (or privately over email)?

It's easy enough to let users provide initial values, I can add that to the next release shortly.

[bacalfa]

Thanks for the quick response! Unfortunately, I can't share the data publicly nor privately. But I'll be happy to report the status after I'm able to use initial points. :) By the way, I had convergence issues with survreg before if I didn't provide any initial point in some cases. So achieving convergence can be sensitive to initial points.

[CamDavidsonPilon]

Can you describe your dataset more?

1. What are the dimensions of it?

Some information about the fit, too:

1. Does lifelines provide a smaller log-likelihood than R survreg? (you can see the log-likelihood using .print_summary() in lifelines)
2. Are any warnings displayed when .fit is called?

[CamDavidsonPilon]

flexsurvreg does some smarter initializations using summary statistics. From their docs:

If not specified, default initial values are chosen from a simple summary of the
survival or censoring times, for example the mean is often used to initialize scale
parameters. See the object flexsurv.dists for the exact methods used. If the
likelihood surface may be uneven, it is advised to run the optimisation starting
from various different initial values to ensure convergence to the true global
maximum.

[bacalfa]

Thanks! There were some problems with the code I was using. Will report after the experiments are finished. But it's an intercept-only model (no covariates) with datasets that range from fewer than 10 to nearly 100 points (most are suspensions).

[CamDavidsonPilon]

What about using the simpler WeibullFitter, which naturally takes no covariates? It will probably be faster too.

[bacalfa]

That could be done. It's just that I wanted to compare the AFT fitter with survreg. :) I have other tests with covariates as well.

[CamDavidsonPilon]

@bacalfa, if you update to 0.20.0 (on PyPI now), please try out the new defaults (i.e don't provide initial_point) to compare against R.

(0.20.0 is python3 only, and has some updated dependencies too FYI)

[bacalfa]

I'm using Anaconda, and the latest version seems to be 0.19.5.

On a related note, survreg offers the option to fix the scale parameter (e.g., scale=1) and to pass parameters to the optimizer via survreg.control. Those are interesting features as well.... :)

[CamDavidsonPilon]

I just updated conda

[bacalfa]

By the way, WeibullAFTFitter was more robust than survreg on my test cases. :) Even without any special initialization. But having the ability to initialize the decision variables is important.

[CamDavidsonPilon]

woohoo! Thanks for reporting back!