Speeding up Aalen Additive Regression

[springcoil]

Hi, I've been working on a project for a few months now and one problem I have is that it can take about 4 days to run on 340k rows, with about 6 features.

I know lifelines isn't necessarily designed for this and I've discovered that the ridge regression solve step is the biggest bottleneck - 60% of the compute time happens there.

Are there alternative algorithms I can use like mini-batch say? Rather than the ridge regression?

[CamDavidsonPilon]

Four days!? That is beyond inappropriate. I’ll look into this for the next release

[springcoil]

https://stats.stackexchange.com/questions/83272/fastest-way-to-run-ridge-regression-on-large-datasets-where-np makes me think that this is a problem with my BLAS/LAPACK. Do you want me to check what they are.

[springcoil]

What sort of debugging information would you need from me? How can I help you?

[springcoil]

The server I was running this on - I took the spec/versions I'm using from it. Let me know if that helps :)

NumPy version:     1.11.3
Python version:    2.7.12 | packaged by conda-forge | (default, Sep  8 2016, 14:22:31) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
Platform:          linux2-x86_64
AMD/Intel CPU?     True
VML available?     False
Number of threads used by default: 8 (out of 24 detected cores)

>>> np.__config__.show() 

lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/net/DataScience_public/jupyter_virtual_envs/python2/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/net/DataScience_public/jupyter_virtual_envs/python2/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/net/DataScience_public/jupyter_virtual_envs/python2/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/net/DataScience_public/jupyter_virtual_envs/python2/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blas_mkl_info:
  NOT AVAILABLE

[springcoil]

In https://github.com/CamDavidsonPilon/lifelines/blob/master/lifelines/fitters/aalen_additive_fitter.py#L188 is this correct. It seems to me it's predicting per row in table not per time event.

[CamDavidsonPilon]

It seems to me it's predicting per row in table not per time event.

the for loop iterates over the times https://github.com/CamDavidsonPilon/lifelines/blob/master/lifelines/fitters/aalen_additive_fitter.py#L170

[springcoil]

Ok I'll close that bugfix. Any idea how to speed this up Cameron? 4 days isn't adequate.

[CamDavidsonPilon]

For now, just sample down to a smaller number of observations

[springcoil]

I've tried that but not really suitable for my problem. But thanks :) Let me know if I can help in anyway :)

[CamDavidsonPilon]

Looking at the profile of the code, most of the time is spent in solving the least-squares problem. I've found some modest performance increases there (~50% faster for tall datasets), slated for the next release

[springcoil]

Ahh cool. Could you use gradient boosting?

…

On Tue, 27 Mar 2018, 3:21 am Cameron Davidson-Pilon, < ***@***.***> wrote: Looking at the profile of the code, most of the time is spent in solving the least-squares problem. I've found some modest performance increases there (~10%) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#421 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA8DiAqJsUHNCOOqMtI0pHDDOjmYn3MWks5tiaJBgaJpZM4Siqbs> .

[CamDavidsonPilon]

I don't know what gradient boosting would solve. The LR step is part of the inference algorithm

[springcoil]

Just so I understand, are you saying that gradient boosting wouldn't work for the inference step?

[CamDavidsonPilon]

That's correct

[springcoil]

Ahh that makes sense. It looks like your solving step change is the right approach. I suspect there's other performance improvements. Looking forward to the new release :)