REF: Multipletests reduce memory usage #1396

Merged
merged 7 commits into from Feb 19, 2014

Projects

None yet

1 participant

@josef-pkt
Member

see #1394
reducing memory consumption and some duplicate, redundant calculations in multipletests

incomplete: not all methods work yet with 5300**2 pvalues, but I want to get TravisCI results for the refactoring

@josef-pkt
Member

After the last commit, all methods except 'hommel' finish without memory error
I attached test script to #1394

I don't know what to do about hommel, but will always be slow since it has a len(pvals) loop.
'hommel' is the most complex method, even if it's just a few lines.

@josef-pkt
Member

about finished with the last commit

I don't know if I should really still add overwrite_input=True
and I think returnsorted should be dropped. I don't really see a strong use case.

@josef-pkt
Member

There is still a bit of slack left, memory consumption could be reduced a bit further by using out arguments in numpy functions, and more inplace operation.
However, since it reduces readability, I'm waiting until that is really necessary.

I'm going to merge when TravisCI comes back green.

@josef-pkt josef-pkt merged commit a167d72 into statsmodels:master Feb 19, 2014

1 check passed

default The Travis CI build passed
Details
@josef-pkt josef-pkt deleted the josef-pkt:multitest_memory branch Feb 19, 2014
@josef-pkt josef-pkt added the PR label Mar 11, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment