parallel processing #72

Closed
josef-pkt opened this Issue Sep 21, 2011 · 1 comment

Projects

None yet

1 participant

@josef-pkt
Member

just some thoughts

do we use joblib?

Alexandre rewrote the permutation t-test to use joblib.

Problems on Windows

http://sourceforge.net/mailarchive/forum.php?thread_name=CAFvE7K5adiMWa9VZHn8G3fTheU3t1_ymdWOZF1bVm1tMA1Sm1w%40mail.gmail.com&forum_name=scikit-learn-general

see especially Robert's comments

cost of creating new process on Windows:
my guess: if each new process needs to import scipy.stats, then there is not much point to multiprocessing on Windows.
correct or not? (example would be an rvs from scipy.stats.distributions)

@josef-pkt
Member

added joblib in #111

for discussion
https://groups.google.com/group/pystatsmodels/browse_thread/thread/72fe8d8369e1d53e
https://groups.google.com/group/pystatsmodels/browse_thread/thread/c2dbea9345778ad7

conclusion was
importing joblib from scikit-learn is slow
needs at least a few seconds of calculations to pay for the extra cost

@josef-pkt josef-pkt closed this May 2, 2012
@josef-pkt josef-pkt added a commit to josef-pkt/statsmodels that referenced this issue Jul 17, 2013
@josef-pkt josef-pkt REF: numdiff add lower bound on epsilon, closes #72 6bc9e2b
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@josef-pkt josef-pkt REF: numdiff add lower bound on epsilon, closes #72 9f6442e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment