Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize hessian estimation #50

Closed
mattwigway opened this issue Oct 1, 2020 · 2 comments
Closed

Parallelize hessian estimation #50

mattwigway opened this issue Oct 1, 2020 · 2 comments

Comments

@mattwigway
Copy link
Owner

Hessian estimation is the slowest part of the model fitting process, and it's single-threaded. This seems like something where parallelization should be possible. Investigate.

@mattwigway
Copy link
Owner Author

Parallelization is definitely possible with the new linear_hessian code introduced in #52. I think the right way to do it is to pre-calculate all steps and store them in shared memory, along with the utility, and then use a multiprocessing pool to chug through all the hessian results in parallel. The challenge is how to do this without having multiprocessing copy the whole MNLFullASC object.

@mattwigway
Copy link
Owner Author

Since Numpy is good about releasing the GIL, I can use threads and not have to deal with shared memory.

@mattwigway mattwigway reopened this Dec 30, 2020
mattwigway added a commit that referenced this issue Dec 31, 2020
multithreaded Hessian computation, fixes #50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant