Skip to content

Conversation

@ayush9pandey
Copy link
Collaborator

Changes:

  • Remove the deprecated threads argument passed in emcee and add parallel support with multiprocessing.

  • Fix inference tests to account for inference changes

  • Add two emcee benchmarks: series and paralle

  • Add example for running a biocircuit model inference with parallel support

One additional (unrelated but important) minor change: specify versions for scipy and numpy in requirements and pyproject. Numpy version greater than 2.0 breaks the installation (to be fixed later).

@ayush9pandey ayush9pandey self-assigned this Mar 22, 2025
Copy link
Collaborator

@WilliamIX WilliamIX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Did you try this and was it any faster? In the past I tried some amount of parallelization and found that on Linux boxes at least, Pool slowed things down (I think because numpy operations were being done in parallel so the OS started fighting for threads).

@ayush9pandey
Copy link
Collaborator Author

Yes, the script emcee_parallel.py runs ~10x faster than emcee_series.py on my laptop CPU with 20 cores. The key is that this is an artificially slowed down likelihood function where it 'sleeps' for a few seconds in each iteration. For simple likelihood functions, the overhead with setting up parallel ends up slowing it down, as you suggested. So, parallelization only helps for complex likelihoods.

For the real example parallel_bioscrape_inference.py with 1 Hill function and identifying 3 params, I saw a 3x improvement on the HPC.

@ayush9pandey ayush9pandey merged commit d9a7fe6 into master Mar 29, 2025
11 checks passed
@ayush9pandey ayush9pandey deleted the parallel branch March 29, 2025 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants