Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add demo of batch processing via concurrent.futures #230

Merged
merged 1 commit into from Oct 27, 2016

Conversation

grlee77
Copy link
Contributor

@grlee77 grlee77 commented Oct 26, 2016

I thought the following demo for processing a set of images in parallel might be of interest. I made this while testing the GIL release implemented in #188 and just cleaned it up a bit for use as a demo.

The demo does require concurrent.futures which is part Python 3. A user on Python 2.6 or 2.7 will receive an error message to first pip install futures if it is not already installed.

Output for an older dual-socket workstation is:

Sequential Case
        Elapsed time: 516.67 ms

Multithreaded Case
        Number of concurrent workers: 16
        Elapsed time: 77.43 ms

Relative speedup with concurrent = 6.6723611214598755

On a 4-core OS X laptop I observed a speedup of 3.34.

@codecov-io
Copy link

Current coverage is 86.53% (diff: 100%)

Merging #230 into master will not change coverage

@@             master       #230   diff @@
==========================================
  Files            20         20          
  Lines          2852       2852          
  Methods           0          0          
  Messages          0          0          
  Branches        262        262          
==========================================
  Hits           2468       2468          
+ Misses          335        334     -1   
- Partials         49         50     +1   

Powered by Codecov. Last update 2151db0...5c5d6d9

@rgommers
Copy link
Member

For a standalone demo I think the extra dependency on futures is no problem at all.

@rgommers
Copy link
Member

I get a similar speedup - just under 50%, also with fewer cores (1 thread per physical core). Images are a bit small to get more I guess.

Example looks clear, no comments.

@rgommers rgommers merged commit 9f602e2 into PyWavelets:master Oct 27, 2016
@rgommers
Copy link
Member

Okay, good as is so merging. Thanks @grlee77!

@rgommers rgommers added this to the v0.5.0 milestone Oct 27, 2016
@grlee77 grlee77 deleted the concurrent branch December 30, 2016 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants