Multithreaded startrail processing. #649
Conversation
Inspired by @sebmueller. This splits up the input images into batches of work more or less evenly divided across the number of processors. Two new flags are added: * -n / --nice to make the workers be nice. default = 10 * -m / --max-threads to control the number of worker threads. default = ncpus Allows me to process a night of startrails on my workstation in under 3.5s. On my pi3b+, it'll use between 350% - 395% CPU without making it feel slow ``` root@skycam:/home/allsky/src# time ./startrails -b 0.255 -d ../images/20210912/ -e jpg -o /tmp/junk.jpg -s 1280x960 Minimum: 0.0265446 maximum: 0.804403 mean: 0.143959 median: 0.137787 real 5m5.951s user 11m22.243s sys 1m5.347s root@skycam:/home/allsky/src# time ./startrails -b 0.255 -d ../images/20210912/ -e jpg -o /tmp/junk2.jpg -s 1280x960 --max-threads 1 Minimum: 0.0265446 maximum: 0.804403 mean: 0.143959 median: 0.137787 real 12m8.758s user 11m3.073s sys 0m54.200s ``` Fixes #628
|
Running on all cores but at nice 10 was entirely arbitrary. But if I had a multiprocessor machine I was throwing at an embarrassingly parallel problem like this, I'd want to use all available cores. The nice level should keep this from making your machine feel laggy but as long as you're not trying to do too much other stuff you'll get a nice speed-up. |
There was a problem hiding this comment.
Line 260: if ((tmp >= 1) || (tmp < cf->num_threads)).
Should it be "&&" instead? tmp of 1000 and -1 are both valid with an "||".
I suggest adding an error msg if tmp is out of range.
Line 265, " | -s] [-m ] [-n ]"
says -m and -n are optional, but lines 193 and 194 say "required_argument". Which is correct?
Line 287, suggest changing "nproc" to "number of processors" since many users won't know what "nproc" means.
|
Check the getopt manage. required_argument means that a value/parameter is required for that option, not that it is necessary to specify this flag. Nproc is fine. Any who knows enough about the computer architecture to know that they have multiple processors/cores and is able to accurately assess the impact of these options will understand nproc. |
|
I decided that |
Inspired by @sebmueller. This splits up the input images into batches of work
more or less evenly divided across the number of processors.
Two new flags are added:
Allows me to process a night of startrails on my workstation in under 3.5s.
On my pi3b+, it'll use between 350% - 395% CPU without making it feel slow
Fixes #628