Skip to content

Commit

Permalink
improve docs of parallel_processor
Browse files Browse the repository at this point in the history
  • Loading branch information
GreatYYX committed Jul 15, 2020
1 parent 4d1f3f9 commit 9e2c594
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions pyrallel/parallel_processor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
If you have a some time-consuming statements in a for-loop and no state is shared among loops, you can map these
statements to different processes. Assume you need to process couple of files, you can do this in parallel::
statements to different processes. Assume you need to process a couple of files, you can do this in parallel::
def mapper(filename):
with open(filename) as f_in, open(filename + '.out') as f_out:
Expand All @@ -18,8 +18,8 @@ def mapper(filename):
pp.task_done()
pp.join()
It's not required to have a loop statement if you have iterable object or type (list, generator, etc),
use a shortcut instead::
It's not required to write a cumbersome loop statement if you have iterable object or type (list, generator, etc).
Instead, you could use `map`::
pp = ParallelProcessor(2, mapper)
pp.start()
Expand All @@ -30,7 +30,7 @@ def mapper(filename):
pp.join()
Usually, some files are small and some are big, it would be better if it can keep all cores busy.
One way is to send line by line to each process (assume content is line-separated)::
One way is to send content line by line to each process (assume content is line-separated)::
def mapper(line, _idx):
with open('processed_{}.out'.format(_idx), 'a') as f_out:
Expand Down Expand Up @@ -61,6 +61,8 @@ def exit(self, *args, **kwargs):
def process(self, line):
self.f.write(process_a_line(line))
pp = ParallelProcessor(..., mapper=MyMapper, ...)
In some situations, you may need to use `collector` to collect data back from child processes to main process::
processed = []
Expand Down

0 comments on commit 9e2c594

Please sign in to comment.