Skip to content

Commit

Permalink
add explanation and example in docstring
Browse files Browse the repository at this point in the history
  • Loading branch information
GreatYYX committed Sep 1, 2018
1 parent 0af0669 commit 09c81ed
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 1 deletion.
3 changes: 2 additions & 1 deletion docs/mod_parallel_processor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ ParallelProcessor

.. automodule:: rltk.parallel_processor
:members:
:special-members:
:special-members:
:exclude-members: __dict__, __weakref__, __init__
31 changes: 31 additions & 0 deletions rltk/parallel_processor.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,30 @@
"""
This module is designed for breaking the restriction of Python Global Interpreter Lock (GIL): It uses multi-processing (compute-intensive operations) and multi-threading (return data collecting) to accelerate computing.
Once it's initialized, it creates a sub process pool, all the added data will be dispatched to different sub processes for parallel computing. The result sends back and consumes in another thread in current main process. The Inter Process Communication (IPC) between main process and sub processes is based on queue.
Example::
result = []
def dummy_computation_with_input(x):
time.sleep(0.0001)
return x * x, x + 5
def output_handler(r1, r2):
result.append(r1 if r1 > r2 else r2)
pp = ParallelProcessor(dummy_computation_with_input, 8, output_handler=output_handler)
pp.start()
for i in range(8):
pp.compute(i)
pp.task_done()
pp.join()
print(result)
"""

import multiprocessing as mp
import threading
import queue
Expand Down Expand Up @@ -33,6 +60,10 @@ class ParallelProcessor(object):
output_handler (Callable): If the output data needs to be get in main process (another thread),
set this handler, the arguments are same to the return from input_handler.
The return result is one by one, order is arbitrary.
Note:
Do NOT implement heavy compute-intensive operations in output_handler, they should be in input_handler.
"""

# Command format in queue. Represent in tuple.
Expand Down

0 comments on commit 09c81ed

Please sign in to comment.