Skip to content

Commit

Permalink
Update logging cookbook to show multiple worker processes using the c…
Browse files Browse the repository at this point in the history
…oncurrent.futures module. (#14905)
  • Loading branch information
vsajip committed Jul 22, 2019
1 parent 7397cda commit d309352
Showing 1 changed file with 35 additions and 0 deletions.
35 changes: 35 additions & 0 deletions Doc/howto/logging-cookbook.rst
Expand Up @@ -948,6 +948,41 @@ This variant shows how you can e.g. apply configuration for particular loggers
machinery in the main process (even though the logging events are generated in
the worker processes) to direct the messages to the appropriate destinations.

Using concurrent.futures.ProcessPoolExecutor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you want to use :class:`concurrent.futures.ProcessPoolExecutor` to start
your worker processes, you need to create the queue slightly differently.
Instead of

.. code-block:: python
queue = multiprocessing.Queue(-1)
you should use

.. code-block:: python
queue = multiprocessing.Manager().Queue(-1) # also works with the examples above
and you can then replace the worker creation from this::

workers = []
for i in range(10):
worker = multiprocessing.Process(target=worker_process,
args=(queue, worker_configurer))
workers.append(worker)
worker.start()
for w in workers:
w.join()

to this (remembering to first import :mod:`concurrent.futures`)::

with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
for i in range(10):
executor.submit(worker_process, queue, worker_configurer)


Using file rotation
-------------------

Expand Down

0 comments on commit d309352

Please sign in to comment.