propagate the config for logging to batch jobs

[TaiSakuma]

propagate the config for logging to batch jobs

[TaiSakuma]

• loggers cannot be pickled in python 2.7 or 3.6
• loggers can be pickled in python 3.7

[TaiSakuma]

• working on multiprocessing first

• using the threading version in the example at https://docs.python.org/3/library/logging.handlers.html#queuehandler

• tried the multiprocessing version as well. decided not to use for now because handlers need to be configured in the other process. copy an example code here for the record.

class LoggingListener(multiprocessing.Process):
    def __init__(self, queue):
        multiprocessing.Process.__init__(self)
        self.queue = queue

    def run(self):
        while True:
            record = self.queue.get()
            if record is None:
                break
            logger = logging.getLogger(record.name)
            logger.handle(record)

the logger in the main process won't received the record.

• idea for the future

  • pass record to the progress bar so that the progress bar can control both progress bars and logging messages on stdout.

• useful links

  • http://plumberjack.blogspot.co.uk/2010/10/logutils-using-recent-logging-features.html
  • http://plumberjack.blogspot.co.uk/2010/09/using-logging-with-multiprocessing.html
  • https://docs.python.org/3/library/logging.handlers.html#queuehandler

[TaiSakuma]

implemented: 611214e