New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
incorrect and inconsistent logging in multiprocessing #48551
Comments
logging in multiprocessing seems to give inconsistent results on Linux According to the docs, when first created "the logger has level This should print a message from the main process and the subprocess: import multiprocessing, logging
def f():
logger = multiprocessing.get_logger()
logger.info('info from subprocess')
if __name__ == '__main__':
logger = multiprocessing.get_logger()
logger.setLevel(logging.INFO)
logger.info('info from main process')
p = multiprocessing.Process(target=f)
p.start()
p.join() but, on Windows (XP) and Linux (ScientificLinux 5.2, python 2.6 compiled No handlers could be found for logger "multiprocessing" We can 'fix' this by specifying a handler, as in: import multiprocessing, logging
def f():
logger = multiprocessing.get_logger()
logger.info('info from subprocess')
if __name__ == '__main__':
logger = multiprocessing.get_logger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter('[%(levelname)s/%(processName)s]
%(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('info from main process')
p = multiprocessing.Process(target=f)
p.start()
p.join() Then on Linux I get: [INFO/MainProcess] info from main process which looks better, but on Windows I get: [INFO/MainProcess] info from main process So, the logger is still not setup correctly in the subprocess on import multiprocessing, logging
def f():
logger = multiprocessing.get_logger()
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter('[%(levelname)s/%(processName)s]
%(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('info from subprocess')
if __name__ == '__main__':
logger = multiprocessing.get_logger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter('[%(levelname)s/%(processName)s]
%(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('info from main process')
p = multiprocessing.Process(target=f)
p.start()
p.join() Now, on Linux I get: [INFO/MainProcess] info from main process i.e. double output from the subprocess! [INFO/MainProcess] info from main process which is almost OK, but the logger is getting set up too late to pick up It seems like the logger is not being setup correctly by |
I ran into this shortly but was able to overcome it by patching 43-44: 270-271: (and some info about %(processName)s into the docstring) to make it After that the following works like intended: import multiprocessing
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s
%(processName)s %(threadName)s %(levelname)s %(message)s')
# omit next line to get only what your code is logging
logger = multiprocessing.get_logger()
def f():
logging.info('info from subprocess')
if __name__ == '__main__':
logging.info('info from main process')
p = multiprocessing.Process(target=f)
p.start()
p.join() Surely no perfect solution, but the monkey-patching in Other option is to not use %(processName)s in logging format, but import multiprocessing
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(process)s
%(threadName)s %(levelname)s %(message)s')
# omit next line to get only what your code is logging
logger = multiprocessing.get_logger()
def f():
logging.info('info from subprocess')
if __name__ == '__main__':
logging.info('info from main process')
p = multiprocessing.Process(target=f)
p.start()
p.join() 2009-01-05 23:48:02,046 4000 MainThread INFO info from main process |
This is a patch to fix the logging module, and remove the crufty code in |
remove the import, not needed. switch to the property |
fixed in r68737, merged to py3k in 68740 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: