Skip to content
A simple, multiprocess-safe logger for Python.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
multilog Initial working Python 2.6 support. Feb 5, 2016
tests Initial working Python 2.6 support. Feb 5, 2016
.gitignore Initial checkin. Jan 18, 2016
.travis.yml
CODE_OF_CONDUCT.md Add a code of conduct Apr 26, 2018
LICENSE Initial checkin. Jan 18, 2016
README.rst Clarify problem statement in README Nov 29, 2016
pylintrc
setup.cfg
setup.py Bump version to 1.1.0 Feb 8, 2016

README.rst

Multilog

https://coveralls.io/repos/github/humangeo/multilog/badge.svg?branch=master

A simple, multiprocess-safe logger for Python

Why

Python's built-in loggers are pretty handy - they're easily customized and come with useful functionality out of the box, including things like file rotation. These file handlers are thread-safe, but not process-safe, so, if you're attempting to write to a common log file from multiple processes (e.g. in a pre-forking web server) you run the risk of your workers trampling over each other. File locking is a possible workaround, but that's yucky.

To avoid this, it is recommended that one uses a socket-based logger (a code sample is helpfully provided in the Logging Cookbook). However, it is just a code snippet. Multilog is a dependency-free implementation of the sample socket logger with some niceties, like fileConfig support, and parameterization.

How

Once installed, the Multilog daemon can be invoked via:

mutlilog

Usage:

usage: multilog [-h] [-s SERVER] [-p PORT] [-c CONFIG_PATH]

A simple logger for multiple Python processes.

optional arguments:
  -h, --help                    show this help message and exit
  -s SERVER, --server SERVER
                                The server hostname (default: localhost)
  -p PORT, --port PORT          The port to listen on. (default: 9020)
  -c CONFIG_PATH, --config CONFIG_PATH
                                The log configuration to load. (default: logging.ini)

By default, it will look for a logging.ini file in the current directory. If one isn't found, Multilog will yell at you. A sample configuration file for the server:

[loggers]
keys=root

[handlers]
keys=multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogServerHandler

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

and for your application:

[loggers]
keys=root

[handlers]
keys=multilogClientHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogClientHandler

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

The important field is the args block in the handler_multilogClientHandler section - those parameters should correspond to the server and ports on which the multilog daemon is listening. By default, the daemon uses localhost and logging.handlers.DEFAULT_TCP_LOGGING_PORT.

For Power Users

If you want to have Multilog share your application's config, you can do the following:

[loggers]
keys=root,appName

[handlers]
keys=multilogClientHandler,multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=%(root_handler)s

[logger_appName]
level=INFO
handlers=
propagate=1
qualname=appName

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

Then, in your application, pass the root handler name into the logging config:

import logging
logging.config.fileConfig(config_path, defaults={"root_handler": "multilogClientHandler"})

Multilog will always load the multilogServerHandler handler. If you don't want to run Multilog (if you're running a single-threaded local dev server, for example), simply change your root_handler value to multilogServerHandler to write to the handler.

Support

Multilog is compatible with Python 2.6, 2.7, and 3.3+.

You can’t perform that action at this time.