offlog is a non-blocking file-append service for Unix systems. It consists of a server
program and Python client library that interact via a Unix domain socket, allowing
applications to offload file appends to the server and mitigating the risk of
application latency caused by blocking file I/O. It has a focus on performance,
specifically on minimising overhead for the client.
Both client and server are written in Python, but the protocol is not Python-specific so other clients could be written as well. This package has no dependencies outside the Python standard library.
Install to your virtualenv or other Python environment using pip
pip install offlogOr build from source
git clone https://github.com/chrisjbillington/offlog
cd offlog
pip install setuptools wheel
pip install .Or build a wheel
git clone https://github.com/chrisjbillington/offlog
cd offlog
pip install setuptools wheel build
python -m build .
Example:
# example client code example.py
from offlog import Logger
logger = Logger("my app", '/tmp/myapp.log')
logger.info("Hello world, params are %s %d", "foo", 7)
try:
1/0
except Exception:
logger.exception("Got an error")Start the the server in one terminal, and then run the client in another. You'll see the following:
$ python -m offlog
[2023-07-03 10:12:21.895 offlog INFO] This is offlog server
[2023-07-03 10:12:21.895 offlog INFO] Listening on socket /tmp/offlog.sock
[2023-07-03 10:12:54.796 offlog INFO] Client 0 connected
[2023-07-03 10:12:54.796 offlog INFO] Client 0 access confirmed for /tmp/myapp.log
[2023-07-03 10:12:54.796 offlog INFO] New client 0 (total: 1) for /tmp/myapp.log
[2023-07-03 10:12:54.799 offlog INFO] Client 0 disconnected
[2023-07-03 10:12:54.799 offlog INFO] Client 0 done (remaining: 0) with /tmp/myapp.log
[2023-07-03 10:12:54.799 offlog INFO] Closed /tmp/myapp.log$ python example.py
[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7
[2023-07-03 10:12:54.796 my app ERROR] Got an error
Traceback (most recent call last):
File "/path/to/example.py", line 6, in <module>
1/0
~^~
ZeroDivisionError: division by zeroAnd the contents of /tmp/myapp.log will be:
$ cat /tmp/myapp.log
[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7
[2023-07-03 10:12:54.796 my app ERROR] Got an error
Traceback (most recent call last):
File "/path/to/example.py", line 6, in <module>
1/0
~^~
ZeroDivisionError: division by zeroThe Logger() class by default outputs DEBUG and higher to file, INFO to stdout,
and ERROR and higher to stderr. These are configurable. Formatting and exception
options are similar to the Python standard library.
The focus of this library is more on low-overhead file appends than the Logger()
class, which is very basic. It is easily subclassable if you want to change some of its
behaviour, or you could feasibly plug the FileProxy() object that sends data to the
offlog server into some other logging framework. You may also use the FileProxy
object directly to proxy file appends, including of binary data, to a file.
Despite being offloaded to another process, all file appends are still reliable, in the sense that all data will eventually be written if the server and client shut down in normal ways. This includes if the client encounters a Python exception, but does not include if the client, e.g. segfaults or is terminated via SIGTERM and Python's default SIGTERM handler (which exits immediately without doing any normal Python cleanup).
The server does flush files after every write, so when things are running normally, logs are written in real-time. The only reason they might be delayed is when the system is under load and messages are getting backed up - in which case clients will not retry sending queued data until the next time they have new data to send, or until program shutdown, whichever comes first.
There is no facility for global look up of loggers. Pass them around yourself. They are not thread-safe, but if you instantiate one per thread pointing to the same file, they will both talk to server which will interleave their appends. But their data won't be guaranteed to be in the same order or even have the same boundaries as they were fired off in your application.
Instantiating a FileProxy is blocking - the server will verify it can open the file and
we wait for it to confirm. Writing data is not ever blocking, and for now data that
fails to send because the server is too busy will just be locally buffered indefinitely
(you can still run out of memory). Unsent data will be retried before sending new data
on subsequent calls to write(). At shutdown, all unsent data will attempted to be
flushed to the server, this is also blocking.
If the server is shut down or encounters an exception writing data on the client's behalf, it will terminate the connection with the client. Upon the next attempt to write the client will get a BrokenPipeError and read the exception message from the server, raising it as an exception in the client code.
offlog.Logger(
name=None,
filepath=None,
file_level=DEBUG,
stdout_level=INFO,
stderr_level=WARNING,
local_file=False,
offlog_socket_path='/tmp/offlog.sock',
offlog_timeout=5000
)Logging object to log to file, stdout and stderr, with optional proxying of file
writes via a offlog server.
Records with level file_level and above will be written to the given file. Records
with level stdout_level and above, up to but not including stderr_level will be
written to stdout, or with no upper limit if stdout_level is None. Records with level
stderr_level and above will be written to stderr. Any of these can be set to None to
disaable writing to that stream. filepath=None will also disable file logging.
if local_file is True, then an ordinary file will be opened for writing. Otherwise
offlog_socket_path is used to connect to a running offlog server, which will open the
file for us, writes will be proxied through it. Any blocking operations communicating
with the server (such as the initial file open, and flushing data at shutdown) will be
subject to a communications timeout of offlog_timeout in milliseconds, default 5000.
UTF-8 encoding is assumed throughout.
offlog.Logger.close()Close the file. Possibly blocking. Idempotent.
Logging methods work similarly to the Python standard library:
offlog.Logger.log(level, msg, *args, exc_info=False):
offlog.Logger.debug(msg, *args, exc_info=False):
offlog.Logger.info(msg, *args, exc_info=False):
offlog.Logger.warning(msg, *args, exc_info=False):
offlog.Logger.error(msg, *args, exc_info=False):
offlog.Logger.exception(msg, *args, exc_info=True)
offlog.Logger.critical(msg, *args, exc_info=False)offlog.ProxyFile(
filepath,
sock_path='/tmp/offlog.sock',
timeout=5000,
)Object to proxy appending file writes via a running offlog server.
offlog.ProxyFile.write( data):Send as much as we can without blocking. If sending would block, queue unsent data for
later. On BrokenPipeError, check if the server sent us an error and raise it if so.
This method will always attempt to re-send previously-queued data before attempting to
send new data.
offlog.ProxyFile.close():Close the socket. Attempt to send all queued unsent data to the server and cleanly close the connection to it, raising exceptions if anything goes wrong
$ python -m offlog -h
usage: python -m offlog [-h] [-n] [-s SOCKET_FILE] [-l LOGFILE]
offlog file-append service
options:
-h, --help show this help message and exit
-n, --notify Notify systemd when the server has started up and is
listening.
-s SOCKET_FILE, --socket-file SOCKET_FILE
Path of the Unix socket the server binds to. Default:
/tmp/offlog.sock
-l LOGFILE, --logfile LOGFILE
Path of the (optional) log file for the offlog server
itself.You may wish to start the service as a systemd service. Here is an example unit file:
# offlog.service
[Unit]
Description=offlog file-append service
After=network.target
[Service]
Type=notify
ExecStart=/path/to/your/.venv/bin/python -u -m offlog --notify
Restart=always
RestartSec=5s
User=<user_to_run_as>
[Install]
WantedBy=multi-user.targetNote: the server will run with permissions of the specific user, which means all file operations will be performed as that user. The server's unix socket file will be created with default access permissions of that user, and thus only users will permission to access that socket will be able to communicate with the server. This means by default, clients will not be able to access an offlog server running as root in order to write to files they would not have otherwise had permission to write to.
To avoid race conditions in systemd starting other services that depend on the offlog
server, ensure you use the --notify flag to the offlog server, which will notify
systemd when the server's socket is bound and it is ready to accept clients. Mark other
units that require the offlog server by adding Requires and After lines to the
[Unit] section of their service files:
[Unit]
Description=Some other service that needs an offlog server
...
Requires=offlog.service
After=offlog.serviceThe server shuts down cleanly upon SIGINT (i.e. pressing ctrl-C when run from a
terminal) or SIGTERM (i.e. what systemd will send by default if you run systemctl stop, or at shutdown). It will close all client connections, and write any data clients
have already sent to files. Running clients will get an exception upon their next
attempt to write data.
TODO - document the protocol