mpacklog.py

Stream dictionaries to files or over the network using MessagePack in Python.

Installation

$ pip install mpacklog

Usage

Asynchronous API

Add messages to the log using the put function, have them written to file in the separate write coroutine.

import asyncio
import mpacklog

async def main():
    logger = mpacklog.AsyncLogger("output.mpack")
    await asyncio.gather(main_loop(logger), logger.write())

async def main_loop(logger):
    for bar in range(1000):
        await asyncio.sleep(1e-3)
        await logger.put({"foo": bar, "something": "else"})
    await logger.stop()

if __name__ == "__main__":
    asyncio.run(main())

Synchronous API

The synchronous API is similar to the asynchronous API, except it doesn't provide a stop method and the put and write methods are blocking.

import mpacklog

logger = mpacklog.SyncLogger("output.mpack")

for bar in range(1000):
    logger.put({"foo": bar, "something": "else"})

# Flush all messages to the file
logger.write()

Command-line

If you pip-installed mpacklog, you can use the mpacklog command to dump logs to JSON:

mpacklog dump my_log.mpack

Alternatively and more generally, two great tools to manipulate logs from the command line are:

• rq: transform from/to MessagePack, JSON, YAML, TOML, ...
• jq: manipulate JSON series to add, remove or extend fields

For instance, mpacklog dump is equivalent to:

rq -mJ < my_log.mpack