Djehouty intends to be a set of logging formatters and handlers to easily send log entries.
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src/djehouty fix: checkSocket Nov 1, 2017
tests fix: Python 3.x support Jan 29, 2016
.gitignore tests for gelf format Jan 28, 2016
.travis.yml devel: fixed pypi travis config Nov 10, 2017
CONTRIBUTING.rst OVH ready Jan 22, 2016
LICENSE be clear Jan 23, 2016
README.md
setup.cfg
setup.py

README.md

Build Status

Djehouty

Djehouty intends to be a set of logging formatters and handlers to easily send log entries.

This package includes:

  • for GELF:

    • a TCP/TLS handler to send log entries over TCP with TLS support
    • a formatter to convert logging record into GELF(1.1).
  • for LTSV:

    • a TCP/TLS handler to send log entries over TCP with TLS support
    • a formatter to convert logging record into LTSV.

Install

Using pip

You can use pip to install Djehouty, make sure you have the latest version:

sh-4.2# pip install --upgrade pip
[...]
Successfully installed pip-7.1.2

sh-4.2# pip install --upgrade djehouty
[...]
Successfully installed djehouty-<version> setuptools-18.3.1

Using sources

You can install from sources too:

sh-4.2$ git clone git@github.com:ovh/djehouty.git
Cloning into 'djehouty'...
remote: Counting objects: 58, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 58 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (58/58), 9.62 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.

sh-4.2$ cd djehouty
sh-4.2$ python setup.py install
[...]
Using /usr/lib/python2.7/site-packages
Finished processing dependencies for djehouty==<version>

How to send logs

The following examples assume that you already have a Flowgger (see the Flowgger documentation for more information) or a Graylog functional.

To send log messages, just use the handler of the desired format with the following parameters:

  • host (required): The hostname or ip address of the server.
  • port (optional): The TCP port to use (default: 12200 for GELF, 5140 for LTSV).
  • level (optional): Sets the threshold for the handler (default: logging.NOTSET).
  • use_tls (optional): Do we use TLS? (default: False).
  • cert_reqs (optional): Specifies whether a certificate is required from the other side of the connection, see SSL constants) (default: ssl.CERT_NONE).
  • ca_certs (optional): File that contains a set of concatenated “certification authority” certificates (default: None).
  • static_fields (optional): Additional static fields sent with all logging. record (default: empty dict).
  • sock_timeout (optional): Socket timeout is seconds (default: 1).
  • null_character (optional): Append a null character at the end of each message (required by Graylog for GELF).

With GELF over TCP/TLS (Flowgger and Graylog)

import logging
from djehouty.libgelf.handlers import GELFTCPSocketHandler

gelf_logger = logging.getLogger('djehouty-gelf')
gelf_logger.setLevel(logging.DEBUG)
gelf_logger.addHandler(GELFTCPSocketHandler(
    host            = "127.0.0.1", 
    port            = 5140, 
    static_fields   = {"app": 'djehouty-gelf'}, 
    use_tls         = True,
    level           = logging.DEBUG,
    null_character  = True,
))

gelf_logger.info('test')

With LTSV over TCP/TLS (Flowgger only)

import logging
from djehouty.libltsv.handlers import LTSVTCPSocketHandler

ltsv_logger = logging.getLogger('djehouty-ltsv')
ltsv_logger.setLevel(logging.DEBUG) 
ltsv_logger.addHandler(LTSVTCPSocketHandler(
    host            = "127.0.0.1", 
    port            = 5140, 
    static_fields   = {"app": 'djehouty-ltsv'}, 
    use_tls         = True,
    level           = logging.DEBUG
))

ltsv_logger.info('test')

Send additional meta data

If you have many handler, you can use the logging.LoggerAdapter class to add extra.

The following example uses the LTSV logger defined above:

mylogger = logging.LoggerAdapter(
    ltsv_logger,
    extra = {"myvar": 5}
)
mylogger.info('test')

You can add specific log meta for each entry using the extra parameter, the following example uses the LTSV logger defined above:

ltsv_logger.info("Hello '%s'", 'Cedric', extra={"lang": 'en'})
ltsv_logger.info("Bonjour '%s'", 'Cedric', extra={"lang": 'fr'})

Other transport

The formatters can be used directly, especially for debugging or write to files.

The following example format each log record into GELF and LTSV:

import sys
import logging
from djehouty.libgelf.formatters import GELFFormatter
from djehouty.libltsv.formatters import LTSVFormatter

logger = logging.getLogger('djehouty')
logger.setLevel(logging.DEBUG)

# GELF
hdr = logging.StreamHandler(sys.stdout)
hdr.setLevel(logging.DEBUG)
hdr.setFormatter(GELFFormatter(static_fields={"app": 'djehouty-gelf'}))
logger.addHandler(hdr)

# LTSV
hdr = logging.StreamHandler(sys.stdout)
hdr.setLevel(logging.DEBUG)
hdr.setFormatter(LTSVFormatter(static_fields={"app": 'djehouty-ltsv'}))
logger.addHandler(hdr)

logger.info("Hello %s", 'cedric', extra={"lang": 'en'})

Output (GELF and LTSV):

{"_lang": "en", "_file": "test_stdout.py", "_levelname": "INFO", "_relativeCreated": 11.034011840820312, "_processName": "MainProcess", "_thread": 140168491099904, "_exc_text": null, "_process": 28542, "version": "1.1", "_exc_info": null, "_facility": "djehouty", "short_message": "Hello cedric", "app": "djehouty-gelf", "_msecs": 308.2098960876465, "_pathname": "./test_stdout.py", "timestamp": 1440761245.30821, "_funcName": "<module>", "host": "linux-426s", "_threadName": "MainThread", "_module": "test_stdout", "level": 6, "_msg": "Hello %s", "_lineno": 30}
relativeCreated:11.0340118408	process:28542	app:djehouty-ltsv	module:test_stdout	funcName:<module>	message:Hello cedric	facility:djehouty	filename:test_stdout.py	levelno:20	processName:MainProcess	lineno:30	msg:Hello %s	host:linux-426s	exc_text:	lang:en	thread:140168491099904	level:6	threadName:MainThread	msecs:308.209896088	pathname:./test_stdout.py	time:2015-08-28T11:27:25.308210Z	exc_info:	levelname:INFO