Watch log files and send new data to a specified http endpoint
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
log2http
tests
.gitignore
LICENSE
README.md
requirements-dev.txt
setup.py

README.md

log2http

log2http watches log files and sends new contents to a specified http endpoint. That's it.

Install

You will need Python >= 3.6.

Install the application like this (preferably in a virtualenv):

pip install log2http

Setup

Place a YAML config file like the following at a location of your preference:

-
  logfile: /Users/demo/example.log
  endpoint: http://endpoint1.example.com/tag/sample
  min_lines: 5
-
  logfile: /var/log/system.log
  endpoint: http://endpoint2.example.com/tag/syslog
  min_lines: 2
  • logfile specifies the file to watch
  • endpoint is the http endpoint you want to send the collected log lines to (as POST). Usually, this will be the url of a logging service, e.g. http://logs-01.loggly.com/bulk/token/tag/example/
  • min_lines sets the minimum of lines that must be collected until a http request is made (use this to minimize http overhead per logged event)

Usage

Once installed, you can launch log2http from your terminal:

log2http --config=/path/to/the/config.yml

When log2http is running, create some sample events to see if they are being collected. For example in bash:

for i in {1..6}; do echo "hello world" >> example.log; done

The output should tell you if events are being collected and sent.

You could also start the log collector from Python like this:

from log2http import load_config, LogCollector

config = load_config('/your/path/to/config.yml')
collector = LogCollector(config)
with collector:
    collector.start() # will run until interrupted

Watch out: log lines always need to end with a newline character (\n). If a line does not end with a newline character, log2http will wait indefinitely until it is written to the file.

Local development

See requirements-dev.txt for development requirements.

Run tests with pytest.

Run static type checking with mypy --ignore-missing-imports log2http.