Python Logger + RSysLog + Logstash. Demo over docker containers.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
logstash
project
rsyslog
.editorconfig
.gitignore
LICENSE
README.md
docker-compose.yml

README.md

Python Syslog Logstash

Demonstration of logging system for Python projects. Fast and safe logs collecting.

Scheme

  1. Python project -- sent logs.
  2. Rsyslog -- collect logs from projects (can be many rsyslogs on many servers).
  3. Redis -- message queue between rsyslog and logstash.
  4. Logstash -- retrieve data from Redis, select index and add into ElasticSearch.
  5. ElasticSearch -- logs storage.
  6. Kibana -- web-interface.

Usage

  1. Run:
    sudo docker-compose up
  2. Open Kibana: 127.0.0.1:5601/app/kibana
  3. Go to Management -> Index patterns.
  4. Click on "refresh fields". If "create" button still inactive then wait while ElasticSearch is ran.
  5. Click "create"
  6. Go to "Discover". This is your data :)

If you want to see indices then go to "Dev Tools" section and run this command:

GET /_cat/indices

Example

Example of one log message from ElasticSearch:

{
    "@timestamp": "2018-04-27T12:18:39.199Z",
    "@version": "1",
    "message": {
        "name": "app_name",
        "module": "app",
        "lineno": 79,
        "message": null,
        "random_string": "ydrvlhdruj",
        "random_integer": 302
    },
    "facility_label": "user",
    "facility": "1",
    "hostname": "pythonsysloglogstash_psl-project_1.pythonsysloglogstash_default",
    "program": "",
    "relayhost": "pythonsysloglogstash_psl-project_1.pythonsysloglogstash_default",
    "relayip": "172.21.0.7",
    "severity_label": "crit",
    "severity": "2",
    "tag": "",
    "type": "syslog"
}