MySQL syslog viewer
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config-example
etc
lib
public
scripts
tests/stories
views
.eslintrc
.gitignore
LICENSE
README.md
app.js
package.json

README.md

MySQL-syslog-viewer logo

MySQL-syslog-viewer

This application is a front-end for browsing Syslog data stored in a MySQL database. It is built onto Node.js and has the following features:

  • Log search/live trail.
  • Saved queries.
  • Mail alerts:
    • Triggered by message patterns.
    • Triggered by the lack of messages.
  • Log message purging.
  • Configured by JSON files.

Here is a screenshot from the working app.

Setup

The app

The application requires either Node.js version 4.x or 6.x. It might run on other versions as well. To install the dependencies, run:

npm install --production

in the project directory (cloned from GIT or downloaded).

Rsyslog

It is strongly recommended to read the official documentation for Rsyslog. Documentation and configuration are not portable between different Rsyslog versions. Make sure you read the right documentation:

An example Rsyslog configuration can be found from etc/rsyslog.conf.

Note that this configuration uses the custom SQL template to ONLY store the log message, the receive date and the tag. This is consistent with the schema distributed with the application. Some distributions like Debian include their own schema with the packaged version of rsyslog-mysql plugin.

An instance of Rsyslog can be run along the main/system instance of Rsyslog. This is recommended for testing in order to not mess up the system configuration.

A good article describing performance consideration can be found [here][handling-rsyslog-mysql]. [handling-rsyslog-mysql]:http://www.rsyslog.com/doc/v8-stable/tutorials/high_database_rate.html

MySQL

The preferred MySQL schema is in the file etc/schema.sql. As an optimization, it does not contain most of fields available from rsyslog, only the message, receive date, and the syslog tag. There is an index defined on the tag column. If there are no queries filtering by tag then the index should be dropped for performance reasons.

MySQL performance can be improved by tuning the flush policy:

innodb_flush_log_at_trx_commit = 0

This can be set though my.cnf file. See your distro manuals for the exact file location. The flush policy is a MySQL instance-wide setting. Make sure you understand consequences for other applications using the database. In the context of logging, the setting's value means that you can lose up to 1 second of logs when the server crashes.

Configuration

The configuration files are in the config directory. There are 4 files:

  • config.json - the main configuration file.
  • users.json - usernames with hashed passwords and salts.
  • alerts.json - configuration for alerts.
  • queries.json - saved queries.

Main configuration

The main configuration is divided into the following sections or objects.

{
    "port": Integer,
    "session": SessionConfiguration,
    "db": DatabaseConfiguration,
    "mail": MailConfiguration,
    "alerts": AlertsConfiguration,
    "purge": PurgeConfiguration
}

Session configuration

{
    "key": String
}

The default session key should be changed right after installing the application.

Database configuration

{
    "host": "localhost",
    "user": "logging",
    "password": "logging123",
    "database": "logging"
}

Full information on the database configuration options can be found in the mysql package documentation. The application creates connection using a pool.

Mail configuration

{
    "from": "noreply@example.com",
    "smtp": {
        "host": "smtp.example.com"
    }
}

Full information on the mail configuration options can be found in the nodemailer package documentation.

Alerts configuration

{
    "interval": 60
}
  • interval - time in seconds between running alert queries.

The specific alert checks are defined in the config/alerts.json file.

Purge configuration

{
    "interval": 600,
    "batchSize": 5000,
    "batchDelay": 10,
    "maxLines": 1000000,
    "enabled": true
}
  • interval - purge interval (seconds).
  • batchSize - single query delete limit.
  • batchDelay - delay between delete queries.
  • maxLines - the number of remaining rows in the database.
  • enabled - whether to enable log row purge or not.

Users

Generating user password/salt hashes (SHA-1):

npm run password

and enter the password. The hash and salt value along with the chose username has to be manually copied into config/users.json. The file structure:

[
    {
        "username": "logging",
        "salt": "0861f06b-4b04-42c9-bc41-41d34b4d853b",
        "hash": "d5485e640a964c23ffa914f93635a057adcf721f"
    }
]

Alerts

The alerts configuration file config/alerts.json has the following structure:

[SingleAlertConfiguration]

where SingleAlertConfiguration is:

{
    "type": String,
    "name": String,
    "mail": [String],
    "tag": String,
    "pattern": String,
    "time": Number,
    "grace": Number,
    "disabled": Boolean
}
  • type - type of the alert, either match or idle.
  • name - name of the alert, must be unique.
  • mail - array of email addresses to send alert to.
  • tag - optional syslog tag to filter rows by.
  • pattern - optional string to filter rows by.
  • time - for idle alerts the max time in seconds from the last row.
  • grace - optional time period before mails, default 3600 seconds.
  • disabled - optional (default false) parameter to disable the alert.

Saved queries

Saved queries are stored in the config/queries.json file. They appear automatically under the Queries button in the UI.

File structure:

[Category]

where a Category is:

{
    "name": String,
    "queries": [Query]
}

and a Query is:

{
    "name": String,
    "filter": {
        "tag": String,
        "tagLike": String,
        "message": String
    }
}
  • tag - optional syslog tag to filter by.
  • tagLike - optional syslog tag pattern to filter by. Used as SQL LIKE expression.
  • message - optional substring of the message to filter by.

The B-tree index on tags makes prefix queries like someapp-% fast.

State file

State file is state.json and must be writable. The state file stores last checked line ID's for alerts and times of lastly sent email alerts.

License

The MIT License. See the LICENSE file.