Home

Introduction

logsniffer is a open source web tool for parsing, viewing, monitoring and analyzing log data. Inspired by shell tools like less, tail and grep we built logsniffer with the vision to make logs accessible in a same way from a smart web interface.

Key concepts

To understand how logsniffer works you should have a quick look on the key concepts and how these stick together.

What is a log entry?

A log entry represents in logsniffer a single logged message in the continuous log stream. logsniffer parses the raw log message and structures it for further processing into fields. In fact a log entry is a map of fields (in Java a Map). These arise from the original message in dependency of the log source / reader configuration. A field is identified by a string key and a value of arbitrary type. Due to the fields are serialized for further processing to JSON the following (Java) types are supported to guarantee correct deserialization: Byte, Boolean, String, Date, SeverityLevel, Integer, Long, Float, Double, List of values, Map of values or a complex object.

The following internal fields are preserved by default:

• The lf_startOffset and the lf_endOffset of the message in the raw log medium to allow navigation
• Optional lf_timestamp value of type Date to represent the timestamp of the log message
• Optional lf_severity value of type SeverityLevel to represent the level of the log message
• The lf_raw value which contains the whole original message in string format

What is a log source?

A log source accesses the log data from it's physical medium like files, database etc. and transfers it to log entries, which will be displayed in the viewer, monitored and analysed.

A log source is defined and configured flexibly on logsniffer's web interface. The configuration addresses typically the below aspects:

• The physical representation of the log e.g. file, database etc. and how this can be accessed.
• A reader configuration for parsing the raw log data into the described log entries format.
• Several filters to be able to post-process the parsed fields. Example: Type conversion from string to date format for fields with date information.

What is a sniffer?

A sniffer is a configuration entity in logsniffer used for monitoring of logs. It's defined and configured flexibly by the user on logsniffer's web interface.

A sniffer is either active or passive. Once activated the sniffer is scheduled continuously with the the defined interval and scans logs for entries of the specific interest. Example: Scan for entries with erroneous severity.

Detected events are stored in the own Elasticsearch index for further retrieval and published optionally to configured publishers like e-mail, http targets etc.

What is an event?

An event stays for one or multiple log entries recognized and recorded by a sniffer. It represents usually an occurrence of specified interest on the underlying system. An event contains also fields with values extracted or calculated from the matching entries. Example: A grep scanner stores the matching groups in fields. Event data is organized in analogy to log entries in a map data structure with the JSON serialization and deserialization to/from JSON in mind.

Beside custom fields depending on the scanner configuration the following event fields are preserved by default:

• _id: Id of the event identifying the event in the Elasticsearch index
• lf_entries: A list of log entries representing the event
• lf_published: Timestamp value when this event was encountered and published
• lf_timestamp: Timestamp of the first log entry (lf_entries[0].lf_timestamp) if present. Otherwise the value will be assigned to the same timestamp as the lf_published field.
• lf_snifferId: Id of the sniffer
• lf_logSourceId: Id of the log source
• lf_logPath: Log path in the source

What is a publisher?

A publisher is configured in context of a sniffer to push the recognized events to the specific channel like mail, http etc. Publishing of events happens additionally to the main use case of storing events in the own Elasticsearch index.