Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
logsniffer is a open source web tool for parsing, viewing, monitoring and analyzing log data. Inspired by shell tools like
grep we built logsniffer with the vision to make logs accessible in a same way from a smart web interface.
To understand how logsniffer works you should have a quick look on the key concepts and how these stick together.
What is a log entry?
A log entry represents in logsniffer a single logged message in the continuous log stream. logsniffer parses the raw log message and structures it for further processing into fields. In fact a log entry is a map of fields (in Java a Map). These arise from the original message in dependency of the log source / reader configuration. A field is identified by a string key and a value of arbitrary type. Due to the fields are serialized for further processing to JSON the following (Java) types are supported to guarantee correct deserialization: Byte, Boolean, String, Date, SeverityLevel, Integer, Long, Float, Double, List of values, Map of values or a complex object.
The following internal fields are preserved by default:
lf_endOffsetof the message in the raw log medium to allow navigation
lf_timestampvalue of type Date to represent the timestamp of the log message
lf_severityvalue of type SeverityLevel to represent the level of the log message
lf_rawvalue which contains the whole original message in string format
What is a log source?
A log source accesses the log data from it's physical medium like files, database etc. and transfers it to log entries, which will be displayed in the viewer, monitored and analysed.
A log source is defined and configured flexibly on logsniffer's web interface. The configuration addresses typically the below aspects:
- The physical representation of the log e.g. file, database etc. and how this can be accessed.
- A reader configuration for parsing the raw log data into the described log entries format.
- Several filters to be able to post-process the parsed fields. Example: Type conversion from string to date format for fields with date information.
What is a sniffer?
A sniffer is a configuration entity in logsniffer used for monitoring of logs. It's defined and configured flexibly by the user on logsniffer's web interface.
A sniffer is either active or passive. Once activated the sniffer is scheduled continuously with the the defined interval and scans logs for entries of the specific interest. Example: Scan for entries with erroneous severity.
Detected events are stored in the own Elasticsearch index for further retrieval and published optionally to configured publishers like e-mail, http targets etc.
What is an event?
An event stays for one or multiple log entries recognized and recorded by a sniffer. It represents usually an occurrence of specified interest on the underlying system. An event contains also fields with values extracted or calculated from the matching entries. Example: A grep scanner stores the matching groups in fields. Event data is organized in analogy to log entries in a map data structure with the JSON serialization and deserialization to/from JSON in mind.
Beside custom fields depending on the scanner configuration the following event fields are preserved by default:
_id: Id of the event identifying the event in the Elasticsearch index
lf_entries: A list of log entries representing the event
lf_published: Timestamp value when this event was encountered and published
lf_timestamp: Timestamp of the first log entry (
lf_entries.lf_timestamp) if present. Otherwise the value will be assigned to the same timestamp as the
lf_snifferId: Id of the sniffer
lf_logSourceId: Id of the log source
lf_logPath: Log path in the source
What is a publisher?
A publisher is configured in context of a sniffer to push the recognized events to the specific channel like mail, http etc. Publishing of events happens additionally to the main use case of storing events in the own Elasticsearch index.