Log-tool is a framework for connecting and processing streams of data. It has a simple modular architecture, consisting of input streams, output streams, and "parsers". The "parsers" are both input and output streams, and they (generally) perform some operation on that data that passes through them. Many of these parsers can be combined in series to perform more complex operations. (Also note that one "connection" can potentially have several inputs and several outputs operating in parallel, also shown in the diagram below.)
All of these components are based on the Node Stream interface, to keep all components mutually-compatible and to increase code reuse between this and other projects.
This project includes the core framework as well as some simple modules of each of these types. The modular architecture is intended to make it easy to develop additional modules as needed, and several of these additional modules are listed below. All modules are mutually-compatible, and all of them receive their configuration options in the same way (described below.)
The original purpose of this project was reading, parsing, pre-processing, and outputting log information, but this pattern is general enough that it should be useful in a variety of applications.
npm start to start the service (with the settings specified in the config files)
npm stop to stop the service
npm restart to restart the service
npm test to run the tests.
All of the following are json files located in the config directory.
The file app.json handles any configuration options for the application overall, such as error logging and verbosity, and recoginzing input/output/parser modules.
The default is to use the development environment, so in production, simply change the environment setting at the top of the file:
"environment": "production"
The remaining options will be stored in the item of that name ("development", "production", etc.) These environment/object names can be arbitrary. These options contain misc. application-wide settings, such as verbosity and location of error logs generated at runtime.
To recognize and use modules that are external to the main project, their names must be listed in the inputModules, parserModules, or outputModules lists, as appropriate. If modules are listed here, they will be loaded from the node_modules directory. Otherwise, they will be loaded from the input output or parsers directories of this project.
The file input.json specifies all needed information for each source of data.
In general, this config file names specific instances using their specified modules, and then pass the remaining values to that module for its constructor to handle. Each instance name acts as a key for getting its configuration object. A simple example may contain:
{
"source1":{
"module":"file",
"fileName":"test/data/some_file",
"encoding":"utf-8"
},
"source2":{
"module":"file-watch",
"fileName":"in.txt",
"timeout": 15000,
"interval": 100,
"encoding":"utf-8"
}
}
"source1" and "source2" are the names of these sources, "file" and "file-watch" are the modules these sources will use, and the remaining options will be passed to these instances of these modules. In this example, the "file" module will be past a relative path and an encoding, while "file-watch" will be passed a relative path, an encoding, an interval to check for changes to this file, and a timeout value for when to stop watching this file.
The file output.json specifies all needed information for each destination of the data.
Like above, this config file names specific instances using their specified modules, and then pass the remaining values to that module for its constructor to handle. Each instance name acts as a key for getting its configuration object. A simple example may contain:
{
"dest1":{
"module":"file",
"fileName":"ids.out.txt"
},
"dest2": {
"module" : "redis-pubsub",
"serverAddress" : "127.0.0.1",
"serverPort" : 6379,
"channel" : "events.nessus"
}
}
"dest1" and "dest2" are the names of these destinations, "file" and "redis-pubsub" are the modules these will use, and the remaining options will be passed to these instances of these modules. In this example, the "file" module only needs to be passed a path, while the "redis-pubsub" module will be passed an address, a port, and the channel to use for sending its messages.
As another example:
"nessus-store": {
"module" : "redis",
"serverAddress" : "127.0.0.1",
"serverPort" : 6379,
"keyPrefix" : "logtool:events:nessus",
"index" : true,
"indexedFields" : ["ip", "port", "vulnid"]
}
This instance is named "nessus-store", uses the "redis" module, and passes it similar values as above, except instead of needing a channel name, it needs information about how to name its keys and which fields to include in its index.
The file parsers.json specifies all needed information for each parser.
Like above, this config file names specific instances using their specified modules, and then pass the remaining values to that module for its constructor to handle. Each instance name acts as a key for getting its configuration object. An example may contain:
{
"null":{
"module":"null"
},
"nessus":{
"module":"nessus"
},
"firewall":{
"module":"regex"
, "regex": "^([^,]*),([^,]*),([^,]+),([^,]+),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)"
, "labels": ["timestamp", "priority", "operation", "messageCode",
"protocol", "sourceIP", "destIP", "sourceHostname", "destHostname", "sourcePort",
"destPort", "destService", "direction", "connectionsBuilt", "connectionsTornDown"
]
, "fields": { "timestamp": {"regex": "DD/MMM/YYYY HH:mm:ss", "type": "moment"} }
, "delimiter": "\r\n|\n"
, "startTime":0
}
}
"null", "nessus", and "firewall" are again the names of these parsers, and "null", "nessus", and "regex" are the modules these sources will use. The "null" and "nessus" modules do not require any additional options, while the "regex" module requires many additional options due to its more generic purpose.
The file connections.json is somewhat different from the other configuration files. This simply contains a list of connections to make between the names in the other config files. For example:
[
{
"input":"source1",
"parser":"null",
"output":"dest1"
},
{
"input":"nessus-file",
"parser":"nessus",
"output":"nessus-store"
}
]
The first entry will connect input to the parser named null to the output dest1 -- this example will simply copy the file unmodified.
The second entry will connect nessus-file to the parser nessus to the output nessus-store -- this example will read and parse the specified nessus file, and store each entry in redis.
As mentioned above, these can also consist of arrays, such as:
[
{
"input":"firewall-vast12",
"parser":["firewall","firewall-slice"],
"output":["fw-store","fw-pubsub"]
}
]
This example will read from the specified source, send the output through all of the parsers in the specified order, and then output to the specified modules in parallel.
logtool.js - launches the main logtool process.
input, output, and parsers directories - contain the modules of each type
config directory - contains configuration files, as described above
test directory - contains all test scripts and data. Tests for each module are included in test/module_type/module_name.test.js
lib directory - contains all core libraries for logtool (ie. everything except for the modules described above)
All tests can be run with npm test This will run jshint and the mocha tests.
Note that when testing, the main logtool process and all modules take their options directly from an opts argument, and do not read or modify the log files above.
log-tool is freely distributable under the terms of the MIT License.
Copyright (c) Michael Iannacone (the "Original Author")
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS, THE U.S. GOVERNMENT, OR UT-BATTELLE BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.