Snap plugin intended to process logs using regular expressions
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github
examples/tasks
processor
scripts
.gitignore
.pluginsync.yml
.sync.yml
.travis.yml
CONTRIBUTING.md
LICENSE
Makefile
README.md
glide.lock
glide.yaml
main.go

README.md

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

Snap plugin processor - logs-regexp

Snap plugin intended to process logs using regular expressions.

  1. Getting Started
  1. Documentation
  1. Community Support
  2. Contributing
  3. License
  4. Acknowledgements

Installation

Download logs-regexp plugin binary:

You can get the pre-built binaries for your OS and architecture from the plugin's GitHub Releases page. Download the plugin from the latest release and load it into snapteld (/opt/snap/plugins is the default location for Snap packages).

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-processor-logs-regexp

Clone repo into $GOPATH/src/github/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-processor-logs-regexp

Build the plugin by running make in repo:

$ make

This builds the plugin in ./build

Configuration and Usage

Documentation

The intention of this plugin is to parse logs according to the form defined by the user in task manifest.

The plugin processes log line using regular expressions with names. These names indicate tags which are added to metric. The regular expression can contain special names (timestamp and message) to replace metric timestamp and value.

The plugin can be configured by following parameters (all parameters are optional):

  • regexp_log - regular expression with method to parse log line, special names:
    • message - the main message in log which will be set as a metric value
    • timestamp - the log timestamp which will be set as a metric timestamp.
  • regexp_message - regular expression with method to parse the main message in log (indicated by message).
  • regexp_time - regular expression with method to parse the log timestamp (indicated by timestamp), this expression needs to contain following names: day, month, year, hour, seconds, minutes, seconds, timezone.

Notice: Special characters in regular expressions needs to be escaped.

On default plugin processes following log line (metric value which is returned by collector plugin):

127.0.0.1 - - [07/Dec/2016:06:00:12 -0500] "GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"

to Snap's metric with:

  • Data: "GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"
  • Timestamp: 07/Dec/2016:06:00:12 -0500
  • Tags:
    • client_ip: 127.0.0.1
    • http_method:GET
    • http_url: /v3/users/fa2b2986c200431b8119035d4a47d420/projects
    • http_version: 1.1
    • http_status: 200
    • http_response_size: 446
    • http_response_time: 21747

Examples

This is an example running snap-plugin-collector-logs, processing collected regexp-logs and writing post-processed data to a file. It is assumed that you are using the latest Snap binary and plugins.

In one terminal window, open the Snap daemon (in this case with logging set to 1 and trust disabled) with appropriate configuration needed by logs collector. To do that properly, please follow the instruction on snap-plugin-collector-logs.

$ snapteld -l 1 -t 0 --config config.json

In another terminal window:

Download and load plugins:

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-logs/latest/linux/x86_64/snap-plugin-collector-log
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-regexp/latest/linux/x86_64/snap-plugin-processor-logs-regexp
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file
$ snaptel plugin load snap-plugin-collector-logs
$ snaptel plugin load snap-plugin-processor-logs-regexp
$ snaptel plugin load snap-plugin-publisher-file

Create a task manifest - see examplary task manifests in examples/tasks:

{
  "version": 1,
  "schedule": {
    "type": "simple",
    "interval": "15s"
  },
  "workflow": {
    "collect": {
      "metrics": {
        "/intel/logs/*": {}
      },
      "process": [
        {
          "plugin_name": "logs-regexp",
          "config": {
            "regexp_log": "(?P<client_ip>\\S+) (\\S{1,}) (\\S{1,}) [[](?P<timestamp>\\d{2}[/]\\S+[/]\\d{4}[:]\\d{2}[:]\\d{2}[:]\\d{2} \\S\\d+)[]] (?P<message>.*)",
            "regexp_message": "(?P<http_method>[A-Z]{3,}) (?P<http_url>/\\S*) HTTP/(?P<http_version>\\d+.\\d+)\" (?P<http_status>\\d*) (?P<http_response_size>\\S*) (?P<http_response_time>\\S*)",
            "regexp_time" : "(?P<day>\\d{2})/(?P<month>[a-zA-Z]+)/(?P<year>\\d{4}):(?P<hour>\\d{2}):(?P<minutes>\\d{2}):(?P<seconds>\\d{2}) (?P<timezone>.\\d+)"
          },
          "process": null,
          "publish": [
            {
              "plugin_name": "file",
              "config": {
                "file": "/tmp/published_logs_with_config.log"
              }
            }
          ]
        }
      ]
    }
  }
}

Create a task:

$ snaptel task create -t task-config.json

To stop task:

$ snaptel task stop <task_id>

Roadmap

There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework

Contributing

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

And thank you! Your contribution, through code and participation, is incredibly important to us.