Merge pull request #1 from sherzberg/initial-implementation

initial implemenation

.gitignore

@@ -0,0 +1,14 @@
+.idea/
+*.iml
+*.ipr
+*.iws
+.classpath
+.project
+.settings/
+target/
+dependency-reduced-pom.xml
+node_modules
+node
+build
+build.config.js.sample
+

.travis.yml

@@ -0,0 +1,27 @@
+sudo: required
+dist: trusty
+language: java
+jdk:
+  - oraclejdk8
+addons:
+  apt:
+    packages:
+      - rpm
+
+before_deploy:
+  - mvn jdeb:jdeb && export RELEASE_DEB_FILE=$(ls target/*.deb)
+  - rm -f target/original-*.jar
+  - export RELEASE_PKG_FILE=$(ls target/*.jar)
+  - echo "Deploying release to GitHub releases"
+
+deploy:
+  provider: releases
+  api_key:
+    secure: A0PVv/9Iclwsvtr8ZR2GQvqjF8Pyirj8IlPA579iVe4rlaMJaNC43I23pMP87EkjchB7JVFTvMNI0Y6QAbDpXGpiS2J3KrgQlHP1UdgwI99eVbYgvjWWkg/tUu1RIdkMQsof5j4gpU7sWfclML3U5HvQkbvcR/JHscOGlw8l2OWvdU1oOnlAbjzN3ev0KIMMmMjIDKfixmLMRyM7oOJPlT8D0nKmAgnidbw5k423pTHJtemoYXKwxoVUeAebRIgBJOiz+auhU0tcDrnbF6EqqnylMQi6H0Y2JzasRuU0wRB47ZyMj5JJN2wu5WxiUwfbjaDSLmWuGEMQ+uRA7/Im4xRtykEiXzWfMRyiBm36vSGb0VovwxwljXMyrTpSfEpI5ewZnpjXuvsvneiStGOq/0ciEecuaNJZMGKjmz9acFH+k2i8rA4rT0hXDVDt9nSja8zQbh6SVXGq/q2HkamBONKX28jhFSBa9c7RQ8tvsITVIFl0+e13cOPW9OQrI0iXknVGVBydnuknb010fqrW8QtcK/HfhFU1t8WJtXQf7SwY7tn7eKFUPZ4/Qc4tTnl0XYOq0MeGDkhFvrwZoZU3wQSjusqjy/Ztu8gKb/RwfKeVcrgeP98pBrW2KGAMF7MEqv/6fUv5zCKj25So+1PvqxM20nbCQKIBvQkS1HnIliw=
+  file:
+    - "${RELEASE_PKG_FILE}"
+    - "${RELEASE_DEB_FILE}"
+  skip_cleanup: true
+  on:
+    tags: true
+    jdk: oraclejdk8

README.md

@@ -0,0 +1,119 @@
+# S3InputPlugin Plugin for Graylog
+
+[![Build Status](https://travis-ci.org/https://github.com/sherzberg/graylog-plugin-s3.svg?branch=master)](https://travis-ci.org/https://github.com/sherzberg/graylog-plugin-s3)
+
+This plugin provides an input plugin for AWS S3 files. It reads [S3 Events via SQS](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) from your AWS account to then pull in the S3 files into Graylog for processing.
+
+**Required Graylog version:** 2.0 and later
+
+## Installation
+
+[Download the plugin](https://github.com/https://github.com/sherzberg/graylog-plugin-s3/releases)
+and place the `.jar` file in your Graylog plugin directory. The plugin directory
+is the `plugins/` folder relative from your `graylog-server` directory by default
+and can be configured in your `graylog.conf` file.
+
+Restart `graylog-server` and you are done.
+
+## S3 setup and configuration
+
+### Step 1: Write some logs to S3
+
+Start by enabling something to write log files to S3, like FluentD (http://docs.fluentd.org/articles/out_s3):
+
+The writer could be unique to your setup, so this is left to the reader. The log line contents can be raw text or json. No processing or extraction is done by this plugin. It is left up to you.
+
+### Step 2: Enable S3 Event Notifications to SNS
+
+Create an SNS topic to write the events to, then configure S3 Events for Create and Delete to write to the SNS topic created above (remember the name of the SNS topic for later).
+
+### Step 3: Set up SQS for S3 notifications
+
+Navigate to the AWS SQS service (in the same region as the SNS topic) and hit **Create New Queue**.
+
+You can leave all settings on their default values for now but write down the **Queue Name** because you will need it for the Graylog configuration later. Our recommended default value is *s3-notifications*.
+
+S3 will write notifications about log files that were created or removed. Let’s subscribe the SQS queue to the S3 SNS topic you created in the first step now:
+
+Right click on the new queue you just created and select *Subscribe Queue to SNS Topic*. Select the SNS topic that you configured when setting up your S3 bucket events. **Hit subscribe and you are all done with the AWS configuration.**
+
+### Step 4: Install and configure the Graylog S3 plugin
+
+Copy the `.jar` file that you received to your Graylog plugin directory which is configured in your `graylog.conf` configuration file using the `plugin_dir` variable.
+
+Restart `graylog-server` and you should see the new input type *AWS S3 Input* at *System -> Inputs -> Launch new input*. The required input configuration should be self-explanatory.
+
+**Important:** The IAM user you configured in “System -> Configurations” has to have permissions to read S3 objects and delete and read notifications from SQS:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "Stmt1411854479000",
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject"
+      ],
+      "Resource": [
+        "arn:aws:s3:::s3-logfiles/*"
+      ]
+    }
+  ]
+}
+```
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "Stmt1411834838000",
+      "Effect": "Allow",
+      "Action": [
+        "sqs:DeleteMessage",
+        "sqs:ReceiveMessage"
+      ],
+      "Resource": [
+        "arn:aws:sqs:eu-west-1:450000000000:s3-write"
+      ]
+    }
+  ]
+}
+```
+
+(Make sure to replace *resource* values with the actual ARNs of your environment)
+
+### Usage
+
+You should see your log messages coming in after launching the input. (Note that it can take a few minutes based on how frequent systems are accessing your AWS resource) **You can even stop Graylog and it will catch up with all S3 log data that have been written since it was stopped when it is started again.**
+
+**Now do a search in Graylog. Select “Search in all messages” and search for:** `source:"s3"`
+
+The raw log lines in your S3 objects should show up in your search results. You can now use Extractors, Plugins, Pipeline, etc to process your raw messages.
+
+For example, if you write json lines to your S3 files, you can use the JSON Extractor to extract all the json attributes.
+
+
+## Development
+
+This project is using Maven 3 and requires Java 8 or higher.
+
+* Clone this repository.
+* Run `mvn package` to build a JAR file.
+* Optional: Run `mvn jdeb:jdeb` and `mvn rpm:rpm` to create a DEB and RPM package respectively.
+* Copy generated JAR file in target directory to your Graylog plugin directory.
+* Restart the Graylog.
+
+## Plugin Release
+
+
+We are using the maven release plugin:
+
+```
+$ mvn release:prepare
+[...]
+$ mvn release:perform
+```
+
+This sets the version numbers, creates a tag and pushes to GitHub. Travis CI will build the release artifacts and upload to GitHub automatically.

build.config.js

@@ -0,0 +1,6 @@
+const path = require('path');
+
+module.exports = {
+  // Make sure that this is the correct path to the web interface part of the Graylog server repository.
+  web_src_path: path.resolve(__dirname, '../graylog2-server', 'graylog2-web-interface'),
+};

package.json

@@ -0,0 +1,23 @@
+{
+  "name": "S3InputPlugin",
+  "version": "1.0.0-SNAPSHOT",
+  "description": "",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/sherzberg/graylog-plugin-s3"
+  },
+  "scripts": {
+    "build": "webpack --bail"
+  },
+  "keywords": [
+    "graylog"
+  ],
+  "author": "Spencer Herzberg <spencer.herzberg@gmail.com>",
+  "license": "MIT",
+  "dependencies": {
+  },
+  "devDependencies": {
+    "webpack": "^1.12.2",
+    "graylog-web-plugin": "file:../graylog2-server/graylog2-web-interface/packages/graylog-web-plugin"
+  }
+}