add sample files to assist with building new parsers or extending exi…

…sting ones.

PULLREQUESTS.md

@@ -1,15 +1,22 @@
 Submitting Pull Requests
 ========================
 
-The code in this repository is used in a number of different environments, so these pull request guidelines are designed to allow the various moving parts to
- continue working.  Please take a moment to read them before submitting a pull request - in fact, before doing any customization.
+The code in this repository is used in a number of different environments, so these pull request guidelines are designed to allow the various moving parts to continue working.  Please take a moment to read them before submitting a pull request - in fact, before doing any customization.
 
- *PLEASE NOTE*: These guidelines are currently under heavy revision and will be changed in the near future.  We still would greatly appreciate your help, but please contact a developer (@PhilHagen) for help before proceeding if you plan to submit a PR.  We just want to prevent you from undergoing our own headaches down the road.
+*PLEASE NOTE*: These guidelines are rudimentary and will certainly be changed in the future.  We still would greatly appreciate your help, but please contact a developer (@PhilHagen) for help before proceeding if you plan to submit a complex PR.  We just want to prevent you from undergoing our own headaches down the road.
 
 1. All development must be done from the "develop" branch.  PRs against master will not be accepted.
-1. Do not modify the included stock dashboards, except for bug-level edits.  New dashboards will be considered, but as the codebase is designed to address a broad community, highly customized dashboards may not be accepted into master.
-1. Any custom parsers must be created in the /configfiles-UNSUPPORTED/ subdirectory.  Any that are suitable for universal deployment will be moved to the /configfiles/ subdirectory by the SOF-ELK® team.
-1. All IP addresses pulled via grok must be in a field with a name formatted as such: <directionality>_ip or <use_case>_ip
-  * Examples: `source_ip`, `destination_ip`, `relay_ip`, `answer_ip`
-1. All IP addresses must be enriched with the GeoIP location and ASN filters (see existing files for examples)
-1. All IP addresses must be added to the "ips" array field (see existing files for examples)
+2. Do not modify the included stock dashboards, except for bug-level edits.  New dashboards will be considered, but as the codebase is designed to address a broad community, highly customized dashboards may not be accepted into master.
+3. Any custom parsers must be created in the `/configfiles-UNSUPPORTED/` subdirectory.  Any that are suitable for universal deployment will be moved to the `/configfiles/` subdirectory by the SOF-ELK® team.
+4. All IP addresses pulled via grok must be in a field with a name formatted as such: `<directionality>_ip` or `<use_case>_ip`
+    * Examples: `source_ip`, `destination_ip`, `relay_ip`, `answer_ip`
+5. All IP addresses must be enriched with the GeoIP location and ASN filters (see existing files for examples)
+6. All IP addresses must be added to the "ips" array field (see existing files for examples)
+
+Consult the [sample Filebeat prospector](lib/filebeat_inputs/filebeat_template.yml.sample) for information on how to create a new Filebeat log source.  If extending an existing log type (e.g. syslog), this is not needed.
+
+Consult the [sample parsing configuration file](configfiles/6xxx-parsing_template.conf.sample) for information on how to build a filter configuration file.
+
+Consult the [sample output configuration file](confgfiles/configfiles/9xxx-output-template.conf.sample) for information on how to build an output configuration file.  If extending an existing log type (e.g. syslog), this is not needed.
+
+Consult the [sample template file](lib/elasticsearch-example-template.json.sample) for a basic Elasticsearch template for your index, unless using an existing index.  If inserting to an existing index, you would instead modify that index template rather than creating a new one.

configfiles/6xxx-parsing_template.conf.sample

@@ -0,0 +1,90 @@
+# SOF-ELK® Configuration File
+# Author: Phil Hagen
+# Email: phil@lewestech.com
+#
+# (C)2020 Lewes Technology Consulting, LLC
+#
+# This is a VERY simple example configuration file that can be used inline with other SOF-ELK files
+# To use:
+# Make a copy of the file, removing the .sample extension and leaving hte .conf.  Replace the
+# '6xxx' prefix with a 4-digit value.  This value will determine WHEN the file is parsed in relation
+# to the others.  Logstash basically just cat's all these to one giant configuration blob, so the digits
+# establish order of parsing.  These numbers have been generally assigned in groups - reach out via GitHub
+# if you'd like to have a number or group of numbers assigned.
+
+filter {
+    # Be sure this file only applies to the necessary type(s).
+    # This improves performance, and is not limited to the [type] field - any comparative logic can be used.
+    if ([type] == "logtype") {
+
+        # Drop what you don't want to handle.
+        # The directive below drops comments (assuming comments are lines starting with a # character.)
+        if ([message] =~ /^#/) {
+            drop {}
+        }
+
+        # If you want to use a particular type of blanket filter, put it here.
+        # Be sure to consult the documentation on the filter capability, as it can vary wildly between filters.
+        # By default this is applied against the "message" field, but this can be changed.
+        csv {
+            columns => ["PSComputerName", "RunspaceId", "PSShowComputerName", "RecordType", "CreationDate", "UserIds", "Operations", "AuditData", "ResultIndex", "ResultCount", "Identity", "IsValid", "ObjectState"]
+            skip_header => "true"
+        }
+
+        # This is critical!
+        # It sets the @timestamp field by default, which is used for most visualizations.
+        # The source field is first ("CreationDate" in this example), and the format filter to apply is second.
+        # The format field is very extensive, and can be written to adapt to nearly any date value you encounter.
+        # Note that SOF-ELK runs in UTC, so unless the time stamp has an offset included, the timestamp will be
+        # interpreted as UTC.
+        # Think of it as constructing a timestamp by applying the format like a regex against the source field.
+        date {
+            match => [ "CreationDate", "ISO8601" ]
+        }
+
+        # If further blanket filters are needed, perform them as needed.
+        # In this case, we're parsing JSON from the "AuditData" source field and placing it in "auditdata".
+        # With the JSON filter, this means the subfields will be named "auditdata.whatever".
+        json {
+            source => "AuditData"
+            target => "auditdata"
+        }
+
+        # You would have to do this AFTER the previous directive - if done in the same directive, Logstash
+        # might remove the field before the parser is applied.
+        mutate {
+            remove_field => "AuditData"
+        }
+
+        # Some common data migration and normalization steps below.
+        mutate {
+            # Rename a field without changing nested level
+            rename => {
+                "PSComputerName" => "hostname"
+                "RecordType" => "record_type"
+            }
+
+            # Create a new field.            
+            # Renaming does not seem to work from a nested field.
+            # Also note the syntax to use the data from a nested field.
+            add_field => {
+                "source_ip" => "%{[auditdata][ClientName]}"
+                "some_field" => "%{RunspaceId} : %{RecordType} event from %{hostname}"
+            }
+            remove_field => "SomethingElse"
+        }
+
+        # If you want to perform GeoIP lookups against IP addresses, name the field "<something>_ip" and ensure
+        # the 8050-postprocess-ip_addresses.conf file is modified (if needed) to enrich the field.  Several fields
+        # are already enriched, so you may not need to do anything here.
+
+        # The possibilities here are literally endless...
+
+        # Last step (or at least somewhere along the way), you should put some meaningful string into the "message" field
+        # This may be the original data in some cases, or a human-readable summary line that is used in many
+        # visualizations on the Dashboards as well as the Discovery tab.
+        mutate {
+            replace => { "message" => "This is a log from %{source_ip} and the operation was %{Operations}." }
+        }
+    }
+}

configfiles/9xxx-output-template.conf.sample

@@ -0,0 +1,20 @@
+# SOF-ELK® Configuration File
+# (C)2020 Lewes Technology Consulting, LLC
+#
+# This file is a template for outputting to Elasticsearch
+# To use:
+# Make a copy of the file, removing the .sample extension.  Replace the '9xxx' prefix with a
+# 4-digit value. See the 6xxx template for an explanation of how these numbers are used.
+
+output {
+  # use standard logic syntax here
+  if [type] == "logtype" or [type] == "other_logtype" {
+    elasticsearch {
+      index => "logtype-%{+YYYY.MM.dd}"
+      # adjust template filename as needed
+      template => "/usr/local/sof-elk/lib/elasticsearch-logtype-template.json"
+      template_name => "logtype"
+      template_overwrite => true
+    }
+  }
+}

lib/elasticsearch-example-template.json.sample

@@ -0,0 +1,54 @@
+{
+  "index_patterns": [
+    "logtype-*"
+  ],
+  "settings": {
+    "index": {
+      "number_of_shards" : 1,
+      "auto_expand_replicas": "0-1",
+      "codec": "best_compression"
+    }
+  },
+  "mappings": {
+    "dynamic_templates": [
+      {
+        "ip_address_fields": {
+          "match": "*_ip",
+          "mapping": {
+            "type": "ip"
+          }
+        }
+      },
+      {
+        "string_fields": {
+          "match": "*",
+          "match_mapping_type": "string",
+          "mapping": {
+            "type": "text",
+            "norms": false,
+            "fields": {
+              "keyword": {
+                "type": "keyword",
+                "ignore_above": 256
+              }
+            }
+          }
+        }
+      }
+    ],
+    "properties": {
+      "@version": {
+        "type": "keyword"
+      },
+
+      "prospector": {
+        "properties": {
+          "type": {
+            "type": "keyword"
+          }
+        }
+      }
+    }
+  },
+  "aliases": {}
+}

lib/filebeat_inputs/filebeat_template.yml.sample

@@ -0,0 +1,21 @@
+# SOF-ELK® Configuration File
+# (C)2020 Lewes Technology Consulting, LLC
+#
+# This is a sample Filebeat prospector file that looks into the /logstash/logtype/ directory for files to load.
+# These are sent to localhost:5044, where Logstash is listening.
+# To use this, make a copy, rename and remove the .sample extension and change "logtype" as needed.  Create the
+# parent directory in /logstash/ and chmod it to 1777.
+
+- type: log
+  paths:
+    - /logstash/logtype/*/*/*/*/*
+    - /logstash/logtype/*/*/*/*
+    - /logstash/logtype/*/*/*
+    - /logstash/logtype/*/*
+    - /logstash/logtype/*
+  exclude_files: [ 'readme.txt', '\.gz$', '\.bz2$', '\.zip$' ]
+  close_inactive: 5m
+  fields_under_root: true
+  fields:
+    # this is important - it's as the [type] field in Logstash
+    type: logtype