+
+ | sort, aggregate, or access values |
+ ES 2.x |
+ ES 5.x |
+ Example |
+
+
+ | no |
+
+"my_property" : {
+ "type": "string",
+ "index": "analyzed"
+}
+
+ |
+
+"my_property" : {
+ "type": "text"
+}
+
+ Additional defaults: "index": "true", "fielddata": "false"
+ |
+
+ "New York" handled via in-mem search as "New" and "York" buckets. No aggregation or sort.
+ |
+
+
+ |
+ yes
+ |
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+ |
+
+"my_property": {
+ "type": "text",
+ "fielddata": "true"
+}
+
+ |
+
+ "New York" handled via in-mem search as "New" and "York" buckets. Can aggregate and sort.
+ |
+
+
+ |
+ yes
+ |
+
+"my_property": {
+ "type": "string",
+ "index": "not_analyzed"
+}
+
+ |
+
+"my_property" : {
+ "type": "keyword"
+}
+
+ |
+
+ "New York" searchable as single value. Can aggregate and sort. A search for "New" or "York" will not match against the whole value.
+ |
+
+
+ |
+ yes
+ |
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+ |
+
+"my_property": {
+ "type": "text",
+ "fields": {
+ "keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+ }
+ }
+}
+
+ |
+
+ "New York" searchable as single value or as text document, can aggregate and sort on the sub term "keyword."
+ |
+
+
-With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information
+If you want to set default string behavior for all strings for a given index and type, you can do so with a mapping similar to the following (replace ${your_type_here} accordingly):
+
+```
+# curl -XPUT 'http://${ES_HOST}:${ES_PORT}/_template/default_string_template' -d '
+{
+ "template": "*",
+ "mappings" : {
+ "${your_type_here}": {
+ "dynamic_templates": [
+ {
+ "strings": {
+ "match_mapping_type": "string",
+ "mapping": {
+ "type": "text"
+ }
+ }
+ }
+ ]
+ }
+ }
+}
+'
+```
+
+By specifying the "template" property with value "*" the template will apply to all indexes that have documents indexed of the specified type (${your_type_here}). This results in the following template.
+
+```
+# curl -XGET 'http://${ES_HOST}:${ES_PORT}/_template/default_string_template?pretty'
+{
+ "default_string_template" : {
+ "order" : 0,
+ "template" : "*",
+ "settings" : { },
+ "mappings" : {
+ "${your_type_here}" : {
+ "dynamic_templates" : [
+ {
+ "strings" : {
+ "match_mapping_type" : "string",
+ "mapping" : {
+ "type" : "text"
+ }
+ }
+ }
+ ]
+ }
+ },
+ "aliases" : { }
+ }
+}
+```
+
+Notes on other settings for types in ES
+* doc_values
+ * on-disk data structure
+ * provides access for sorting, aggregation, and field values
+ * stores same values as _source, but in column-oriented fashion better for sorting and aggregating
+ * not supported on text fields
+ * enabled by default
+* fielddata
+ * in-memory data structure
+ * provides access for sorting, aggregation, and field values
+ * primarily for text fields
+ * disabled by default because the heap space required can be large
+
+
+##### Type Mapping References
+* [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html)
+* [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html)
+* [https://www.elastic.co/blog/strings-are-dead-long-live-strings](https://www.elastic.co/blog/strings-are-dead-long-live-strings)
+
+## Using Metron with Elasticsearch 5.6.2
+
+There is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information
Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service's logs.
@@ -63,7 +282,7 @@ QueryParsingException[[nested] failed to find nested object under path [alert]];
There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.
-Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it's not necessary to do each index individually. The example here appends `index*` to get all indexes for a the provided sensor.
+Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it's not necessary to do each index individually. The example here appends `index*` to get all indexes for the provided sensor.
```
export ELASTICSEARCH="node1"
@@ -89,11 +308,11 @@ To update existing indexes, update Elasticsearch mappings with the new field for
```
curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
{
- "properties" : {
- "alert" : {
- "type" : "nested"
- }
- }
+ "properties" : {
+ "alert" : {
+ "type" : "nested"
+ }
+ }
}
'
rm ${SENSOR}.template
diff --git a/metron-platform/metron-elasticsearch/pom.xml b/metron-platform/metron-elasticsearch/pom.xml
index d924891054..97f4062aed 100644
--- a/metron-platform/metron-elasticsearch/pom.xml
+++ b/metron-platform/metron-elasticsearch/pom.xml
@@ -175,9 +175,15 @@