Elasticsearch - add fields to index template

add-fields-to-index-template.md

@@ -0,0 +1,154 @@
+Elasticsearch - add fields to index template
+============================================
+
+Intro
+-----
+
+Problem: How to add fields to the default index template so that they are
+parsed using a different syntax than `string`?
+
+These instructions are primarily for OpenShift logging but should apply to any
+Elasticsearch installation by removing the OpenShift specific bits.  They also
+apply to Elasticsearch 2.x for OpenShift 3.4 -> 3.10, so may require
+some tweaking to work with ES 5.x.  The instructions assume your logging
+namespace is `logging` - use `openshift-logging` with OpenShift 3.10 and later.
+
+OpenShift logging Elasticsearch 2.x uses the following default index templates for
+`.operations.*` and `project.*` indices:
+https://github.com/ViaQ/elasticsearch-templates/releases/download/0.0.16/com.redhat.viaq-openshift-operations.2.4.4.template.json
+and
+https://github.com/ViaQ/elasticsearch-templates/releases/download/0.0.16/com.redhat.viaq-openshift-project.2.4.4.template.json
+Any undefined field not specified here will be treated as a string:
+
+    "mappings": {
+        "_default_": {
+          "dynamic_templates": [
+            {
+              "string_fields": {
+                "mapping": {
+                  "index": "analyzed",
+                  "norms": {
+                    "enabled": true
+                  },
+                  "type": "string"
+                },
+                "match": "*",
+                "match_mapping_type": "string"
+
+If you have time valued fields that you want to use for sorting and collation, you
+will need to define these separately in separate index templates.
+
+Steps
+-----
+
+Identify the index pattern for the indices you want to add the fields to.  For OpenShift
+logging this will be `.operations.*` or `project.*`.  If there is a specific project that
+will only have these fields, you can specify a specific index pattern for the indices for
+this project e.g. `project.this-project-has-time-fields.*`.
+
+Create a JSON file for each index pattern, like this:
+
+    {
+        "order": 20,
+        "mappings": {
+          "_default_": {
+            "properties": {
+              "mytimefield1": {
+                "doc_values": true,
+                "format": "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
+                "index": "not_analyzed",
+                "type": "date"
+              },
+              "mytimefield2": {
+                "doc_values": true,
+                "format": "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
+                "index": "not_analyzed",
+                "type": "date"
+              }
+            }
+          }
+        }
+        "template": ".operations.*"
+    }
+
+Call this one `add-time-fields-to-operations-indices.json`.
+
+    {
+        "order": 20,
+        "mappings": {
+          "_default_": {
+            "properties": {
+              "mytimefield1": {
+                "doc_values": true,
+                "format": "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
+                "index": "not_analyzed",
+                "type": "date"
+              },
+              "mytimefield2": {
+                "doc_values": true,
+                "format": "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
+                "index": "not_analyzed",
+                "type": "date"
+              }
+            }
+          }
+        }
+        "template": "project.*"
+    }
+
+
+Call this one `add-time-fields-to-project-indices.json`.
+
+Load these into Elasticsearch.  You'll need the name of one of the Elasticsearch
+pods:
+
+    oc get -n logging pods -l component=es
+
+Pick one and call it `$espod`.  If you have a separate OPS cluster, you'll need
+to identify one of the es-ops Elasticsearch pods too, for the `.operations.*`
+indices:
+
+    oc get -n logging pods -l component=es-ops
+
+Pick one and call it `$esopspod`.
+
+Load the file `add-time-fields-to-project-indices.json` into `$espod`:
+
+    file=add-time-fields-to-project-indices.json
+    cat $file | \
+    oc exec -n logging -i -c elasticsearch $espod -- \
+        curl -s -k --cert /etc/elasticsearch/secret/admin-cert \
+        --key /etc/elasticsearch/secret/admin-key \
+        https://localhost:9200/_template/$file -XPUT -d@- | \
+    python -mjson.tool
+
+Load the file `add-time-fields-to-operations-indices.json` into `$esopspod`, or
+`$espod` if you do not have a separate OPS cluster:
+
+    file=add-time-fields-to-operations-indices.json
+    cat $file | \
+    oc exec -n logging -i -c elasticsearch $esopspod -- \
+        curl -s -k --cert /etc/elasticsearch/secret/admin-cert \
+        --key /etc/elasticsearch/secret/admin-key \
+        https://localhost:9200/_template/$file -XPUT -d@- | \
+    python -mjson.tool
+
+**NOTE** The settings will not apply to existing indices.  You will need to
+perform a reindexing for that to work.  However, it is usually not a problem,
+as the settings will apply to new indices, and curator will eventually delete
+the old ones.
+
+Results
+-------
+
+To see if this is working, wait until new indices are created, and use the
+`oc exec` command as above to query:
+
+    oc exec -n logging -i -c elasticsearch $esopspod -- \
+        curl -s -k --cert /etc/elasticsearch/secret/admin-cert \
+        --key /etc/elasticsearch/secret/admin-key \
+        https://localhost:9200/project.*/_search?sort=mytimefield2:desc | \
+    python -mjson.tool
+
+You should see your records sorted in descending order by the `mytimefield2` field
+with the latest time listed first.