global lowercase normaliser index setting - feature request

[mbudge]

Hi,

We have spent a lot of time adding lowercase normalisers to the templates exported from beats. We would like to make it easier to apply normalisers to all the keywords for when new beats templates are released, as it will save us time porting over our changes when a new template is released.

Would it be possible to add a setting to the index so all keyword fields are converted to lowercase before indexing? This wouldn't be applied to multi-field keywords so we can store the original value. Our understanding is the normaliser is applied at index and search time, but not to the _source document so fields visible in Kibana aren't affected by the normaliser.

For example we would want to store the following as original keyword or add a multi-field uppercase normaliser:

• Filepaths
• GUID
• SID

original

• filepath.original

Uppercase

• guid.uppercase
• sid.uppercase

We do this as "Keyword fields are only searchable by their exact value" and most of the fields in our indexes are keywords.

https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html

We add the case normalises to mitigate that risk of a security analyst searching for logs in Elasticsearch, and missing a load of events because the case they entered was different to what was indexes in elasticsearch.

For example if they search for a malware/C2 domain and the case doesn't match, we still need them to find the logs.

Thanks,

Matthew

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[jtibshirani]

I assume this would be a setting like "index.analysis.normalizer.default.type": "lowercase", similar to the default analyzer setting. This seems like a reasonable addition to me.

This wouldn't be applied to multi-field keywords so we can store the original value.

We add the case normalises to mitigate that risk of a security analyst searching for logs in Elasticsearch, and missing a load of events because the case they entered was different to what was indexes in elasticsearch.

I don't think we would want a special exemption where the setting doesn't apply to keyword multi-fields. This makes me wonder -- is your issue more related to the default behavior of Beats templates, rather than Elasticsearch normalization settings? You could pass along this feedback around the importance of lowercasing some keyword fields to the Beats team in their repo.

[mbudge]

Your probably right, the lowercase setting should be applied to all fields including multi-field keywords.

We could use a dynamic_template, but the problem is Beats explicitly set all fields as they use a generator to create the template from a yaml file. When using ordered templates the explicit fields set by Beats take priority over the dynamic templates.