query_key value is a string instead of a dict

[AntoineBlaud]

Behavior

Method check_matches in ruletypes.py appends the query key as a string.
So match is a dict composed of simple type like string or integer.

Rules types impacted:

• metric
• percentage

On another side, all other rules create match from documents content. The method involved is add_data(self, data)

Example

Suppose that we store documents like this one in the index:
{'@timestamp': value, metadata': { 'ip': value} }

• If we use a frequency rule on this index , the match generated will have this template :

  [{**'metadata': {'ip**': "10.0.0.1"}, '@timestamp': datetime.datetime(20...o=tzutc()), '@version': '1', '_id': 'tR46i3oBTsznpu6_lwkh', '_index': 'index', '_type': '_doc'}]

• If we use a metric rule, the match generated will have this template :

  [{**'metadata.ip**': '10.0.0.1' , '@timestamp': datetime.datetime(20...o=tzutc())'}]

Possible Solution

Fix check_matches by converting query_key value="string1.string2.stringN" to ["string1']["string2]['stringN"]

[AntoineBlaud]

I can submit a pull request if you're agree with the problem.

[AntoineBlaud]

This bug is especially annoying when we use match_enhancement because depending of the rule type, match data does not have the same data structure.

[jertel]

This could break existing behavior for those who have written rules that rely upon the string format "foo.bar": 12 instead of "foo":{"bar": 12}. With that in mind it might be best to add the structured object to the match dict, as you are suggesting, but also keep the existing string match for now, and potentially mark it as deprecated in the docs.

[AntoineBlaud]

I understand the problematic but we can't put both version of a field (string and dict) into the same doc. Elasticsearch accepts only one because it converts automatically { "string1.string2.string3" : "..."} to {"string1": { "string2": { "string3" }}}

[jertel]

If that's the case then the we will have to identify this as a breaking change in the next release. Long term, I think making all rule types store match data consistently is the right direction.