Recursive path_match in dynamic template does not work for trailing *

[kylelyk]

Elasticsearch version: 6.2.4

Plugins installed: []

JVM version: 1.8.0_172

OS version: MacOS (Darwin Kernel Version 15.6.0)

Description of the problem including expected versus actual behavior:

PUT test
{
  "mappings": {
    "_doc": {
      "dynamic_templates": [
        {
          "template": {
            "path_match": "prop1.prop2.*",
            "mapping": {
              "type": "keyword",
              "index": false
            }
          }
        }
      ],
      "properties": {
        "prop1": {
          "properties": {
            "prop2": {
              "type": "object",
              "dynamic": true
            }
          }
        }
      }
    }
  }
}

PUT test/_doc/1
{
  "prop1": {
    "prop2": {
      "prop3": {
        "prop4": {
          "field": "bar"
        }
      }
    }
  }
}

Returns the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse [prop1.prop2.prop3]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse [prop1.prop2.prop3]",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "Can't get text on a START_OBJECT at 4:16"
    }
  },
  "status": 400
}

However, if you replace the path_match in the dynamic template to prop1.prop2.*.field, it works as expected and matches prop1.prop2.prop3.prop4.field.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jimczi]

The trailing * works as expected, it matches every field that appears in the path prop1.prop2.* , prop1.prop2.prop3 matches the rule so it tries to map this path to a keyword field and fails.
Dynamic templates operate at every level of the document (objects or leaves) so you need to take intermediary level into consideration when choosing your match rule. I am going to close this issue because the behavior is expected, @kylelyk please comment if I missed anything.

[kylelyk]

I now understand the reasoning behind how this works. However, the documentation is very vague on exactly how the * matcher works. Maybe this could be updated with the three different use cases: starting, middle, trailing?

[jimczi]

I see how confusing this can be so I am reopening this issue to clarify the docs.
Thanks @kylelyk .

[jtibshirani]

Closing now that the docs have been updated in #41658.

[eddycharly]

@jimczi @jtibshirani do you have a solution to path_match only on leaves ?
I hit the same issue, trying to index everything as keyword for all leaves under a top level property (context.*).

Thanks

[jtibshirani]

@eddycharly in this issue we only documented the existing behavior, but didn't make plans to change or enhance the behavior. Would you be able to file a new 'enhancement request' issue? You could describe your use case, including an example of your mappings.

[eddycharly]

@jtibshirani i found a possible solution that seems to work. Just wanted to know if there was a valid approach to implement the use case described here.

[jtibshirani]

Got it. To clarify, currently there's no dedicated way to ensure path_match only matches leaves.

[eddycharly]

The solution that seems to work:

    "mappings" : {
      "dynamic_templates" : [
        {
          "strings_as_keywords" : {
            "mapping" : {
              "type" : "keyword"
            },
            "match_mapping_type" : "string"
          }
        },
        {
          "context_objects_as_objects" : {
            "path_match":   "context.*",
            "match_mapping_type": "object",
            "mapping" : {
              "type": "{dynamic_type}"
            }
          }
        },
        {
          "context_as_keyword" : {
            "path_match":   "context.*",
            "mapping" : {
              "type" : "keyword"
            }
          }
        }
      ]
    }

Basically if it matches an object, it keeps the object (this is not a leaf node), else it uses a keyword (for leaf nodes).

What do you think ? Are there some cases where it would not work ?
Thanks.

[jtibshirani]

@eddycharly apologies, I misunderstood your earlier comment and thought you were suggesting an enhancement/ feature. That approach seems like it would work to me. If the goal is just to handle this context object, I think you could simplify and remove the "strings_as_keywords" template.

We try to reserve GitHub for discussions about bugs or feature development, as opposed to usage questions. If you have more questions, I suggest we hop over to the Elastic forums to continue the discussion.

[eddycharly]

If the goal is just to handle this context object, I think you could simplify and remove the "strings_as_keywords" template.

strings_as_keywords template makes sure strings are not analyzed at all, i don't need full text search. "match_mapping_type" : "string" ensures that it applies only to strings.

context_objects_as_objects takes care of nested objects (non leaf nodes) under the context property.

context_as_keyword indexes all leaf nodes under the context property as strings, regardless of their original type (to avoid clashes, same keys can have different type in the document so forcing indexing type as strings fixes clashes).