Make flattened synthetic source concatenate object keys on scalar/object mismatch

[parkertimmins]

There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source.

This fixes the issue by replacing the problematic object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by .. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.

For example if the following flattened field values is indexed:

{
   "a": {
       "b": 5
       "b": {
          "c": 10,
          "d": { "e": 15 }
       }
   }
}

The following synthetic source will be produced:

{
   "a": {
      "b": 5
      "b.c": 10,
      "b.d.e": 15
   }
}

Fixes #122936

[kkrik-es]

What if the conflict doesn't happen on the first part of the prefix, e.g.

field {
  path {
    to: 10
    to {
      foo: bar
    }
  }
}

Would this be caught here?

[parkertimmins]

Yep, so this will become:

field {
  path {
    to: 10
    to.foo: bar
 }
}

When it get to the first key/value field.path.to|10 it will take the else block and traverse down into the object, adding field and path to the openObject context. When it reaches the key value field.path.to.foo|bar that object will still be open, and seeing that lastScalarSingleLeaf has a value of to, and that to is the first token in the startPrexix (to.foo), it will make a concatenated path.

(Updated a test to this situation to verify it)

[parkertimmins]

Interestingly, there was a test which had a scalar/object mismatch. But it produced duplicate keys. When the xcontent was converted to jsont these duplicate keys originally threw an error, but now just drop the duplicates. (Something must have changed in xcontent stuff since the issue was opened to cause this change from an error to deduplication)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Hi @parkertimmins, I've created a changelog YAML for you.