Add dotexpander processor

[martijnvg]

Adds a processor the turns fields with dots into object fields, so that other processors can clean the data up before indexing.

[talevy]

LGTM

UPDATED

[rjernst]

I'm confused on the purpose of this feature? Dots in field names don't matter in master. They all get read the same way, as field name separators. So to document parsing, this:

{
  "foo" : {
      "bar.baz" : "value"
   }
}

will be no different than:

{
  "foo" : {
      "bar": {
          "baz" : "value"
      }
   }
}

[talevy]

@rjernst

This is for pre-indexing, we have our own fieldname resolution and do not have such a differentiator without this PR. am I missing something?

[rjernst]

@talevy There is no reason to structure the fields with dots in the names, they are not treated any different by document parsing than if the structure was there as in my example.

[talevy]

@rjernst, I agree that there is no reason, but users may have some other reasons for having them be that way and it makes sense for us to have a way to support it; no?

[rjernst]

but users may have some other reasons for having them be that way

IMO this is untenable long term. The point of ingest is to get the data setup to be searchable in a particular way. What the source looks like should not matter, and continuing to add things which allow users to rely on the structure of _source prevent improvements like #9034.

[talevy]

I kind of view it the other way. I view this as enabling users to convert between untenable structures into better ones where they can use ingest to move their legacy sources away from the dots-in-fields structure.

That being said, responsibility for resolving this on the client end seems acceptable for me as well.

[talevy]

after speaking with @rjernst offline, I think we should pre-process the document beforehand to convert objects of this type:

{ "foo.bar": "baz" } 

into objects of this type:

{ "foo": { "bar" : "baz" } }

Since the object mappers do this internally while indexing anyways... it makes sense for Ingest to do it upfront. This will allow us to freely describe fields with dots without the need for escaping

thanks @rjernst for the chime, I think this is a friendlier solution since this is the default behavior in core anyways

[martijnvg]

I like idea to convert fields with dots into object fields.

I just wonder what the behaviour should if a document being processed has both object and with field names with dots (very rare case). Something like this:

{
  "foo.bar" : "value1",
   "foo" : {
      "bar" : "value2"
    }
}

Should we then just turn the bar field under foo field into an array field holding both values?

[clintongormley]

I'm not sure I agree with automatically converting dots to objects here, for exactly the reason @martijnvg has pointed out: you need to deal with conflicts. The ingest processor is the point at which you can take your raw messy documents and transform them into something more useful. In other words, you can take a doc like { "foo": 5, "foo.bar": 10 } and rewrite it into something that can be indexed into Elasticsearch.

Instead of transforming dots to objects automatically, i would provide this as function as a processor. Instead of adding support for escaping of dots, I would recommend using the script processor (which already has syntax for accessing fields with dots). This would allow the user to fix their document as they see fit, after which they can apply the dots-to-objects processor.

[martijnvg]

@talevy @clintongormley I've updated the PR to add a dedot processor instead of adding general support for dots in field names.

[clintongormley]

I think this should be Dedot Options?

[talevy]

happens to me all the time

[martijnvg]

my shameless copy pasting...

[clintongormley]

thanks @martijnvg - what happens when there are conflicting fields? Worth adding to the docs?

[clintongormley]

I'm also wondering if this should be called dedot given that its behaviour is very different from the logstash version. What about dot_expander?

[talevy]

I agree with @clintongormley dedot is rather confusing. especially since it used to have a different meaning.

[martijnvg]

what happens when there are conflicting fields?

It turns it into an array and you need to deal with it (even when types are conflicting, and if that isn't resolved then when serializing to json or in ES it will scream).

What about dot_expander?

+1

[clintongormley]

It turns it into an array and you need to deal with it (even when types are conflicting, and if that isn't resolved then when serializing to json or in ES it will scream).

What about this document?

{
  "foo": 5,
  "foo.bar": 10
}

[martijnvg]

@clintongormley So this would need to be done in two steps, first rename foo to foo.bar and then dot expend foo.bar (field with the actual dot) field. This way the actual implementation remains simple for this processor.

[clintongormley]

@martijnvg what i mean is: what would this processor do if it encountered this document? Throw an exception? What does that exception look like? And should we add this info to the docs?

[martijnvg]

The error that is now thrown is not clear (java.lang.IllegalArgumentException: cannot set [foo] with parent object of type [java.lang.String] as part of path [foo.bar]), so I can try to improve that. I did update the docs, explaining to use the rename processor in this situation.

[talevy]

missing a be

Otherwise these fields can't be accessed by any processor.

[talevy]

as a side note, should we link to this section here? https://www.elastic.co/guide/en/elasticsearch/reference/master/accessing-data-in-pipelines.html

[talevy]

LGTM

[talevy]

unused import