Include/Exclude Filtering Behavior

[RobCherry]

Currently there is a bug in elasticsearch (#4047) where empty objects are not stored in _source when an include/exclude list is present.

This is because elasticsearch aggressively removes empty objects from the _source.

For example, if I have an object

{ 'data': { 'key': 'value' } }

the following filters will all result in removing data from _source: excludes = ['data'], excludes = ['data.*'], excludes = ['data.key'], excludes = ['*.key'], includes = ['data.other']

I believe this behavior is incorrect. I think that we should only remove an object if it is explicitly excluded (excludes = ['data'], excludes = ['data.*']) or if no elements are included (includes = ['other_data.*']). For situations where the object is referenced in an includes list but there is no match, I think the object should remain as an empty object (includes = ['data.other']`).

This use case makes more sense if we are talking about some nested object that is indexed...

Example:

{
  "name": "John Doe",
  "identifiers": {
    "ssn": "987-65-4320",
    "facebook_uid": "12345",
    "twitter_uid": "54321"
  }

and excludes = [ "*.ssn"] would drop the entire identifiers object if the only key was ssn for that object, even if we want the empty identifiers object to remain under all circumstances. We have the same problem with includes = ["*.instagram_uid"].

[bleskes]

Thanks for opening it. One quick comment about:

We have the same problem with includes = ["*.instagram_uid"].

This is tricky - in your example it makes sense but the implications are that we'd have to leave all other objects intact - which will potentially result in a lot of empty objects (trees). At least for this case I'd use include [ "identifiers", "*.instagram_uid" .

[RobCherry]

Fair enough. Using includes = [ "identifiers", "*.instagram_uid"] would be correct in that case.

[bleskes]

Hi Rob,

I've implemented a solution that is not exactly what you ask for but I feel it's simpler to reason & understand how it works. Effectively it makes excludes maintain document structure above the point of exclusion. You can see it here: #4715

I used part of the tests you wrote as a basis so I included your name in the commit to give you credit for the work.

Can you please check if you still feel this case is needed and if not close it?

Thx,
Boaz

[RobCherry]

Ignore that commit for now, it is not actually testing what I wanted to test. I will update it tomorrow.

[RobCherry]

@bleskes

What you're describing as the proposed change

maintain document structure above the point of exclusion

seems like a great solution.

I was originally going to add some additional tests to ensure that document structure is maintained both with and without includes, but then I noticed that testNotOmittingObjectWithNestedExcludedObject was covering most of what I wanted.

[bleskes]

@RobCherry happy you like it. Can we close this issue then?

[RobCherry]

Yes

[bleskes]

cool. Once again, thx for your help?.