Response filtering using negative filter_path is removing too many fields by skipping empty arrays

[wylieconlon]

Elasticsearch version 8.0 snapshot

Description of the problem including expected versus actual behavior:

I am trying to reduce the JSON payload from ES to Kibana by up to 50%. One part of this is that I'm trying to use a negative filter_path parameter to exclude one particular path, -**.key_as_string. The bug is that the response filtering is not as targeted as it should be, and it's removing other empty fields.

Steps to reproduce:

This reproduction works for any index, and it will exclude the hits: [] response property which I was expecting to have.

a. Query showing full response

curl -u elastic -X POST http://localhost:9200/kibana_sample_data_logs/_search -H 'Content-Type: application/json' -d '{
  "size": 0
}'

{"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null,"hits":[]}}

b. Same query with response filtering

curl -u elastic -X POST http://localhost:9200/kibana_sample_data_logs/_search?filter_path=**,-**.key_as_string -H 'Content-Type: application/json' -d '{
  "size": 0
}'

{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null}}

Notice that the hits: [] is no longer returned.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/REST API)

[javanna]

I briefly looked into this. Response filtering is known not to preserve empty objects and arrays, unfortunately, since it is supported in Elasticsearch. I also find it counter intuitive. The reason for this is that this is how the library that we use to parse json (and other formats) implements filtering: only when there is a value, it checks if the property needs to be added.

[wylieconlon]

@javanna Thanks for the context. Are you saying we should document the existing behavior instead of changing it?

[javanna]

@wylieconlon not sure yet, I don't think my quick analysis is enough to get to a conclusion, but I wanted to point out that this may seem an easy bug to fix at first glance but it is not in practice. If all you'd need response filtering for is trimming down aggregations response is #63898 , which seems doable and also cleaner.

[nik9000]

@jaymode pinged me on this and I was already looking at xcontent so I had to trace through the code. The behavior is a little more precisely described as "if a filter could match an empty object or array it will be removed". The code looks very keen to remove the whole path to something if the whole thing is filtered. We're all used to this. It just that the filter can't tell the difference between when the whole result was filtered out and when it received an empty string or array. I opened FasterXML/jackson-core#715 to talk about this over there.

[nik9000]

We ended up merging support to jackson-core to opt into or out of this behavior on a very granular level. So, once Jackson 2.14 is released, we can upgrade and fix this issue on our side. Now, we're blocked on upgrading jackson because of Azure, but this is another thing waiting on it....