-
Notifications
You must be signed in to change notification settings - Fork 24.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Include/Exclude Filtering Behavior #4491
Comments
Thanks for opening it. One quick comment about:
This is tricky - in your example it makes sense but the implications are that we'd have to leave all other objects intact - which will potentially result in a lot of empty objects (trees). At least for this case I'd use |
Fair enough. Using |
When excluding '*.f1' from `{ "obj": { "f1": 1, "f2": 2 } }` XContentMapValues.filter returns `{ "obj": { "f2": 2}}`. When run on `{ "obj": { "f1" : 1 }}` we should return `{ "obj": { }}` to maintain object structure. People currently need to always check whether `obj` is there or not. Closes elastic#4715 Closes elastic#4047 Related to elastic#4491
When excluding '*.f1' from `{ "obj": { "f1": 1, "f2": 2 } }` XContentMapValues.filter returns `{ "obj": { "f2": 2}}`. When run on `{ "obj": { "f1" : 1 }}` we should return `{ "obj": { }}` to maintain object structure. People currently need to always check whether `obj` is there or not. Closes #4715 Closes #4047 Related to #4491
Hi Rob, I've implemented a solution that is not exactly what you ask for but I feel it's simpler to reason & understand how it works. Effectively it makes excludes maintain document structure above the point of exclusion. You can see it here: #4715 I used part of the tests you wrote as a basis so I included your name in the commit to give you credit for the work. Can you please check if you still feel this case is needed and if not close it? Thx, |
Ignore that commit for now, it is not actually testing what I wanted to test. I will update it tomorrow. |
What you're describing as the proposed change
seems like a great solution. I was originally going to add some additional tests to ensure that document structure is maintained both with and without includes, but then I noticed that |
@RobCherry happy you like it. Can we close this issue then? |
Yes |
cool. Once again, thx for your help?. |
When excluding '*.f1' from `{ "obj": { "f1": 1, "f2": 2 } }` XContentMapValues.filter returns `{ "obj": { "f2": 2}}`. When run on `{ "obj": { "f1" : 1 }}` we should return `{ "obj": { }}` to maintain object structure. People currently need to always check whether `obj` is there or not. Closes elastic#4715 Closes elastic#4047 Related to elastic#4491
Currently there is a bug in elasticsearch (#4047) where empty objects are not stored in
_source
when an include/exclude list is present.This is because elasticsearch aggressively removes empty objects from the
_source
.For example, if I have an object
the following filters will all result in removing
data
from_source
:excludes = ['data']
,excludes = ['data.*']
,excludes = ['data.key']
,excludes = ['*.key']
,includes = ['data.other']
I believe this behavior is incorrect. I think that we should only remove an object if it is explicitly excluded (
excludes = ['data']
,excludes = ['data.*']
) or if no elements are included (includes = ['other_data.*']). For situations where the object is referenced in an includes list but there is no match, I think the object should remain as an empty object (
includes = ['data.other']`).This use case makes more sense if we are talking about some nested object that is indexed...
Example:
and
excludes = [ "*.ssn"]
would drop the entireidentifiers
object if the only key wasssn
for that object, even if we want the empty identifiers object to remain under all circumstances. We have the same problem withincludes = ["*.instagram_uid"]
.The text was updated successfully, but these errors were encountered: