GEODE-9004: Fix issues in queries targeting a Map field by albertogpz · Pull Request #6200 · apache/geode

albertogpz · 2021-03-26T14:56:11Z

Queries in which map fields are involved
in the condition, sometimes do not return the same entries
if map indexes are used, compared to when map indexes
are not used.

Differences were found when the condition on the
map field was of type '!=' and also when the condition
on the map field was comparing the value with null.

Example1:
Running a query with the following condition:
positions['SUN'] = null

in a region with the following entries:
entry1.positions=null
entry2.positions={'a'=>'b'}
entry3.positions={'SUN'=>null}

will return entry1 and entry3 if there are no
indexes defined.
will return no entries if the following index is defined:
positions['SUN','c']
will return entry3 if the following index is defined:
positions[*]

Example2:
Running a query with the following condition:
positions['SUN'] != '3'

in a region with the following entries:
entry1.positions=null
entry2.positions={'a'=>'b'}
entry3.positions={'SUN'=>'4'}
entry4.positions={'SUN'=>'3'}
entry5.positions={'SUN'=>null}

will return all entries except for entry4 if:
there are no indexes defined.
will return entry3 if the following index is defined:
positions['SUN','c']
will return entry3 and entry5 if the following index is defined:
positions[*]

In order to have the same results for these
queries no matter if indexes are used,
the following changes have been made:

When running queries, the result of getting the
value from a Map that does not contain the key used
to access the value will be
considered UNDEFINED instead of null.
This will imply that queries where the condition is
positions['SUN'] = null will not return entries
that do not have the 'SUN' key in the positions Map.
Queries with map indexes will not be able to use
the indexes if the query is of type '!=' against
any of the indexed map fields.
Example:
for index positions['SUN', 'ERIC'] or
positions[*]
the following query will not use indexes:
positions['SUN'] != '44'

These changes must be documented accordingly
in the corresponding indexes section.

Thank you for submitting a contribution to Apache Geode.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
Has your PR been rebased against the latest commit within the target branch (typically develop)?
Is your initial contribution a single, squashed commit?
Does gradlew build run cleanly?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:

Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

Queries in which map fields are involved in the condition, sometimes do not return the same entries if map indexes are used, compared to when map indexes are not used. Diferences were found when the condition on the map field was of type '!=' and also when the condition on the map field was comparing the value with null. Example1: Running a query with the following condition: positions['SUN'] = null in a region with the following entries: entry1.positions=null entry2.positions={'a'=>'b'} entry3.positions={'SUN'=>null} - will return entry1 and entry3 if there are no indexes defined. - will return no entries if the following index is defined: positions['SUN','c'] - will return entry3 if the following index is defined: positions[*] Example2: Running a query with the following condition: positions['SUN'] != '3' in a region with the following entries: entry1.positions=null entry2.positions={'a'=>'b'} entry3.positions={'SUN'=>'4'} entry4.positions={'SUN'=>'3'} entry5.positions={'SUN'=>null} - will return all entries except for entry4 if: there are no indexes defined. - will return entry3 if the following index is defined: positions['SUN','c'] - will return entry3 and entry5 if the following index is defined: positions[*] In order to have the same results for these queries no matter if indexes are used, the following changes have been made: - When running queries, the result of getting the value from a Map that does not contain the key used to access the value will be considered UNDEFINED instead of null. This will imply that queries where the condition is positions['SUN'] = null will not return entries that do not have the 'SUN' key in the positions Map. - Queries with map indexes will not be able to use the indexes if the query is of type '!=' against any of the indexed map fields. Example: for index positions['SUN', 'ERIC'] or positions[*] the following query will not use indexes: positions['SUN'] != '44' These changes must be documented accordingly in the corresponding indexes section.

DonalEvans · 2021-03-26T17:54:46Z

The PR description says that "These changes must be documented accordingly in the corresponding indexes section." Is there a GEODE ticket for this docs work? If not, one should be created, or the docs changes included as part of this PR.

albertogpz · 2021-03-26T18:09:16Z

The PR description says that "These changes must be documented accordingly in the corresponding indexes section." Is there a GEODE ticket for this docs work? If not, one should be created, or the docs changes included as part of this PR.

I agree, Donal. My plan is to update the documentation as part of this PR if the solution is accepted. Given that the outcome of some queries without indexes change and that indexes will not be used with queries using "!=" with Map indexes, I want to have first the blessing from the community and only then update the documentation.

DonalEvans · 2021-03-26T18:26:37Z

I'm not a query expert, so I don't understand what the reasoning is behind map indexes no longer being used for queries using !=. Would it be possible to get an explanation of the motivation behind this change?

albertogpz · 2021-03-26T18:35:25Z

I'm not a query expert, so I don't understand what the reasoning is behind map indexes no longer being used for queries using !=. Would it be possible to get an explanation of the motivation behind this change?

With the current implementation, map indexes are used (if defined) in queries that have a condition using "!=" against a value in the map field indexed. The problem is that these queries offer different outcome if the index is defined vs when the index is not defined.
In this PR I have made the queries offer the same output no matter if the index is defined or not but at the expense of not using the index if the condition is "!=" when the index is defined. I have not been able to come up with a solution without losing this "feature". That's the reason behind "map indexes no longer being used for queries using !=."

DonalEvans

A few requests for code clean-up/clarifying comments. The content of the changes looks okay to me, but the conceptual changes need someone with a better understanding of the expected behaviour of queries and indexes.

DonalEvans · 2021-03-26T20:14:27Z