MSQ: Subclass CalciteJoinQueryTest, other supporting changes.#14105
Merged
gianm merged 4 commits intoapache:masterfrom Apr 25, 2023
Merged
MSQ: Subclass CalciteJoinQueryTest, other supporting changes.#14105gianm merged 4 commits intoapache:masterfrom
gianm merged 4 commits intoapache:masterfrom
Conversation
The main change is the new tests: we now subclass CalciteJoinQueryTest in CalciteSelectJoinQueryMSQTest twice, once for Broadcast and once for SortMerge. Two supporting production changes for default-value mode: 1) InputNumberDataSource is marked as concrete, to allow leftFilter to be pushed down to it. 2) In default-value mode, numeric frame field readers can now return nulls. This is necessary when stacking joins on top of joins: nulls must be preserved for semantics that match broadcast joins and native queries. 3) In default-value mode, StringFieldReader.isNull returns true on empty strings in addition to nulls. This is more consistent with the behavior of the selectors, which map empty strings to null as well in that mode. As an effect of change (2), the InsertTimeNull change from apache#14020 (to replace null timestamps with default timestamps) is reverted. IMO, this is fine, as either behavior is defensible, and the change from apache#14020 hasn't been released yet.
gianm
added
the
Area - MSQ
cryptoe
reviewed
Apr 25, 2023
| /** | ||
| * Runs {@link CalciteJoinQueryTest} but with MSQ engine. | ||
| */ | ||
| @RunWith(Enclosed.class) |
| + "FROM foo LEFT JOIN lookup.lookyloo ON foo.dim2 = lookyloo.k\n" | ||
| + "WHERE lookyloo.v <> 'xa' OR lookyloo.v IS NULL\n" | ||
| + "GROUP BY lookyloo.v", | ||
| + "FROM foo LEFT JOIN lookup.lookyloo ON foo.dim2 = lookyloo.k\n" |
Contributor
There was a problem hiding this comment.
There are lot of formatting changes hence its very easy to skip the actual change.
- Couple of tests are marked notMsqCompatible();
- sortIfSortBased result validation.
Are there more changes in this class?
Contributor
Author
There was a problem hiding this comment.
Ah, yeah, that was the result of applying formatting in IntelliJ. I guess the last time this file was updated, the formatting wasn't applied. To help with reading the diff, Github has this "hide whitespace" feature for PR diffs. If you turn that on it looks like this:
The substantive changes are indeed adding various notMsqCompatible and sortIfSortBased.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The main change is the new tests: we now subclass CalciteJoinQueryTest in CalciteSelectJoinQueryMSQTest twice, once for Broadcast and once for SortMerge.
Two supporting production changes for default-value mode:
InputNumberDataSource is marked as concrete, to allow leftFilter to
be pushed down to it.
In default-value mode, numeric frame field readers can now return nulls.
This is necessary when stacking joins on top of joins: nulls must be
preserved for semantics that match broadcast joins and native queries.
In default-value mode, StringFieldReader.isNull returns true on empty
strings in addition to nulls. This is more consistent with the behavior
of the selectors, which map empty strings to null as well in that mode.
As an effect of change (2), the InsertTimeNull change from #14020 (to replace null timestamps with default timestamps) is reverted. IMO, this is fine, as either behavior is defensible, and the change from #14020 hasn't been released yet.