[FLINK-30637] [Python] Try to replace 'is' with '==' to ensure matching the window type of overwindow correctly #21649
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Description: In linux-arch64 environment, “window_type is OverWindow.ROW_UNBOUNDED_FOLLOWING” in the PandasBatchOverWindowAggregateFunctionOperation class of the pyflink source code has returned the wrong result. For example, when window_type is 6, it represents the window type of ‘ROW_UNBOUNDED_FOLLOWING’, but “window_type is OverWindow.ROW_UNBOUNDED_FOLLOWING” return false because the memory address of window_type has changed. It will lead to the wrong type of window, such as row sliding window, so, the wrong results of pandas udf have appeared. Solutions： Since the window selections need to compare the values of two integer variables, I recommend replacing ‘is’ with ‘==’. And it has been verified to perform correctly on both x86 and aarch64 environments.
YasuoStudyJava
changed the title
Collaborator
HuangXingBo
approved these changes
Jan 12, 2023
Contributor
HuangXingBo
left a comment
HuangXingBo
left a comment
There was a problem hiding this comment.
Thanks a lot for the fix.
HuangXingBo
pushed a commit
that referenced
this pull request
Jan 17, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes #21649.
HuangXingBo
pushed a commit
that referenced
this pull request
Jan 17, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes #21649.
chucheng92
pushed a commit
to chucheng92/flink
that referenced
this pull request
Feb 3, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes apache#21649.
sergeitsar
pushed a commit
to fentik/flink
that referenced
this pull request
Feb 8, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes apache#21649.
sergeitsar
pushed a commit
to fentik/flink
that referenced
this pull request
Feb 8, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes apache#21649.
akkinenivijay
pushed a commit
to krisnaru/flink
that referenced
this pull request
Feb 11, 2023
…ssues with OverWindow Aggregation On Linux-arch64 This closes apache#21649.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
In linux-arch64 environment, “window_type is OverWindow.ROW_UNBOUNDED_FOLLOWING” in the PandasBatchOverWindowAggregateFunctionOperation class of the pyflink source code has returned the wrong result. For example, when window_type is 6, it represents the window type of ‘ROW_UNBOUNDED_FOLLOWING’, but “window_type is OverWindow.ROW_UNBOUNDED_FOLLOWING” return false because the memory address of window_type has changed. It will lead to the wrong type of window, such as row sliding window, so, the wrong results of pandas udf have appeared.
Solutions：
Since the window selections need to compare the values of two integer variables, I recommend replacing ‘is’ with ‘==’. That can also prevents erroneous results caused by Python small integer object pool failure. And it has been verified to perform correctly on both x86 and aarch64 environments.
Related Jira Issue: FLINK-30637 (https://issues.apache.org/jira/browse/FLINK-30637)