Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix][Spark]: Add default value of table-filters in spark scan builder to avoid NPE when comparing the table scan. #2313

Merged
merged 2 commits into from
Nov 17, 2023

Conversation

baiyangtx
Copy link
Contributor

Why are the changes needed?

The NPE may be throwed when spark comparing the table scan during plan.

Brief change log

  • Add default value to SparkScanBuilder

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? ( no)

@github-actions github-actions bot added the module:mixed-spark Spark module for Mixed Format label Nov 15, 2023
@baiyangtx baiyangtx marked this pull request as ready for review November 15, 2023 11:13
Copy link

codecov bot commented Nov 15, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a87a2a9) 52.68% compared to head (623563a) 52.94%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2313      +/-   ##
============================================
+ Coverage     52.68%   52.94%   +0.25%     
+ Complexity     4208     3612     -596     
============================================
  Files           511      465      -46     
  Lines         29263    24502    -4761     
  Branches       2860     2340     -520     
============================================
- Hits          15418    12973    -2445     
+ Misses        12592    10510    -2082     
+ Partials       1253     1019     -234     
Flag Coverage Δ
core 52.94% <ø> (-0.02%) ⬇️
trino ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wangtaohz
Copy link
Contributor

Can we check the com.netease.arctic.spark.reader.KeyedSparkBatchScan#filterExpressions should not be null in the constructor? And then we can remove all the filterExpressions != null before using it.

@baiyangtx
Copy link
Contributor Author

Can we check the com.netease.arctic.spark.reader.KeyedSparkBatchScan#filterExpressions should not be null in the constructor? And then we can remove all the filterExpressions != null before using it.

Fixed

@baiyangtx baiyangtx changed the title [Fix][Spark]: Add default value of table-filters in spark scan builder to avoid NPE when comparing the table scan. [BugFix][Spark]: Add default value of table-filters in spark scan builder to avoid NPE when comparing the table scan. Nov 17, 2023
Copy link
Contributor

@wangtaohz wangtaohz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangtaohz wangtaohz merged commit 84dd431 into apache:master Nov 17, 2023
5 checks passed
@baiyangtx baiyangtx deleted the fix-npe-when-spark-scan branch November 17, 2023 03:28
@zhoujinsong zhoujinsong mentioned this pull request Dec 19, 2023
33 tasks
zhoujinsong pushed a commit that referenced this pull request Dec 21, 2023
…lder to avoid NPE when comparing the table scan. (#2313)

* fix npe problem when scan

(cherry picked from commit 84dd431)
Signed-off-by: zhoujinsong <463763777@qq.com>
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…lder to avoid NPE when comparing the table scan. (apache#2313)

* fix npe problem when scan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:mixed-spark Spark module for Mixed Format
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants