-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-4182][SQL] Fixes ColumnStats classes for boolean, binary and complex data types #3059
Conversation
Test build #22734 has started for PR 3059 at commit
|
Test build #22736 has started for PR 3059 at commit
|
Test build #22734 has finished for PR 3059 at commit
|
Test FAILed. |
Test build #22736 has finished for PR 3059 at commit
|
Test FAILed. |
Hm, the above test failure is legitimate. However, the test failure is sensitive to execution order, namely, it only fails when some test case had cached the test table |
Test build #22743 has started for PR 3059 at commit
|
Test build #22743 has finished for PR 3059 at commit
|
Test PASSed. |
Thanks! Merged to master. |
NoopColumnStats
was once used for binary, boolean and complex data types. ThisColumnStats
doesn't return properly shaped column statistics and causes caching failure if a table contains columns of the aforementioned types.This PR adds
BooleanColumnStats
,BinaryColumnStats
andGenericColumnStats
, used for boolean, binary and all complex data types respectively. In addition,NoopColumnStats
returns properly shaped column statistics containing null count and row count, but this class is now used for testing purpose only.