Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-4182][SQL] Fixes ColumnStats classes for boolean, binary and complex data types #3059

Closed
wants to merge 2 commits into from

Conversation

liancheng
Copy link
Contributor

NoopColumnStats was once used for binary, boolean and complex data types. This ColumnStats doesn't return properly shaped column statistics and causes caching failure if a table contains columns of the aforementioned types.

This PR adds BooleanColumnStats, BinaryColumnStats and GenericColumnStats, used for boolean, binary and all complex data types respectively. In addition, NoopColumnStats returns properly shaped column statistics containing null count and row count, but this class is now used for testing purpose only.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22734 has started for PR 3059 at commit 38cf5b5.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22736 has started for PR 3059 at commit fb3ee85.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22734 has finished for PR 3059 at commit 38cf5b5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22734/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22736 has finished for PR 3059 at commit fb3ee85.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22736/
Test FAILed.

@liancheng
Copy link
Contributor Author

Hm, the above test failure is legitimate. However, the test failure is sensitive to execution order, namely, it only fails when some test case had cached the test table src previously.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22743 has started for PR 3059 at commit b398cfd.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 2, 2014

Test build #22743 has finished for PR 3059 at commit b398cfd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22743/
Test PASSed.

@marmbrus
Copy link
Contributor

marmbrus commented Nov 2, 2014

Thanks! Merged to master.

@asfgit asfgit closed this in e4b8089 Nov 2, 2014
@liancheng liancheng deleted the spark-4182 branch November 3, 2014 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants