New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-10422] [SQL] String column in InMemoryColumnarCache needs to override clone method #8578
Conversation
👍 |
Test build #41944 has finished for PR 8578 at commit
|
ctx.range(1, 30000).selectExpr("id % 500 as id").rdd.map(id => Tuple1(s"str_$id")).toDF("i") | ||
val cached = df.cache() | ||
// count triggers the caching action. It should not throw. | ||
cached.count() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we check the results?
|
||
test("SPARK-10422: String column in InMemoryColumnarCache needs to override clone method") { | ||
val df = | ||
ctx.range(1, 100).selectExpr("id % 10 as id").rdd.map(id => Tuple1(s"str_$id")).toDF("i") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reduced the number of rows to make this test faster. I also checked that it did fail without the fix.
Test build #1715 has finished for PR 8578 at commit
|
Test build #41952 has finished for PR 8578 at commit
|
Test build #1716 has finished for PR 8578 at commit
|
LGTM, merging this into master 1.5 |
…verride clone method https://issues.apache.org/jira/browse/SPARK-10422 Author: Yin Huai <yhuai@databricks.com> Closes #8578 from yhuai/SPARK-10422. (cherry picked from commit 03f3e91) Signed-off-by: Davies Liu <davies.liu@gmail.com>
…verride clone method https://issues.apache.org/jira/browse/SPARK-10422 Author: Yin Huai <yhuai@databricks.com> Closes apache#8578 from yhuai/SPARK-10422. (cherry picked from commit 03f3e91) Signed-off-by: Davies Liu <davies.liu@gmail.com> (cherry picked from commit 2fce5d8)
https://issues.apache.org/jira/browse/SPARK-10422