[SPARK-22363][SQL][TEST] Add unit test for Window spilling#20022
[SPARK-22363][SQL][TEST] Add unit test for Window spilling#20022gaborgsomogyi wants to merge 2 commits intoapache:masterfrom
Conversation
gaborgsomogyi
changed the title
gaborgsomogyi
changed the title
jiangxb1987
left a comment
jiangxb1987
left a comment
There was a problem hiding this comment.
Looks pretty good, also cc @gatorsmile @cloud-fan
| test("SPARK-21258: complex object in combination with spilling") { | ||
| // Make sure we trigger the spilling path. | ||
| withSQLConf(SQLConf.WINDOW_EXEC_BUFFER_SPILL_THRESHOLD.key -> "17") { | ||
| withSQLConf(SQLConf.WINDOW_EXEC_BUFFER_IN_MEMORY_THRESHOLD.key -> "0", |
There was a problem hiding this comment.
Why should we set this value to 0?
There was a problem hiding this comment.
WINDOW_EXEC_BUFFER_IN_MEMORY_THRESHOLD drives how much items is guaranteed to kept in memory. If this limit is not hit spilling not considered.
There was a problem hiding this comment.
Yeah, i mean, how about set it to 1 instead of 0?
There was a problem hiding this comment.
Ahh, now I see 🙂 Sure, I'll set it soon.
There was a problem hiding this comment.
We can accept any value. No limit. both are fine.
|
OK to test |
|
ok to test |
|
Test build #85540 has finished for PR 20022 at commit
|
|
retest this please |
|
Test build #85547 has finished for PR 20022 at commit
|
| df.select($"key", sum("value").over(window)).collect() | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Normally, we will create a helper function for avoiding the duplicate codes. Since the test cases are pretty small, it is also fine.
|
LGTM |
|
Thanks! Merged to master. |
|
@gatorsmile @cloud-fan @jiangxb1987 Thanks for the help! |
5 participants
What changes were proposed in this pull request?
There is already test using window spilling, but the test coverage is not ideal.
In this PR the already existing test was fixed and additional cases added.
How was this patch tested?
Automated: Pass the Jenkins.