-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-23222][SQL] Make DataFrameRangeSuite not flaky #20431
Conversation
does this significantly increase the test runtime? |
I didn't see significant difference. It is about 450~500 milliseconds. |
Test build #86796 has finished for PR 20431 at commit
|
retest this please. |
@@ -154,7 +154,7 @@ class DataFrameRangeSuite extends QueryTest with SharedSQLContext with Eventuall | |||
test("Cancelling stage in a query with Range.") { | |||
val listener = new SparkListener { | |||
override def onJobStart(jobStart: SparkListenerJobStart): Unit = { | |||
eventually(timeout(10.seconds)) { | |||
eventually(timeout(10.seconds), interval(1.millis)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default interval is 15millis, IIUC. It is more possibly that the range stage finishes in the interval. So reduce it to 1millis.
Test build #86811 has finished for PR 20431 at commit
|
thanks, merging to master/2.3! |
## What changes were proposed in this pull request? It is reported that the test `Cancelling stage in a query with Range` in `DataFrameRangeSuite` fails a few times in unrelated PRs. I personally also saw it too in my PR. This test is not very flaky actually but only fails occasionally. Based on how the test works, I guess that is because `range` finishes before the listener calls `cancelStage`. I increase the range number from `1000000000L` to `100000000000L` and count the range in one partition. I also reduce the `interval` of checking stage id. Hopefully it can make the test not flaky anymore. ## How was this patch tested? The modified tests. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #20431 from viirya/SPARK-23222. (cherry picked from commit 84bcf9d) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
It is reported that the test
Cancelling stage in a query with Range
inDataFrameRangeSuite
fails a few times in unrelated PRs. I personally also saw it too in my PR.This test is not very flaky actually but only fails occasionally. Based on how the test works, I guess that is because
range
finishes before the listener callscancelStage
.I increase the range number from
1000000000L
to100000000000L
and count the range in one partition. I also reduce theinterval
of checking stage id. Hopefully it can make the test not flaky anymore.How was this patch tested?
The modified tests.