Skip to content

Commit

Permalink
[SPARK-23019][CORE] Wait until SparkContext.stop() finished in SparkL…
Browse files Browse the repository at this point in the history
…auncherSuite

## What changes were proposed in this pull request?
In current code ,the function `waitFor` call https://github.com/apache/spark/blob/cfcd746689c2b84824745fa6d327ffb584c7a17d/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java#L155 only wait until DAGScheduler is stopped, while SparkContext.clearActiveContext may not be called yet.
https://github.com/apache/spark/blob/1c9f95cb771ac78775a77edd1abfeb2d8ae2a124/core/src/main/scala/org/apache/spark/SparkContext.scala#L1924

Thus, in the Jenkins test
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-maven-hadoop-2.6/ ,  `JdbcRDDSuite` failed because the previous test `SparkLauncherSuite` exit before SparkContext.stop() is finished.

To repo:
```
$ build/sbt
> project core
> testOnly *SparkLauncherSuite *JavaJdbcRDDSuite
```

To Fix:
Wait for a reasonable amount of time to avoid creating two active SparkContext in JVM in SparkLauncherSuite.
Can' come up with any better solution for now.

## How was this patch tested?

Unit test

Author: Wang Gengliang <ltnwgl@gmail.com>

Closes #20221 from gengliangwang/SPARK-23019.

(cherry picked from commit 344e3aa)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
  • Loading branch information
gengliangwang authored and Marcelo Vanzin committed Jan 10, 2018
1 parent 60d4d79 commit 5b5851c
Showing 1 changed file with 5 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.TimeUnit;

import org.junit.Test;
import static org.junit.Assert.*;
Expand Down Expand Up @@ -133,6 +134,10 @@ public void testInProcessLauncher() throws Exception {
p.put(e.getKey(), e.getValue());
}
System.setProperties(p);
// Here DAGScheduler is stopped, while SparkContext.clearActiveContext may not be called yet.
// Wait for a reasonable amount of time to avoid creating two active SparkContext in JVM.
// See SPARK-23019 and SparkContext.stop() for details.
TimeUnit.MILLISECONDS.sleep(500);
}
}

Expand Down

0 comments on commit 5b5851c

Please sign in to comment.