Skip to content

Commit

Permalink
[SPARK-34115][CORE] Check SPARK_TESTING as lazy val to avoid slowdown
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Check SPARK_TESTING as lazy val to avoid slow down when there are many environment variables

### Why are the changes needed?
If there are many environment variables, sys.env slows is very slow. As Utils.isTesting is called very often during Dataframe-Optimization, this can slow down evaluation very much.

An example for triggering the problem can be found in the bug ticket https://issues.apache.org/jira/browse/SPARK-34115

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
With the example provided in the ticket.

Closes apache#31244 from nob13/bug/34115.

Lead-authored-by: Norbert Schultz <norbert.schultz@reactivecore.de>
Co-authored-by: Norbert Schultz <noschultz@gmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
(cherry picked from commit c3d8352)
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
2 people authored and HyukjinKwon committed Jan 20, 2021
1 parent 5a93bcb commit b5b1da9
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/util/Utils.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1879,7 +1879,9 @@ private[spark] object Utils extends Logging {
* Indicates whether Spark is currently running unit tests.
*/
def isTesting: Boolean = {
sys.env.contains("SPARK_TESTING") || sys.props.contains(IS_TESTING.key)
// Scala's `sys.env` creates a ton of garbage by constructing Scala immutable maps, so
// we directly use the Java APIs instead.
System.getenv("SPARK_TESTING") != null || System.getProperty(IS_TESTING.key) != null
}

/**
Expand Down

0 comments on commit b5b1da9

Please sign in to comment.