Please sign in to comment.
[SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map when task ends
## What changes were proposed in this pull request? [SPARK-25144](https://issues.apache.org/jira/browse/SPARK-25144) reports memory leaks on Apache Spark 2.0.2 ~ 2.3.2-RC5. ```scala scala> case class Foo(bar: Option[String]) scala> val ds = List(Foo(Some("bar"))).toDS scala> val result = ds.flatMap(_.bar).distinct scala> result.rdd.isEmpty 18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID = 125 res0: Boolean = false ``` This is a backport of cloud-fan 's #21738 which is a single commit among 3 commits of SPARK-21743. In addition, I added a test case to prevent regressions in branch-2.3 and branch-2.2. Although SPARK-21743 is reverted due to regression, this subpatch can go to branch-2.3 and branch-2.2. This will be merged as cloud-fan 's commit. ## How was this patch tested? Pass the jenkins with a newly added test case. Closes #22150 from dongjoon-hyun/SPARK-25144. Lead-authored-by: Wenchen Fan <email@example.com> Co-authored-by: Dongjoon Hyun <firstname.lastname@example.org> Signed-off-by: hyukjinkwon <email@example.com>
- Loading branch information...
Showing with 35 additions and 15 deletions.
- +12 −5 sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- +1 −1 sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
- +1 −1 sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
- +8 −0 sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- +13 −8 sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala