Skip to content

Commit

Permalink
[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Fix a broadcasted variable leak occurring at each invocation of CostFun in L-BFGS.

## How was this patch tested?

UTests + check that fixed fatal memory consumption on Criteo's use cases.

This contribution is made on behalf of Criteo S.A.
(http://labs.criteo.com/) under the terms of the Apache v2 License.

Author: Anthony Truchet <a.truchet@criteo.com>

Closes #16040 from AnthonyTruchet/SPARK-18612-lbfgs-cost-fun.
  • Loading branch information
Anthony Truchet authored and srowen committed Nov 30, 2016
1 parent fe854f2 commit c5a64d7
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,9 @@ object LBFGS extends Logging {
(grad1, loss1 + loss2)
})

// broadcasted model is not needed anymore
bcW.destroy()

/**
* regVal is sum of weight squares if it's L2 updater;
* for other updater, the same logic is followed.
Expand Down

0 comments on commit c5a64d7

Please sign in to comment.