"$$outer" variable in closures causes objects to unnecessarily remain referenced

[scabug]

When the Scala compiler encounters a closure that references some variables in the scope just outside it, it copies these variables as fields into the closure's class (or uses ObjectRef etc for vars). However, it appears that if the closure happens within another closure, the compiler adds a field called $$outer in the inner closure's class that points to the object for the outer closure. Although this may be good for simplicity or efficiency, it also causes the inner closure to indirectly reference all of the other fields of the outer closure's class, which can have several undesirable side effects:

1. If the inner closure is executed much later than the outer closure, there may be an unexpected memory leak because references to large objects used in the outer closure are kept.
2. If the inner closure is serialized (e.g. to be run on another machine), it "pulls in" all of the variables in the outer closure, potentially leading to a very large serialized object. In addition, serializing the inner closure will just fail if one of the variables in the outer closure is not serializable, even if the inner closure does not use this variable.

As an example, suppose an application contains code like this:

def run() {
  val array = <some large array>
  var x = 7
  for (i <- 1 to 10) {
    <do some stuff with array>
    doLater( () => <do some stuff with x> )
  }
}

Here doLater is a method that takes a closure and executes it later (for example, as an event handler). Alternatively, doLater might attempt to serialize the closure and run it on another machine (this is actually how I ran into this problem - I'm writing a framework for running parallel computations on a cluster by submitting them as closures).

The Scala compiler will generate a class (called App$$$$anonfun$$run$$1) for the body of the for loop, which is what I'm calling the "outer closure". Then, it will also generate a class (called App$$$$anonfun$$run$$1$$$$anonfun$$apply$$1) for the closure passed to doLater (the "inner closure"). In order to let the inner closure access x, the compiler will create a field called x$$1 in the outer closure and set it to x. However, the outer closure also has a second field, array$$1, that refers to array. This causes two problems:

1. A reference to array is kept even after run() finishes. This might lead to a memory leak eventually (e.g. if run is repeatedly called on large arrays, but doLater executes closures much later).
2. Serializing the inner closure also serializes array, potentially defeating the point of sending the serialized closure over the network. Furthermore, if array is not serializable, then it's not even possible to serialize the inner closure, even though the inner closure only uses serializable variables for its computation (like x).

I'm sure that the $$outer reference is a useful optimization in some cases, but because of these issues, would it be possible to at least add an option to prevent the compiler from using $$outer to reference variables outside the scope of a closure? The serialization case is especially important to me because I want to run closures on other machines on the network, but I imagine the memory leak aspect might also lead to unexpected surprises in some high-throughput apps.

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-2617?orig=1
Reporter: Matei Zaharia (matei)

[scabug]

@adriaanm said:
would be too much of a special case