Changed localProperties to use ThreadLocal (not DynamicVariable).

[kayousterhout]

The fact that DynamicVariable uses an InheritableThreadLocal
can cause problems where the the localProperties variable is
shared across threads; Spark should use a TheadLocal instead. The
bug only occurs when someone runs a Spark job in a thread,
and then runs many concurrent Spark jobs in child threads.
Here's how the problem can occur:

sc = new SparkContext(...)
sc.setJobDescription("Foo")

(1 to 2).map { i ->
new Thread() {
override def run() {
sc.setJobDescription("Job " + i)
Thread.sleep(100)
sc.makeRdd(.....)
}
}

In this code, both jobs will end up with the same description.

On the first call to setJobDescription(), SparkContext.setLocalProperty() will find that localProperties (a Dynamic variable) is null -- so it will initialize it, and then set the job description property to foo.

When each of the two new threads is created, the value of localProperties is inherited from the value in the parent thread, which is an InheritableThreadLocal. The way that the InheritableThreadLocal childValue() function works is that it just returns a reference to the parent value (http://docs.oracle.com/javase/6/docs/api/java/lang/InheritableThreadLocal.html#childValue(T)) -- so now the localProperties variable in both child threads refers to the same underlying Properties.

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/920/

[rxin]

@pwendell

[pwendell]

@rxin - what's up...

[mateiz]

This looks good to me. Good catch, Kay!

[pwendell]

Thanks @kayousterhout.