[SPARK-2585] Remove special handling of Hadoop JobConf

[JoshRosen]

Previously we broadcast JobConf for HadoopRDD because it is large. Now we always broadcast RDDs and task closures so it should no longer be necessary to broadcast the JobConf anymore.

This is a resubmission of @rxin's #1648 (closes #1648).

[SparkQA]

QA tests have started for PR 2683 at commit 1d67d9d.

• This patch merges cleanly.

[JoshRosen]

@rxin I'm not sure that it's safe to remove this synchronization around the JobConf constructor. This synchronization was originally added to address HADOOP-10456, which affects a number of Hadoop versions that we still want to be compatible with.

[SparkQA]

QA tests have finished for PR 2683 at commit 1d67d9d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21350/Test PASSed.

[JoshRosen]

Due to the CONFIGURATION_INSTANTIATION_LOCK thread-safety issue, I think that we'll still end up having to serialize the Configuration separately. If we didn't, then we'd have to hold CONFIGURATION_INSTANTIATION_LOCK while deserializing each task, which could have a huge performance penalty (it's fine to hold the lock while loading the Configuration, since that doesn't take too long).

Therefore, I don't think that we'll be able to safely remove this complexity, so I'm going to recommend closing this PR in favor of applying the small #2684 fix in 1.2.