[SPARK-9969][YARN] Remove old MR classpath API support

[jerryshao]

Here propose to remove old MRJobConfig#DEFAULT_APPLICATION_CLASSPATH support, since we now move to Yarn stable API.

@vanzin and @sryza , any opinion on this? If we still want to support old API, I can close it. But as far as I know now major Hadoop releases has moved to stable API.

[srowen]

This would prevent Hadoop 1.x from working, right? or would it?
I think that's still required, although I would also advocate for dropping 1.x support.

[jerryshao]

I think it is Spark on Yarn specific code, will not impact hadoop 1.x.

[jerryshao]

Since Hadoop 1.x will not even compile Yarn related module.

[SparkQA]

Test build #40855 timed out for PR 8192 at commit 07a4e4f after a configured wait of 175m.

[vanzin]

The change itself LGTM, but I wonder whether this code is needed at all. Shouldn't the YARN classpath be sufficient?

[SparkQA]

Test build #1619 has finished for PR 8192 at commit 07a4e4f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

@vanzin, not specifically sure about why need MR default classpath, I guess we still need some APIs for HadoopRDD related codes.

[jerryshao]

Jenkins, retest this please.

[SparkQA]

Test build #41005 timed out for PR 8192 at commit 07a4e4f after a configured wait of 175m.

[sryza]

Yeah, it's needed for the InputFormats that Spark relies on to read Hadoop data.

[vanzin]

Let's try once again. retest this please

[SparkQA]

Test build #41170 timed out for PR 8192 at commit 07a4e4f after a configured wait of 175m.

[vanzin]

All the yarn tests ran and passed, so I'm gonna merge this.

[vanzin]

Merged to master, thanks!

[jerryshao]

Thanks @vanzin :)