[SPARK-5096] Use sbt tasks instead of vals to get hadoop version

[marmbrus]

This makes it possible to compile spark as an external ProjectRef where as now we throw a FileNotFoundException

[SparkQA]

Test build #25089 has started for PR 3905 at commit fd63aae.

• This patch merges cleanly.

[marmbrus]

/cc @andrewor14

I'm throwing away the special casing for network-yarn/assembly... but even on master:

> network-yarn/assembly
[error] Expected ID character
[error] Not a valid command: network-yarn
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Not a valid key: network-yarn
[error] network-yarn/assembly

Is this okay?

[SparkQA]

Test build #25089 has finished for PR 3905 at commit fd63aae.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25089/
Test PASSed.

[andrewor14]

Hm I think we need to keep the network yarn jar called the same as before in both SBT and maven. Other than that I believe it's ok to not support network-yarn/assembly.

[pwendell]

I thin this is good for now, I can pull it in.