Fix build warnings

[carlyeks]

There were a few build errors that have been collecting. I've fixed them, and also made warnings fatal so that they are addressed earlier.

All of the changes were because of deprecation in an imported library; most were related to Hadoop's Job constructor being deprecated. The biggest change was in Genotype.getAttributeAsString. I don't know why it was deprecated, but I've included a scala version of it.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM/102/

[massie]

Nice flag addition. It will definitely force us to address warnings instead of ignore them. :)

Please indent the arg at the same level as the others.

[massie]

Aside from the indentation nit, this looks ready to merge to me.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM/105/

[massie]

Thanks, Carl!

[fnothaft]

FYI - this breaks the build for those using Hadoop 1.0.4 (the default hadoop version for the Spark EC2 scripts).

[INFO] Compiling 52 Scala sources to /Users/fnothaft/adam-new/adam/adam-commands/target/scala-2.9.3/classes...
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/commands/AdamCommand.scala:47: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance()
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamContext.scala:106: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance(sc.hadoopConfiguration)
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamContext.scala:116: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance(sc.hadoopConfiguration)
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamContext.scala:210: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance(sc.hadoopConfiguration)
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamRDDFunctions.scala:41: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance(rdd.context.hadoopConfiguration)
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/util/ParquetFileTraversable.scala:34: value isDirectory is not a member of org.apache.hadoop.fs.FileStatus
[ERROR] if (status.isDirectory) {
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/util/ParquetFileTraversable.scala:42: value isFile is not a member of org.apache.hadoop.fs.FileStatus
[ERROR] } else if (status.isFile) {
[ERROR] ^
[ERROR] /Users/fnothaft/adam-new/adam/adam-commands/src/main/scala/edu/berkeley/cs/amplab/adam/util/PileupTraversable.scala:86: value getInstance is not a member of object org.apache.hadoop.mapreduce.Job
[ERROR] val job = Job.getInstance()
[ERROR] ^
[ERROR] 8 errors found

[fnothaft]

This also does not run out of the box with the spark-ec2 scripts with hadoop-major-version=2. Digging further...

[tdanford]

So, two things -- is there a more automated way for us to test in this kind of environment?

Carl and I are in a meeting at the moment, but it sounds like he'll dig into this when we're done here. Frank, what's your preference, should we revert? Or fix it in some other way?

[fnothaft]

Long term, I think that we need to figure out what versions of all tools we need to support. From there, we'll probably set up matrix builds in Jenkins to test this out.

I wouldn't fret from your side right now — I'm going through hadoop versions trying to sort out which version works. There's nothing really to do until then. Realistically, I think this problem will be resolved through documentation and testing, not a code patch.

TL;DR: Jenkins and documentation.

[massie]

We can protect against this by updating our Jenkins job. I'll do that soon (possibly today).

[fnothaft]

It seems that Spark requires hadoop-client:2.0.0-cdh4.2.0. The Maven jar for this does not still appear to be available: http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client

Thoughts? I assume this is a rough patch that is going to be fixed by spark-0.8.1?

[fnothaft]

I've reverted this change in master.