[SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule

[dongjoon-hyun]

What changes were proposed in this pull request?

After SPARK-6990, dev/lint-java keeps Java code healthy and helps PR review by saving much time.
This issue aims remove unused imports from Java/Scala code and add UnusedImports checkstyle rule to help developers.

How was this patch tested?

./dev/lint-java
./build/sbt compile

[srowen]

I'm OK with this. Can we enforce this in Scala too?

Since we've bothered to fail the build if imports aren't in the right order I think it's reasonable to enforce removal of unused imports everywhere too.

@dongjoon-hyun I think you could add that too if you feel so inclined. If you're busy I might add it to https://issues.apache.org/jira/browse/SPARK-13423 but I like your focused PR here.

[srowen]

Jenkins, test this please

[dongjoon-hyun]

Thank you, @srowen !
Up to know, I'm not sure about enforcing Scala Imports but I'll check that, too.
I'll leave a message soon.

By the way, your SPARK-13423 looks great.

[dongjoon-hyun]

Hi, @srowen .
Surprisingly, according to the ScalaStyle Ruleset , it seems that we can not do that for Scala as of today. I think it's strange, do you think is there any other ways to do that?

[zsxwing]

FYI because mvn checkstyle:check depends on mvn install which cost huge time for sbt, java style checker is disabled now: https://github.com/apache/spark/blob/master/dev/run-tests.py#L546

[dongjoon-hyun]

Oh, thank you for informing that, @zsxwing .

By the way, dev\lint-java does not depend on mvn install. I think It depends on compiled classes.
You can do that by the following. How do you think of that, @zsxwing ? Is there any way to re-enable that?

$ ./build/sbt compile
$ # Add "<module name="UnusedImports"/>" into checkstyle.xml.
$ time ./dev/lint-java
...
real    0m14.880s
user    0m35.663s
sys 0m1.741s

[dongjoon-hyun]

@srowen , for the scala code, should we take advantage of -Ywarn-unused-import option for that in some way?

[zsxwing]

@dongjoon-hyun I mean lint-java will fail on a clean machine without maven cache. It will fail to resolve the snapshot jars which don't exist before running mvn install.

[dongjoon-hyun]

Oh, I see. Then, this PR will not enforce automatically, just helps developers to check before pushing PR.

Thank you, @zsxwing !

[dongjoon-hyun]

Hi, @srowen .
I've tested more, but found that -Ywarn-unused-import option is also not reliable yet in Scala 2.11.7.
Please refer https://issues.scala-lang.org/browse/SI-9616 .
I think I can not handle Scala part in this PR.
Sorry about that.

[SparkQA]

Test build #52203 has finished for PR 11438 at commit 5e82490.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

It seems that Jenkins fails due to irrelevant things like the following.

Error instrumenting class:org.apache.spark.mllib.regression.IsotonicRegressionModel$SaveLoadV1_0$
...

Other PRs' test fail with similar logs. Should we wait for a while and re-trigger to test?

[dongjoon-hyun]

Rebased to trigger the Jenkins test.

[srowen]

If you're willing, I think you're welcome to also remove unused Scala imports, even if you can't enforce it. IDEs can make this easy.

Yes the test failure looks unrelated.

[srowen]

Jenkins, retest this please

[SparkQA]

Test build #52233 has finished for PR 11438 at commit 93b50e2.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you for the testing, @srowen . Unfortunately, it still fails.
For the Scala Unused Imports, I see what you mean.
Then, I'll update Jira issue and remove them in this PR, too.
Thank you again.

[zsxwing]

ok to test

[dongjoon-hyun]

Thank you, @zsxwing !

[SparkQA]

Test build #52254 has finished for PR 11438 at commit 93b50e2.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @srowen .
The unused imports in Scala code are removed, too.

[dongjoon-hyun]

I've compared the Jenkins logs. My PR hangs at the following test.

[info] WithAggregationKinesisBackedBlockRDDSuite:
Using endpoint URL https://kinesis.us-west-2.amazonaws.com for creating Kinesis streams for tests.
[2016-03-01 13:06:10.815940] [0x00007fbb51998700] [info] [kinesis_producer.cc:79] Created pipeline for stream 

Other PR passing this test looks like the following.

[info] WithAggregationKinesisBackedBlockRDDSuite:
[info] - Basic reading from Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available in both block manager and Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!

Up to now, I cannot figure out how to make this PR pass the Jenkins. Could you give me some clue?

[srowen]

Yeah I think it's worth biting the bullet and cleaning up imports in one big go for 2.0.
The failures are certainly spurious, let's try again.

[SparkQA]

Test build #52313 has finished for PR 11438 at commit 627d561.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52310 has finished for PR 11438 at commit 90d0241.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you. I see. I rebased to test one more time.

[SparkQA]

Test build #52323 has finished for PR 11438 at commit 1938445.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52333 has finished for PR 11438 at commit f85d167.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52340 has finished for PR 11438 at commit 4ecb8d7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

It passes the test finally. I updated the title and description of this PR and JIRA.
To sum up,

• Remove all detected unused imports in Java (except one line in JavaKinesisStreamSuite.java)
• Remove all observed unused imports in Scala (by manually with IntelliJ)
• Add UnusedImports rule to checkstyle.xml

Thanks to your advice and help, I can finish this PR.
Thank you, @srowen and @zsxwing .

[srowen]

Merged to master

[dongjoon-hyun]

Thank you for merging!