use Iterator#size in RDD#count

[cloud-fan]

in RDD#count, we used while loop to get the size of Iterator because that Iterator#size used a for loop, which was slightly slower in that version of Scala. But for now, the current version of scala will translate the for loop in Iterator#size into foreach, which uses while loop to iterate the Iterator. So we can use Iterator#size directly now.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

Thanks for submitting this. Do you have a link to the Scala compiler/collection library ticket that impacted this?

[rxin]

Jenkins, test this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14931/

[cloud-fan]

@rxin I'm sorry I didn't got a link for that, but I didn't find any discussion about performance issue of Iterator#size, either. I just checked the source code of Iterator and desugar scala for loop to see what happened.
scala -Xprint:parser -e "var count = 0;for (i <- List(1,2,3).iterator) count += 1"
You can try it yourself, it will be translated into var count = 0;List(1, 2, 3).iterator.foreach( i => count += 1)
And the Iterator#foreach implementation is { while (hasNext) f(next()) }, it's same with what spark do to calculate the size of iterator.
Anyway, get size of an iterator is a basic function, it is logical to let the language lib do it.

[mridulm]

This is not equivalent performance wise from casual look.
Even assuming everything is same, it is still invoking function in loop versus direct addition.

[cloud-fan]

I wrote a simple benchmark to test performance, Iterator#size really sucks... Sorry for my mistake, I'll close this pull request :(