Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception #5489

Closed
wants to merge 2 commits into from
Closed

SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception #5489

wants to merge 2 commits into from

Conversation

erikvanoosten
Copy link
Contributor

No description provided.

@srowen
Copy link
Member

srowen commented Apr 13, 2015

(Could you start the title with SPARK-xxxx [CORE] ... ? helps the merge messages and PR system. https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark)

srowen
srowen reviewed Apr 13, 2015
View reviewed changes
core/src/test/scala/org/apache/spark/rdd/DoubleRDDSuite.scala

import org.apache.spark._

class DoubleRDDSuite extends FunSuite with SharedSparkContext {
test("sum") {
sc.parallelize(Seq.empty[Double]).sum() should be(0.0 +- 0.0001)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just use assert here as int he rest of the file. Also I think you can assert exact equality in these cases. It really should be 3.0 in the last instance, precisely.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Experience thought me that exact equality with doubles is unreliable. Now I err on the safe side and always use inexact matchers in unit tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubles aren't unstable; 0.0 == 0.0 always. Yes i know what you mean but in these cases we can expect the result to exact to machine precision, always.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't say unstable :)
Anyway, changed as requested.

@srowen
Copy link
Member

srowen commented Apr 13, 2015

ok to test

@erikvanoosten erikvanoosten changed the title Fix for sum on empty RDD fails with exception (SPARK-6878) SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception Apr 13, 2015
@srowen
Copy link
Member

srowen commented Apr 13, 2015

LGTM pending tests, thank you.

@SparkQA
Copy link

SparkQA commented Apr 13, 2015

Test build #30158 has finished for PR 5489 at commit f1708c9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class StringIndexer extends Estimator[StringIndexerModel] with StringIndexerBase
    • class VectorAssembler extends Transformer with HasInputCols with HasOutputCol
    • class VectorIndexer extends Estimator[VectorIndexerModel] with VectorIndexerParams
  • This patch does not change any dependencies.

@SparkQA
Copy link

SparkQA commented Apr 13, 2015

Test build #30162 has finished for PR 5489 at commit 1c91954.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

asfgit pushed a commit that referenced this pull request Apr 14, 2015
@srowen
SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception
899ffdc 
Author: Erik van Oosten <evanoosten@ebay.com>

Closes #5489 from erikvanoosten/master and squashes the following commits:

1c91954 [Erik van Oosten] Rewrote double range matcher to an exact equality assert (SPARK-6878)
f1708c9 [Erik van Oosten] Fix for sum on empty RDD fails with exception (SPARK-6878)

(cherry picked from commit 51b306b)
Signed-off-by: Sean Owen <sowen@cloudera.com>
asfgit pushed a commit that referenced this pull request Apr 14, 2015
@srowen
SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception
2954468 
Author: Erik van Oosten <evanoosten@ebay.com>

Closes #5489 from erikvanoosten/master and squashes the following commits:

1c91954 [Erik van Oosten] Rewrote double range matcher to an exact equality assert (SPARK-6878)
f1708c9 [Erik van Oosten] Fix for sum on empty RDD fails with exception (SPARK-6878)

(cherry picked from commit 51b306b)
Signed-off-by: Sean Owen <sowen@cloudera.com>
@asfgit asfgit closed this in 51b306b Apr 14, 2015
markhamstra pushed a commit to markhamstra/spark that referenced this pull request Apr 15, 2015
@markhamstra
SPARK-6878 [CORE] Fix for sum on empty RDD fails with exception
6d4d912 
Author: Erik van Oosten <evanoosten@ebay.com>

Closes apache#5489 from erikvanoosten/master and squashes the following commits:

1c91954 [Erik van Oosten] Rewrote double range matcher to an exact equality assert (SPARK-6878)
f1708c9 [Erik van Oosten] Fix for sum on empty RDD fails with exception (SPARK-6878)

(cherry picked from commit 51b306b)
Signed-off-by: Sean Owen <sowen@cloudera.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@erikvanoosten @srowen @SparkQA