[SPARK-2479][MLlib] Comparing floating-point numbers using relative error in UnitTests #1425

dbtsai · 2014-07-16T01:37:58Z

Floating point math is not exact, and most floating-point numbers end up being slightly imprecise due to rounding errors.

Simple values like 0.1 cannot be precisely represented using binary floating point numbers, and the limited precision of floating point numbers means that slight changes in the order of operations or the precision of intermediates can change the result.

That means that comparing two floats to see if they are equal is usually not what we want. As long as this imprecision stays small, it can usually be ignored.

Based on discussion in the community, we have implemented two different APIs for relative tolerance, and absolute tolerance. It makes sense that test writers should know which one they need depending on their circumstances.

Developers also need to explicitly specify the eps, and there is no default value which will sometimes cause confusion.

When comparing against zero using relative tolerance, a exception will be raised to warn users that it's meaningless.

For relative tolerance, users can now write

assert(23.1 ~== 23.52 relTol 0.02)
assert(23.1 ~== 22.74 relTol 0.02)
assert(23.1 ~= 23.52 relTol 0.02)
assert(23.1 ~= 22.74 relTol 0.02)
assert(!(23.1 !~= 23.52 relTol 0.02))
assert(!(23.1 !~= 22.74 relTol 0.02))

// This will throw exception with the following message.
// "Did not expect 23.1 and 23.52 to be within 0.02 using relative tolerance."
assert(23.1 !~== 23.52 relTol 0.02)

// "Expected 23.1 and 22.34 to be within 0.02 using relative tolerance."
assert(23.1 ~== 22.34 relTol 0.02)

For absolute error,

assert(17.8 ~== 17.99 absTol 0.2)
assert(17.8 ~== 17.61 absTol 0.2)
assert(17.8 ~= 17.99 absTol 0.2)
assert(17.8 ~= 17.61 absTol 0.2)
assert(!(17.8 !~= 17.99 absTol 0.2))
assert(!(17.8 !~= 17.61 absTol 0.2))

// This will throw exception with the following message.
// "Did not expect 17.8 and 17.99 to be within 0.2 using absolute error."
assert(17.8 !~== 17.99 absTol 0.2)

// "Expected 17.8 and 17.59 to be within 0.2 using absolute error."
assert(17.8 ~== 17.59 absTol 0.2)

Authors:
DB Tsai dbtsai@alpinenow.com
Marek Kolodziej marek@alpinenow.com

SparkQA · 2014-07-16T01:42:54Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16698/consoleFull

mengxr · 2014-07-16T08:23:29Z

mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala

-    val weight0 = model.weights(0)
-    assert(weight0 >= -1.60 && weight0 <= -1.40, weight0 + " not in [-1.6, -1.4]")
-    assert(model.intercept >= 1.9 && model.intercept <= 2.1, model.intercept + " not in [1.9, 2.1]")
+    assert(model.weights(0).almostEquals(-1.5244128696247), "weight0 should be -1.5244128696247")


Maybe we should leave some buffer space for the weights. In case we update the implementation, it would be tedious to update the numbers here.

We can have higher relative error here instead. If the implementation is changed, it's also nice to have a test which can catch the slightly different behavior. Also, updating those numbers will not take too much time comparing with the implementation work.

mengxr · 2014-07-16T08:34:29Z

@dbtsai The assertions with === were all tested to work, but I agree it is more robust to allow numerical errors. One downside of this change is that === reports the values in comparison when something is wrong but now almostEquals only returns true/false. It would be great if we can make the implementation similar to ===.

Btw, Scalatest 2.x has this tolerance feature, where you can use +- to indicate a range. We are not using Scalatest 2.x but it is a useful feature.

srowen · 2014-07-16T09:42:16Z

mllib/src/test/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetricsSuite.scala


 class BinaryClassificationMetricsSuite extends FunSuite with LocalSparkContext {
+
+  implicit class SeqDoubleWithAlmostEquals(val x: Seq[Double]) {
+    def almostEquals(y: Seq[Double], eps: Double = 1E-6): Boolean =


1.0e-6 is way bigger than an ulp for a double; 1.0e-12 is more like it. I understand a complex calculation might legitimately vary by significantly more than an ulp depending on the implementation. As @mengxr says where you mean to allow significantly more than machine precision worth of noise, that's probably good to do with an explicitly larger epsilon. But this is certainly a good step forward already.

Yeah, for one ulp, it might be 10e-15. Lots of time, I manually type the numbers or just copy the first couple dights of numbers to save the line space, so that's why I chose 1.0e-6. Thus, I can just type around 7 digits of numbers.

I agree with you that in this case, we may want to explicitly specify with larger epsilon.

dbtsai · 2014-07-16T20:22:57Z

@mengxr Scalatest 2.x has the tolerance feature, but it's absolute error not relative error. For large numbers, the absolute error may not be meaningful. With ===, it will return false even the difference is only one unit of least precision (ULP), and it often happens when running the unittest under different architecture of machine. For example, ARM and X86 may have different numerical rounding , and we don't run any test other than X86. C++ boost has their numerical === test with the relative error for this reason.

I probably can add method called ~= and ~== method for Double, and Vector type using implicit class, and ~== will raise the exception for the message purpose like === does.

mengxr · 2014-07-16T20:27:32Z

almostEquals reads better than ~===. The feature we like is having the values in comparison in the error message but not the name :)

dbtsai · 2014-07-16T20:35:30Z

I learn almostEquals from boost library. Anyway, in this case, how do we distinguish the one with throwing out the message, and the one just returning true/false?

almostEquals and almostEqualsWithMessage?

SparkQA · 2014-07-17T02:47:53Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16763/consoleFull

dbtsai · 2014-07-17T02:50:36Z

@mengxr and @srowen What do you think assert((0.0001 !~== 0.0) +- 1E-5)? We have ~== and !~== which will have the error message in the latest commit from my co-worker.

SparkQA · 2014-07-17T14:57:54Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16780/consoleFull

SparkQA · 2014-07-17T15:22:56Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16781/consoleFull

mengxr · 2014-07-21T23:54:30Z

@dbtsai The suggestion of using ~== and ~= looks good to me. But is !~== really used somewhere in the tests?

dbtsai · 2014-07-22T00:03:10Z

!~== will be used in the test since !(a~==b) will not work due to that (a~==b) is not returning false but throwing exception for messaging. I will replace the almostEquals with ~==. Thanks.

dorx · 2014-07-22T00:13:58Z

@dbtsai this is awesome! I actually created a JIRA on this after trying to use TestUtils in one of my unit suites, but it looks like you're already taking care of it. https://issues.apache.org/jira/browse/SPARK-2599

mengxr · 2014-07-22T05:05:53Z

@dbtsai I saw why !(a~==b) doesn't work but the question was that !~== was not used in our tests except the unit tests for itself.

srowen · 2014-07-22T08:48:54Z

mllib/src/test/scala/org/apache/spark/mllib/util/TestingUtils.scala

        true
-      } else if(math.abs(x) > math.abs(y)) {
-        math.abs(x - y) / math.abs(x) < epsilon
+      } else if (absX < 1E-15 || absY < 1E-15) {


See commentary at https://issues.apache.org/jira/browse/SPARK-2599?focusedCommentId=14068293&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14068293 I can see the idea here, but all of these kinds of efforts seem to lead to errors or unintuitive behavior. For example this line means that:

Fails:
expected = 1e-15
actual = 2e-15
eps = 0.1

Passes:
expected = 1e-16
actual = 2e-16
eps = 0.1

Why is 1e-15 special anyway?

srowen · 2014-07-22T08:55:07Z

Is it possible to support syntax like 0.3 +- 0.1 for absolute error, and 0.3 +- 10% for relative error? Seems like the kind of crazy thing that Scala just might support. Maybe it's a nice way to support both semantics; I think relative error semantics have to be a separate method anwayy.

Also, there is the method Math.ulp (http://docs.oracle.com/javase/7/docs/api/java/lang/Math.html#ulp(double)) whichs tell you how big the gap is between floating-point values around a given value. How about using this to pick a reasonable absolute error around the test's expected value? At least it scales automatically.

dbtsai · 2014-07-22T21:25:32Z

I also noticed that it will be suffering when comparing against 0.0. As @srowen pointed out, it's meaningless to comparing against 0.0 (or a really small number) with relative error. However, people may just want to write unittest using relative error for even comparing those small numbers. So I purpose the following APIs.
a ~== b +- eps for relative error, and when a or b near zero, let's say 1e-15, it falls back to absolute error.
a === b +- eps which is already in scalatest 2.0 for absolute error, but since we don't use scalatest 2.0 yet, we build the same APIs in mllib for absolute error.

mengxr · 2014-07-22T22:12:08Z

@srowen +- 10% is not very practical because we usually need +- 0.000000000001%. For most numerical computation, the switching point is 1.0. Above 1.0, use relative error or absolute error otherwise. Because of 1.0, this is a smooth change.

srowen · 2014-07-23T09:46:23Z

@mengxr Sure, maybe the % syntax isn't helpful. I just mean two different operators or methods of some kind. Why bother with these issues instead of making two methods?

Yes, choosing 1.0 as the switching point removes the discontinuity. I think it will surprise readers to find that, in a test with a series of checks like "0.1 +- 0.01", "1 +- 0.01", "3 +- 0.01" that the latter doesn't mean [2.99,3.01], when the first two do in fact mean [0.09,0.11] and [0.99,1.01], which matches what one would expect from all these other unit testing frameworks.

@dbtsai I think developing two operators is a good solution If there is a separate operator for relative error, you don't need to special-case the behavior. Sure it's meaningless to make a relative error test at 0 but you can just warn the caller; it's well-defined what happens.

SparkQA · 2014-07-24T00:18:26Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17071/consoleFull

SparkQA · 2014-07-24T00:28:47Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17073/consoleFull

dbtsai · 2014-07-24T00:35:13Z

@srowen @mengxr and @dorx

Based on our discussion, I've implemented two different APIs for relative error, and absolute error. It makes sense that test writers should know which one they need depending on their circumstances.

Developers also need to explicitly specify the eps now, and there is no default value which will sometimes cause confusion.

When comparing against zero using relative error, a exception will be raised to warn users that it's meaningless.

For relative error in percentage, users can now write

assert(23.1 ~== 23.52 %+- 2.0)
assert(23.1 ~== 22.74 %+- 2.0)
assert(23.1 ~= 23.52 %+- 2.0)
assert(23.1 ~= 22.74 %+- 2.0)
assert(!(23.1 !~= 23.52 %+- 2.0))
assert(!(23.1 !~= 22.74 %+- 2.0))

// This will throw exception with the following message.
// "Did not expect 23.1 and 23.52 to be within 2.0% using relative error."
assert(23.1 !~== 23.52 %+- 2.0)

// "Expected 23.1 and 22.34 to be within 2.0% using relative error."
assert(23.1 ~== 22.34 %+- 2.0)

For absolute error,

assert(17.8 ~== 17.99 +- 0.2)
assert(17.8 ~== 17.61 +- 0.2)
assert(17.8 ~= 17.99 +- 0.2)
assert(17.8 ~= 17.61 +- 0.2)
assert(!(17.8 !~= 17.99 +- 0.2))
assert(!(17.8 !~= 17.61 +- 0.2))

// This will throw exception with the following message.
// "Did not expect 17.8 and 17.99 to be within 0.2 using absolute error."
assert(17.8 !~== 17.99 +- 0.2)

// "Expected 17.8 and 17.59 to be within 0.2 using absolute error."
assert(17.8 ~== 17.59 +- 0.2)

SparkQA · 2014-07-24T01:13:26Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17080/consoleFull

SparkQA · 2014-07-24T02:02:06Z

QA results for PR 1425:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(val x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(val fun: (Double, Double, Double) => Boolean,
case class CompareVectorRightSide(val fun: (Vector, Vector, Double) => Boolean,
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17071/consoleFull

SparkQA · 2014-07-24T02:14:29Z

QA results for PR 1425:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(val x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(val fun: (Double, Double, Double) => Boolean,
case class CompareVectorRightSide(val fun: (Vector, Vector, Double) => Boolean,
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17073/consoleFull

SparkQA · 2014-07-24T02:54:19Z

QA results for PR 1425:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(val x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(val fun: (Double, Double, Double) => Boolean,
case class CompareVectorRightSide(val fun: (Vector, Vector, Double) => Boolean,
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17080/consoleFull

SparkQA · 2014-07-24T03:08:33Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17091/consoleFull

SparkQA · 2014-07-24T03:50:40Z

QA results for PR 1425:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(val x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(val fun: (Double, Double, Double) => Boolean,
case class CompareVectorRightSide(val fun: (Vector, Vector, Double) => Boolean,
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17091/consoleFull

mengxr · 2014-07-24T16:49:28Z

@dbtsai As discussed with @srowen , the % sign is not helpful because we need 0.00000000000001% in many cases and people never use notations like 1e-10%.

Instead of adding more sugar in testing, how about using absErr/absTol and relErr/relTol? For example,

assert(a ~== b absErr 0.01)
assert(a ~== b relTol 1e-5)

I still recommend using err/tol that switches automatically between absolute error and relative error at 1.0. But it is fine to be explicit.

mengxr · 2014-07-27T14:28:59Z

@dbtsai I think it is very uncommon to combine the scientific notation with percentage, like 1e-10 percent. Shall we switch to absTol and relTol instead? I feel *Tol is better than *Err because we are testing an equality with tolerance but not error. MATLAB also uses tolerance: http://www.mathworks.com/help/matlab/ref/matlab.unittest.constraints.tolerance-class.html

SparkQA · 2014-07-27T23:43:49Z

QA tests have started for PR 1425. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17255/consoleFull

SparkQA · 2014-07-27T23:48:58Z

QA tests have started for PR 1425. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17257/consoleFull

SparkQA · 2014-07-27T23:58:44Z

QA tests have started for PR 1425. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17258/consoleFull

mengxr · 2014-07-28T00:09:41Z

mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala

@@ -40,27 +41,51 @@ class KMeansSuite extends FunSuite with LocalSparkContext {
    // No matter how many runs or iterations we use, we should get one cluster,
    // centered at the mean of the points

+<<<<<<< HEAD


This was not merged cleanly.

Tried to rebase against master with conflicts. I addressed them in the next push.

SparkQA · 2014-07-28T00:14:22Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17261/consoleFull

SparkQA · 2014-07-28T00:57:42Z

QA results for PR 1425:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(
case class CompareVectorRightSide(
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17261/consoleFull

SparkQA · 2014-07-28T01:26:57Z

QA results for PR 1425:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17255/consoleFull

SparkQA · 2014-07-28T01:34:03Z

QA results for PR 1425:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17257/consoleFull

dbtsai · 2014-07-28T01:36:13Z

@mengxr Resolved all the conflicts after rebasing, and all the unittests are passed. Thanks.

SparkQA · 2014-07-28T01:39:09Z

QA results for PR 1425:
- This patch FAILED unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17258/consoleFull

mengxr · 2014-07-28T06:08:20Z

There were some problems with pyspark. Let's call Jenkins again.

mengxr · 2014-07-28T06:08:27Z

Jenkins, retest this please.

mengxr · 2014-07-28T16:41:44Z

Jenkins, retest this please.

SparkQA · 2014-07-28T16:43:52Z

QA tests have started for PR 1425. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17283/consoleFull

SparkQA · 2014-07-28T17:33:43Z

QA results for PR 1425:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class VectorWithCompare(x: Vector) extends Ordered[VectorWithCompare] {
case class CompareDoubleRightSide(
case class CompareVectorRightSide(
class TestingUtilsSuite extends FunSuite {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17283/consoleFull

mengxr · 2014-07-28T18:34:04Z

LGTM. I'm merging this into master! Thanks!!

@dbtsai

Allow small errors in comparison. @dbtsai , this unit test blocks apache#1562 . I may need to merge this one first. We can change it to use the tools in apache#1425 after that PR gets merged. Author: Xiangrui Meng <meng@databricks.com> Closes apache#1576 from mengxr/fix-binary-metrics-unit-tests and squashes the following commits: 5076a7f [Xiangrui Meng] fix binary metrics unit tests

…rror in UnitTests Floating point math is not exact, and most floating-point numbers end up being slightly imprecise due to rounding errors. Simple values like 0.1 cannot be precisely represented using binary floating point numbers, and the limited precision of floating point numbers means that slight changes in the order of operations or the precision of intermediates can change the result. That means that comparing two floats to see if they are equal is usually not what we want. As long as this imprecision stays small, it can usually be ignored. Based on discussion in the community, we have implemented two different APIs for relative tolerance, and absolute tolerance. It makes sense that test writers should know which one they need depending on their circumstances. Developers also need to explicitly specify the eps, and there is no default value which will sometimes cause confusion. When comparing against zero using relative tolerance, a exception will be raised to warn users that it's meaningless. For relative tolerance, users can now write assert(23.1 ~== 23.52 relTol 0.02) assert(23.1 ~== 22.74 relTol 0.02) assert(23.1 ~= 23.52 relTol 0.02) assert(23.1 ~= 22.74 relTol 0.02) assert(!(23.1 !~= 23.52 relTol 0.02)) assert(!(23.1 !~= 22.74 relTol 0.02)) // This will throw exception with the following message. // "Did not expect 23.1 and 23.52 to be within 0.02 using relative tolerance." assert(23.1 !~== 23.52 relTol 0.02) // "Expected 23.1 and 22.34 to be within 0.02 using relative tolerance." assert(23.1 ~== 22.34 relTol 0.02) For absolute error, assert(17.8 ~== 17.99 absTol 0.2) assert(17.8 ~== 17.61 absTol 0.2) assert(17.8 ~= 17.99 absTol 0.2) assert(17.8 ~= 17.61 absTol 0.2) assert(!(17.8 !~= 17.99 absTol 0.2)) assert(!(17.8 !~= 17.61 absTol 0.2)) // This will throw exception with the following message. // "Did not expect 17.8 and 17.99 to be within 0.2 using absolute error." assert(17.8 !~== 17.99 absTol 0.2) // "Expected 17.8 and 17.59 to be within 0.2 using absolute error." assert(17.8 ~== 17.59 absTol 0.2) Authors: DB Tsai <dbtsaialpinenow.com> Marek Kolodziej <marekalpinenow.com> Author: DB Tsai <dbtsai@alpinenow.com> Closes apache#1425 from dbtsai/SPARK-2479_comparing_floating_point and squashes the following commits: 8c7cbcc [DB Tsai] Alpine Data Labs

mengxr reviewed Jul 16, 2014
View reviewed changes

srowen reviewed Jul 16, 2014
View reviewed changes

srowen reviewed Jul 22, 2014
View reviewed changes

mengxr reviewed Jul 28, 2014
View reviewed changes

Alpine Data Labs

8c7cbcc

asfgit closed this in 255b56f Jul 29, 2014

dbtsai deleted the SPARK-2479_comparing_floating_point branch August 11, 2014 22:34

[SPARK-2479][MLlib] Comparing floating-point numbers using relative error in UnitTests #1425

[SPARK-2479][MLlib] Comparing floating-point numbers using relative error in UnitTests #1425

Conversation

dbtsai commented Jul 16, 2014

SparkQA commented Jul 16, 2014

mengxr Jul 16, 2014

Choose a reason for hiding this comment

dbtsai Jul 16, 2014

Choose a reason for hiding this comment

mengxr commented Jul 16, 2014

srowen Jul 16, 2014

Choose a reason for hiding this comment

dbtsai Jul 16, 2014

Choose a reason for hiding this comment

dbtsai commented Jul 16, 2014

mengxr commented Jul 16, 2014

dbtsai commented Jul 16, 2014

SparkQA commented Jul 17, 2014

dbtsai commented Jul 17, 2014

SparkQA commented Jul 17, 2014

SparkQA commented Jul 17, 2014

mengxr commented Jul 21, 2014

dbtsai commented Jul 22, 2014

dorx commented Jul 22, 2014

mengxr commented Jul 22, 2014

srowen Jul 22, 2014

Choose a reason for hiding this comment

srowen commented Jul 22, 2014

dbtsai commented Jul 22, 2014

mengxr commented Jul 22, 2014

srowen commented Jul 23, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

dbtsai commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

mengxr commented Jul 24, 2014

mengxr commented Jul 27, 2014

SparkQA commented Jul 27, 2014

SparkQA commented Jul 27, 2014

SparkQA commented Jul 27, 2014

mengxr Jul 28, 2014

Choose a reason for hiding this comment

dbtsai Jul 28, 2014

Choose a reason for hiding this comment

SparkQA commented Jul 28, 2014

SparkQA commented Jul 28, 2014

SparkQA commented Jul 28, 2014

SparkQA commented Jul 28, 2014

dbtsai commented Jul 28, 2014

SparkQA commented Jul 28, 2014

mengxr commented Jul 28, 2014

mengxr commented Jul 28, 2014

mengxr commented Jul 28, 2014

SparkQA commented Jul 28, 2014

SparkQA commented Jul 28, 2014

mengxr commented Jul 28, 2014