[SPARK-3128][MLLIB] Use streaming test suite for StreamingLR #2037
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Test predictOnValues for accuracy on a test stream
- Made mllib depend on tests from streaming - Rewrote all streamingLR tests to use the setupStreams & runStreams functions
|
Jenkins, test this please. |
| @@ -242,7 +242,7 @@ trait TestSuiteBase extends FunSuite with BeforeAndAfter with Logging { | |||
| logInfo("numBatches = " + numBatches + ", numExpectedOutput = " + numExpectedOutput) | |||
|
|
|||
| // Get the output buffer | |||
| val outputStream = ssc.graph.getOutputStreams.head.asInstanceOf[TestOutputStreamWithPartitions[V]] | |||
| val outputStream = ssc.graph.getOutputStreams.filter(_.isInstanceOf[TestOutputStreamWithPartitions[_]]).head.asInstanceOf[TestOutputStreamWithPartitions[V]] | |||
There was a problem hiding this comment.
This line is over 100 characters.
|
QA tests have started for PR 2037 at commit
|
|
|
||
| // compute the mean absolute error and check that it's always less than 0.1 | ||
| val errors = output.map(batch => batch.map( | ||
| p => math.abs(p._1 - p._2)).reduce(_+_) / nPoints.toDouble) |
There was a problem hiding this comment.
.reduce(_+_) -> sum
Then it could be fit into one line:
val errors = output.map(batch => batch.map(p => math.abs(p._1 - p._2)).sum / nPoints)
sum returns Double so we don't need to call toDouble explicitly.
|
QA tests have finished for PR 2037 at commit
|
|
Jenkins, test this please. |
|
QA tests have started for PR 2037 at commit
|
|
QA tests have started for PR 2037 at commit
|
| // compute the mean absolute error and check that it's always less than 0.1 | ||
| val errors = output.map(batch => batch.map(p => math.abs(p._1 - p._2)).sum / nPoints) | ||
| assert(errors.forall(x => x <= 0.1)) | ||
|
|
|
LGTM, if tests pass. Small nits, which can be ignored if there is nothing else to be changed. |
|
QA tests have finished for PR 2037 at commit
|
|
QA tests have finished for PR 2037 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Aug 19, 2014
Refactored tests for streaming linear regression to use existing streaming test utilities. Summary of changes: - Made ``mllib`` depend on tests from ``streaming`` - Rewrote accuracy and convergence tests to use ``setupStreams`` and ``runStreams`` - Added new test for the accuracy of predictions generated by ``predictOnValue`` These tests should run faster, be easier to extend/maintain, and provide a reference for new tests. mengxr tdas Author: freeman <the.freeman.lab@gmail.com> Closes #2037 from freeman-lab/streamingLR-predict-tests and squashes the following commits: e851ca7 [freeman] Fixed long lines 50eb0bf [freeman] Refactored tests to use streaming test tools 32c43c2 [freeman] Added test for prediction (cherry picked from commit 31f0b07) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
|
Thanks a lot @freeman-lab |
|
@tdas It seems that this creates a folder called |
xiliu82
pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
Refactored tests for streaming linear regression to use existing streaming test utilities. Summary of changes: - Made ``mllib`` depend on tests from ``streaming`` - Rewrote accuracy and convergence tests to use ``setupStreams`` and ``runStreams`` - Added new test for the accuracy of predictions generated by ``predictOnValue`` These tests should run faster, be easier to extend/maintain, and provide a reference for new tests. mengxr tdas Author: freeman <the.freeman.lab@gmail.com> Closes apache#2037 from freeman-lab/streamingLR-predict-tests and squashes the following commits: e851ca7 [freeman] Fixed long lines 50eb0bf [freeman] Refactored tests to use streaming test tools 32c43c2 [freeman] Added test for prediction
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactored tests for streaming linear regression to use existing streaming test utilities. Summary of changes:
mllibdepend on tests fromstreamingsetupStreamsandrunStreamspredictOnValueThese tests should run faster, be easier to extend/maintain, and provide a reference for new tests.
@mengxr @tdas