[MXNET-62] add test against spark integration #10462
Conversation
| $SPARK_HOME/bin/spark-submit --master spark://$HOST:7077 \ | ||
| --class org.apache.mxnet.spark.example.ClassificationExample \ | ||
| $SPARK_HOME/bin/spark-submit --master local[*] \ | ||
| --class ml.dmlc.mxnet.spark.example.ClassificationExample \ |
|
|
||
| test("run spark with MLP") { | ||
| val trainData = parseRawData(sc, | ||
| "/Users/nanzhu/code/mxnet/scala-package/spark/train.txt") |
There was a problem hiding this comment.
@szha Any proper places can we put these testing files?
There was a problem hiding this comment.
any update here about uploading test files?
There was a problem hiding this comment.
ping for uploading test files @szha ?
There was a problem hiding this comment.
I can't download the file linked in the description. Could you provide a file in a publicly accessible place?
There was a problem hiding this comment.
is there any way I can upload my local copy to somewhere so that you can download it?
There was a problem hiding this comment.
@szha @piiswrong , thanks ! I have uploaded file at dmlc/web-data#63
There was a problem hiding this comment.
There was a problem hiding this comment.
Hi, @szha, would you please help to uncompress and reupload, I zipped it just because github does not allow a large file...I am not sure if jenkins env has unzip command line equipped, etc.
There was a problem hiding this comment.
As requested I decompressed the file and put it here: http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/mxnet-spark-test/train.txt
Why can't you decompress the file in test code?
CodingCat
force-pushed
the
spark_unit_test
branch
from
22f17e2
to
837106c
Compare
CodingCat
force-pushed
the
spark_unit_test
branch
from
e1a3203
to
548cc9f
Compare
|
Ping |
|
sorry about the delay, really got trapped by daily jobs these weeks, will update it this week |
|
@yzhliu want to take another look? I think the failed test is unrelated to my change you can check the stdout of scala cpu run, Spark ML unit tests are executed |
|
good to me. Just push some trivial changes, e.g., modify the commit msg, to re-trigger the CI. |
|
eh, for some reason it failed for another test suite (not scala one), and for scala one, the test was killed? (terminate called without an active exception) |
|
@CodingCat Can you check whether the failed test is expected? Looks like it is related to what you modified. |
|
scala test was killed in the middle...not sure why.... |
CodingCat
force-pushed
the
spark_unit_test
branch
from
7f987ba
to
220b116
Compare
|
@yzhliu it passed now |
* fix bug * temp * temp * temp * update * update * update * remove debugging stubs * remove unused * stylistic fix * fix typo * Pulled down update to submodule_dir * add test * retrigger it * sync 3rd party
* fix bug * temp * temp * temp * update * update * update * remove debugging stubs * remove unused * stylistic fix * fix typo * Pulled down update to submodule_dir * add test * retrigger it * sync 3rd party
the test would be passed only after we
review and merge Allows ps to be shutdown and turn up in the same node dmlc/ps-lite#133
upload the original https://s3-us-west-2.amazonaws.com/mxnet.liuyz/data/mnist/train.txt to somewhere and I update the code accordingly
the test has been run locally