Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix jvm tests. #4903

Closed
wants to merge 3 commits into from
Closed

Fix jvm tests. #4903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db1bb0f
Removed hardcoded paths from 'xgboost4j'
trams Jul 18, 2019
73c520f
Update checktyle xml to point to proper DTD
trams Jul 19, 2019
ad9c910
[Do not merge] Make mvn more verbose to ease debugging
trams Sep 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions jvm-packages/checkstyle-suppressions.xml
View file
Open in desktop
Expand Up @@ -16,8 +16,8 @@
-->

<!DOCTYPE suppressions PUBLIC
"-//Puppy Crawl//DTD Suppressions 1.1//EN"
"http://www.puppycrawl.com/dtds/suppressions_1_1.dtd">
"-//Checkstyle//DTD SuppressionFilter Configuration 1.1//EN"
"https://checkstyle.org/dtds/suppressions_1_1.dtd">

<!--

Expand Down
4 changes: 2 additions & 2 deletions jvm-packages/checkstyle.xml
View file
Open in desktop
Expand Up @@ -16,8 +16,8 @@
-->

<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
"-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
"https://checkstyle.org/dtds/configuration_1_3.dtd">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just question, what happened to them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly I do not know.
It may be worth reporting to the owner of puppycrawl.com


<!--

Expand Down
9 changes: 3 additions & 6 deletions jvm-packages/create_jni.py
View file
Open in desktop
Expand Up @@ -110,16 +110,13 @@ def normpath(path):
"xgboost4j/src/main/resources")

print("copying train/test files")
maybe_makedirs("xgboost4j-spark/src/test/resources")
maybe_makedirs("xgboost4j/src/test/resources")
with cd("../demo/regression"):
run("{} mapfeat.py".format(sys.executable))
run("{} mknfold.py machine.txt 1".format(sys.executable))

for file in glob.glob("../demo/regression/machine.txt.t*"):
cp(file, "xgboost4j-spark/src/test/resources")
for file in glob.glob("../demo/data/agaricus.*"):
cp(file, "xgboost4j-spark/src/test/resources")

maybe_makedirs("xgboost4j/src/test/resources")
for file in glob.glob("../demo/regression/machine.txt.t*"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if machine.txt.t* is only used for xgboost4j-spark, why we put it in xgboost4j?

cp(file, "xgboost4j/src/test/resources")
for file in glob.glob("../demo/data/agaricus.*"):
cp(file, "xgboost4j/src/test/resources")
7 changes: 7 additions & 0 deletions jvm-packages/xgboost4j-spark/pom.xml
View file
Open in desktop
Expand Up @@ -26,6 +26,13 @@
<artifactId>xgboost4j_${scala.binary.version}</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j_${scala.binary.version}</artifactId>
<version>1.0.0-SNAPSHOT</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion ...xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/CheckpointManagerSuite.scala
View file
Open in desktop
Expand Up @@ -18,7 +18,7 @@ package ml.dmlc.xgboost4j.scala.spark

import java.io.File

import ml.dmlc.xgboost4j.scala.{Booster, DMatrix, XGBoost => SXGBoost}
import ml.dmlc.xgboost4j.scala.{Classification, Booster, DMatrix, XGBoost => SXGBoost}
import org.scalatest.FunSuite
import org.apache.hadoop.fs.{FileSystem, Path}

Expand Down
2 changes: 1 addition & 1 deletion ...j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/DeterministicPartitioningSuite.scala 100644 → 100755
View file
Open in desktop
Expand Up @@ -16,8 +16,8 @@

package ml.dmlc.xgboost4j.scala.spark

import ml.dmlc.xgboost4j.scala.Classification
import org.scalatest.FunSuite

import org.apache.spark.sql.functions._

class DeterministicPartitioningSuite extends FunSuite with TmpFolderPerSuite with PerTest {
Expand Down
2 changes: 1 addition & 1 deletion ...ackages/xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/ParameterSuite.scala
View file
Open in desktop
Expand Up @@ -17,8 +17,8 @@
package ml.dmlc.xgboost4j.scala.spark

import ml.dmlc.xgboost4j.java.XGBoostError
import ml.dmlc.xgboost4j.scala.MultiClassification
import org.scalatest.{BeforeAndAfterAll, FunSuite}

import org.apache.spark.ml.param.ParamMap

class ParameterSuite extends FunSuite with PerTest with BeforeAndAfterAll {
Expand Down
3 changes: 1 addition & 2 deletions ...kages/xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/PersistenceSuite.scala
View file
Open in desktop
Expand Up @@ -20,10 +20,9 @@ import java.io.File
import java.util.Arrays

import scala.io.Source
import ml.dmlc.xgboost4j.scala.{Classification, DMatrix, Regression}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we have many adding of imports but never use them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked. We use all of those added imports.
One needs to understand why they are not in this diff

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I think you moved Classification and Regression to xgboost4j leading to these changes


import ml.dmlc.xgboost4j.scala.DMatrix
import scala.util.Random

import org.apache.spark.ml.feature._
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.scalatest.FunSuite
Expand Down
4 changes: 1 addition & 3 deletions ...s/xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/RabitRobustnessSuite.scala
View file
Open in desktop
Expand Up @@ -19,12 +19,10 @@ package ml.dmlc.xgboost4j.scala.spark
import java.util.concurrent.LinkedBlockingDeque

import scala.util.Random

import ml.dmlc.xgboost4j.java.{IRabitTracker, Rabit, RabitTracker => PyRabitTracker}
import ml.dmlc.xgboost4j.scala.rabit.{RabitTracker => ScalaRabitTracker}
import ml.dmlc.xgboost4j.java.IRabitTracker.TrackerStatus
import ml.dmlc.xgboost4j.scala.DMatrix

import ml.dmlc.xgboost4j.scala.{Classification, DMatrix}
import org.apache.spark.{SparkConf, SparkContext}
import org.scalatest.FunSuite

Expand Down
4 changes: 1 addition & 3 deletions ...xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostClassifierSuite.scala
View file
Open in desktop
Expand Up @@ -16,13 +16,11 @@

package ml.dmlc.xgboost4j.scala.spark

import ml.dmlc.xgboost4j.scala.{DMatrix, XGBoost => ScalaXGBoost}

import ml.dmlc.xgboost4j.scala.{Classification, DMatrix, MultiClassification, Synthetic, XGBoost => ScalaXGBoost}
import org.apache.spark.ml.linalg._
import org.apache.spark.ml.param.ParamMap
import org.apache.spark.sql._
import org.scalatest.FunSuite

import org.apache.spark.Partitioner

class XGBoostClassifierSuite extends FunSuite with PerTest {
Expand Down
2 changes: 1 addition & 1 deletion .../xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostConfigureSuite.scala
View file
Open in desktop
Expand Up @@ -16,7 +16,7 @@

package ml.dmlc.xgboost4j.scala.spark

import ml.dmlc.xgboost4j.scala.{Booster, DMatrix}
import ml.dmlc.xgboost4j.scala.{Booster, Classification, DMatrix}
import org.apache.spark.sql._
import org.scalatest.FunSuite

Expand Down
2 changes: 1 addition & 1 deletion .../xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostRegressorSuite.scala
View file
Open in desktop
Expand Up @@ -16,7 +16,7 @@

package ml.dmlc.xgboost4j.scala.spark

import ml.dmlc.xgboost4j.scala.{DMatrix, XGBoost => ScalaXGBoost}
import ml.dmlc.xgboost4j.scala.{DMatrix, Ranking, Regression, XGBoost => ScalaXGBoost}
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.sql.functions._
import org.apache.spark.sql.{DataFrame, Row}
Expand Down
53 changes: 10 additions & 43 deletions jvm-packages/xgboost4j/src/test/java/ml/dmlc/xgboost4j/java/BoosterImplTest.java
View file
Open in desktop
Expand Up @@ -24,6 +24,7 @@
import java.util.Map;

import junit.framework.TestCase;
import org.junit.Before;
import org.junit.Test;

/**
Expand Down Expand Up @@ -84,12 +85,18 @@ private Booster trainBooster(DMatrix trainMat, DMatrix testMat) throws XGBoostEr
return XGBoost.train(trainMat, paramMap, round, watches, null, null);
}

private DMatrix trainMat;
private DMatrix testMat;

@Before
public void setUp() throws XGBoostError {
trainMat = new DMatrix(Classification.trainFile.toString());
testMat = new DMatrix(Classification.testFile.toString());
}

@Test
public void testBoosterBasic() throws XGBoostError, IOException {

DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);

//predict raw output
Expand All @@ -103,8 +110,6 @@ public void testBoosterBasic() throws XGBoostError, IOException {

@Test
public void saveLoadModelWithPath() throws XGBoostError, IOException {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
IEvaluation eval = new EvalError();

Booster booster = trainBooster(trainMat, testMat);
Expand All @@ -121,8 +126,6 @@ public void saveLoadModelWithPath() throws XGBoostError, IOException {

@Test
public void saveLoadModelWithStream() throws XGBoostError, IOException {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);

Expand Down Expand Up @@ -310,8 +313,6 @@ public void testAscendMetrics() {

@Test
public void testBoosterEarlyStop() throws XGBoostError, IOException {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 3);
Expand Down Expand Up @@ -363,8 +364,6 @@ private void testWithQuantileHisto(DMatrix trainingSet, Map<String, DMatrix> wat

@Test
public void testQuantileHistoDepthWise() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 3);
Expand All @@ -383,8 +382,6 @@ public void testQuantileHistoDepthWise() throws XGBoostError {

@Test
public void testQuantileHistoLossGuide() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 0);
Expand All @@ -404,8 +401,6 @@ public void testQuantileHistoLossGuide() throws XGBoostError {

@Test
public void testQuantileHistoLossGuideMaxBin() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 0);
Expand All @@ -425,8 +420,6 @@ public void testQuantileHistoLossGuideMaxBin() throws XGBoostError {

@Test
public void testDumpModelJson() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] dump = booster.getModelDump("", false, "json");
Expand All @@ -441,9 +434,6 @@ public void testDumpModelJson() throws XGBoostError {

@Test
public void testGetFeatureScore() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] featureNames = new String[126];
for(int i = 0; i < 126; i++) featureNames[i] = "test_feature_name_" + i;
Expand All @@ -453,9 +443,6 @@ public void testGetFeatureScore() throws XGBoostError {

@Test
public void testGetFeatureImportanceGain() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] featureNames = new String[126];
for(int i = 0; i < 126; i++) featureNames[i] = "test_feature_name_" + i;
Expand All @@ -465,9 +452,6 @@ public void testGetFeatureImportanceGain() throws XGBoostError {

@Test
public void testGetFeatureImportanceTotalGain() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] featureNames = new String[126];
for(int i = 0; i < 126; i++) featureNames[i] = "test_feature_name_" + i;
Expand All @@ -477,9 +461,6 @@ public void testGetFeatureImportanceTotalGain() throws XGBoostError {

@Test
public void testGetFeatureImportanceCover() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] featureNames = new String[126];
for(int i = 0; i < 126; i++) featureNames[i] = "test_feature_name_" + i;
Expand All @@ -489,9 +470,6 @@ public void testGetFeatureImportanceCover() throws XGBoostError {

@Test
public void testGetFeatureImportanceTotalCover() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
String[] featureNames = new String[126];
for(int i = 0; i < 126; i++) featureNames[i] = "test_feature_name_" + i;
Expand All @@ -501,7 +479,6 @@ public void testGetFeatureImportanceTotalCover() throws XGBoostError {

@Test
public void testQuantileHistoDepthwiseMaxDepth() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 3);
Expand All @@ -519,8 +496,6 @@ public void testQuantileHistoDepthwiseMaxDepth() throws XGBoostError {

@Test
public void testQuantileHistoDepthwiseMaxDepthMaxBin() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
Map<String, Object> paramMap = new HashMap<String, Object>() {
{
put("max_depth", 3);
Expand All @@ -544,9 +519,6 @@ public void testQuantileHistoDepthwiseMaxDepthMaxBin() throws XGBoostError {
*/
@Test
public void testCV() throws XGBoostError {
//load train mat
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");

//set params
Map<String, Object> param = new HashMap<String, Object>() {
{
Expand All @@ -573,8 +545,6 @@ public void testCV() throws XGBoostError {
*/
@Test
public void testTrainFromExistingModel() throws XGBoostError, IOException {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");
IEvaluation eval = new EvalError();

Map<String, Object> paramMap = new HashMap<String, Object>() {
Expand Down Expand Up @@ -624,9 +594,6 @@ public void testTrainFromExistingModel() throws XGBoostError, IOException {
*/
@Test
public void testSetAndGetAttrs() throws XGBoostError {
DMatrix trainMat = new DMatrix("../../demo/data/agaricus.txt.train");
DMatrix testMat = new DMatrix("../../demo/data/agaricus.txt.test");

Booster booster = trainBooster(trainMat, testMat);
booster.setAttr("testKey1", "testValue1");
TestCase.assertEquals(booster.getAttr("testKey1"), "testValue1");
Expand Down
10 changes: 10 additions & 0 deletions jvm-packages/xgboost4j/src/test/java/ml/dmlc/xgboost4j/java/Classification.java
View file
Open in desktop
@@ -0,0 +1,10 @@
package ml.dmlc.xgboost4j.java;


import java.io.File;

public class Classification {
public static File trainFile = TrainTestData.getResourceFile("/agaricus.txt.train");
public static File testFile = TrainTestData.getResourceFile("/agaricus.txt.test");
}

3 changes: 1 addition & 2 deletions jvm-packages/xgboost4j/src/test/java/ml/dmlc/xgboost4j/java/DMatrixTest.java 100644 → 100755
View file
Open in desktop
Expand Up @@ -60,8 +60,7 @@ public void testCreateFromDataIterator() throws XGBoostError {
@Test
public void testCreateFromFile() throws XGBoostError {
//create DMatrix from file
String filePath = writeResourceIntoTempFile("/agaricus.txt.test");
DMatrix dmat = new DMatrix(filePath);
DMatrix dmat = new DMatrix(Classification.testFile.toString());
//get label
float[] labels = dmat.getLabel();
//check length
Expand Down
27 changes: 27 additions & 0 deletions jvm-packages/xgboost4j/src/test/java/ml/dmlc/xgboost4j/java/TrainTestData.java
View file
Open in desktop
@@ -0,0 +1,27 @@
package ml.dmlc.xgboost4j.java;

import java.io.File;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;

public class TrainTestData {
static File getResourceFile(String resource) {
InputStream is = TrainTestData.class.getResourceAsStream(resource);
if (is == null) {
throw new RuntimeException("Failed to resolve resource " + resource);
}
try {
try {
File file = File.createTempFile(resource.substring(1), "");
Files.copy(is, file.toPath(), StandardCopyOption.REPLACE_EXISTING);
file.deleteOnExit();
return file;
} finally {
is.close();
}
} catch (Exception e) {
throw new RuntimeException("Failed to load the resource " + resource, e);
}
}
}
2 changes: 1 addition & 1 deletion jvm-packages/xgboost4j/src/test/scala/ml/dmlc/xgboost4j/scala/DMatrixSuite.scala
View file
Open in desktop
Expand Up @@ -25,7 +25,7 @@ import ml.dmlc.xgboost4j.java.{DMatrix => JDMatrix}

class DMatrixSuite extends FunSuite {
test("create DMatrix from File") {
val dmat = new DMatrix("../../demo/data/agaricus.txt.test")
val dmat = new DMatrix(Classification.testFile.toString)
// get label
val labels: Array[Float] = dmat.getLabel
// check length
Expand Down