Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[jvm-packages] call setGroup for ranking task #2066

Merged
merged 12 commits into from Mar 6, 2017
Merged

[jvm-packages] call setGroup for ranking task #2066

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
70c5240
[jvm-packages] call setGroup for ranking task
cloverrose Feb 26, 2017
aac2ede
passing groupData through xgBoostConfMap
cloverrose Feb 26, 2017
5854fae
fix original comment position
cloverrose Mar 2, 2017
d197ab8
make groupData param
cloverrose Mar 2, 2017
6b09bf5
remove groupData variable, use xgBoostConfMap directly
cloverrose Mar 2, 2017
31b93ed
set default groupData value
cloverrose Mar 3, 2017
deb5a1e
add use groupData tests
cloverrose Mar 5, 2017
63fcb77
Merge remote-tracking branch 'upstream/master' into setGroup-spark
cloverrose Mar 5, 2017
d29e6f9
reduce rank-demo size
cloverrose Mar 6, 2017
537368e
use TaskContext.getPartitionId() instead of mapPartitionsWithIndex
cloverrose Mar 6, 2017
2aa8a4a
add DF use groupData test
cloverrose Mar 6, 2017
56fcfbc
remove unused varable
cloverrose Mar 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 7 additions & 2 deletions jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/XGBoost.scala
View file
Open in desktop
Expand Up @@ -107,8 +107,8 @@ object XGBoost extends Serializable {
// to workaround the empty partitions in training dataset,
// this might not be the best efficient implementation, see
// (https://github.com/dmlc/xgboost/issues/1277)
partitionedTrainingSet.mapPartitions {
trainingSamples =>
partitionedTrainingSet.mapPartitionsWithIndex {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put the original comments back

case (partIndex, trainingSamples) =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this change, you don't need to use mapPartitionsWithIndex, and you can just directly use TaskContext.getPartitionId() to get partitionIndex

rabitEnv.put("DMLC_TASK_ID", TaskContext.getPartitionId().toString)
Rabit.init(rabitEnv)
var booster: Booster = null
Expand All @@ -123,6 +123,11 @@ object XGBoost extends Serializable {
}
val partitionItr = fromDenseToSparseLabeledPoints(trainingSamples, missing)
val trainingSet = new DMatrix(new JDMatrix(partitionItr, cacheFileName))
if (xgBoostConfMap.isDefinedAt("groupData")
&& xgBoostConfMap.get("groupData").get != null) {
trainingSet.setGroup(
xgBoostConfMap.get("groupData").get.asInstanceOf[Seq[Seq[Int]]](partIndex).toArray)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the difference to serialize groupData as a part of xgboostConfMap and an independent variable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I can not understand the intent of the question correctly.
If my answer dose not make sense, please let me know.

I want to call setGroup(group: Array[Int])

And checking xgBoostConfMap is costly.

So I check xgBoostConfMap and make independ variable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

say 'xgBoostConfMap' has M bytes, 'groupData' has N bytes, with a naive (and incorrect) xgBoostConfMapFiltered will have M - N bytes (which should be larger than M-N considering JVM overhead, etc.)

You still call groupData(partIndex) in the closure of mapPartition, so groupData (N bytes) will still be serialized and pass to executors, and xgBoostConfMapFiltered is also called within mapPartition, so another (M - N) will be serialized and passed to executors...

I didn't see any difference here

booster = SXGBoost.train(trainingSet, xgBoostConfMap, round,
watches = new mutable.HashMap[String, DMatrix] {
put("train", trainingSet)
Expand Down
9 changes: 8 additions & 1 deletion ...oost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/params/LearningTaskParams.scala
View file
Open in desktop
Expand Up @@ -53,7 +53,14 @@ trait LearningTaskParams extends Params {
s" {${LearningTaskParams.supportedEvalMetrics.mkString(",")}}",
(value: String) => LearningTaskParams.supportedEvalMetrics.contains(value))

setDefault(objective -> "reg:linear", baseScore -> 0.5, numClasses -> 2)
/**
* group data specify each group sizes for ranking task. To correspond to partition of
* training data, it is nested.
*/
val groupData = new Param[Seq[Seq[Int]]](this, "groupData", "group data specify each group size" +
" for ranking task. To correspond to partition of training data, it is nested.")

setDefault(objective -> "reg:linear", baseScore -> 0.5, numClasses -> 2, groupData -> null)
}

private[spark] object LearningTaskParams {
Expand Down
7,423 changes: 7,423 additions & 0 deletions jvm-packages/xgboost4j-spark/src/test/resources/rank-demo-0.txt.train
View file
Open in desktop

Large diffs are not rendered by default.