setting number of thread per executor

[perrital]

I'm using version com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc2 on a k8s cluster.
Specifically the lightgbm classifier with binary response.
I'm setting num executors to 10 and allocating spark.executor.cores=80 and park.task.cpus=80 so that each machine runs exactly one task with 80 cores available for the task.
What I was expecting to see is full utilisation of 80 cores, instead only ~8 cores are utilised.
My best guess is that it is related to the num_threads parameter, which you have exposed as numThreads a long time ago, but currently is not present in the param set.
I have dataset of 30,000,000 samples with a large amount of features, each tree take around 10 sec to generate
rebagging takes a lot of time too and uses a single core.
Please advise

[brunocous]

Same problem as in #292. Not fixed yet.

Try to play with the number of partitions, and partition key(s). This resulted in 20% utilisation instead of 10%. So still not there.

[shuDaoNan9]

Same problem +1

[imatiach-msft]

@jwenbin have you tried the new single dataset mode parameter on latest master?
#1066
In our benchmarking it resolved the low CPU utilization issue.