Top k multi error

[btrotta]

Implements the feature requested in issue #1139 . Added new parameter top_k_threshold. When the metric is set to multi_error, parameter top_k_threshold can be used to obtain the top-k multi-error. By default this parameter is set to 1, which gives the usual multi-error.

[msftclas]

All CLA requirements met.

[StrikerRUS]

@btrotta Thank you very much for your contribution!

Seems that your new tests fail on Python 2.7. Can you please take a look?

Also please move them to the existent test_engine.py file. Maybe in the future we should divide our tests in more elegant way, but for now there is no need to create a new separate file for 2 small tests.

[StrikerRUS]

We already have top_k/topk parameter for voting parallel: https://lightgbm.readthedocs.io/en/latest/Parameters.html#top_k.
I suppose that for this parameter we should take less confusing name. What about multi_error_top_k_threshold or simply multi_error_top_k?
@guolinke

[guolinke]

yes, I think it is better to change the parameter name.

[StrikerRUS]

Seems that new behavior differs from previous one in case of model predicts all classes to be equally likely.
Consider topk@1 and score = [0.33, 0.33, 0.33]
Old:
LossOnPoint => 1.0
New:
LossOnPoint => 0.0

[btrotta]

@StrikerRUS Good point! I've fixed this in the latest commit. The metric is now defined so that the top-k error on a sample is 0 if there are at least num_classes - k predictions strictly less than the prediction on the true class.

[StrikerRUS]

@btrotta Thank you for hotfixing that!
I think we can enhance the metric name to help users orient in logs.
Just like map or ndcg:

LightGBM/src/metric/map_metric.hpp

Lines 38 to 40 in 0a4a7a8

	for (auto k : eval_at_) {
	name_.emplace_back(std::string("map@") + std::to_string(k));
	}

The only thing I'm doubt about is that we cannot change multi_error -> multi_error@1 due to backward compatibility with users' existing codebase which relies on the previous name. So, it should be a special case.

[btrotta]

@StrikerRUS Done

[StrikerRUS]

Thanks! Now the general concept looks good to me. Just several nitpicks below for the consistency with existing codebase and formatting style.

[btrotta]

Formatting issues fixed in latest commit.

[StrikerRUS]

@btrotta Thank you very much for your contribution!
LGTM except two comments below!

[btrotta]

@StrikerRUS Thanks so much for your careful review! Those 2 issues with docs are fixed now.

[StrikerRUS]

@guolinke Can you please give a second review to this PR?

[guolinke]

it looks good to me

[StrikerRUS]

@guolinke Should this PR be in 2.2.4 release?

[guolinke]

@StrikerRUS yeah, it can be.

[StrikerRUS]

@guolinke Then I'm merging 😃