Remove default tuning objective and make it a required field.

[desilinguist]

Right now, the default metric for all learners is f1_score_micro which doesn't make sense for regressors.

(Note: This issue is related to but separate from #350.)

[desilinguist]

I am not really sure how to do this honestly since the default objective is set during config parsing and to override the defaults means actually checking whether all the learners are regressors and doing so means actually instantiating them early during config parsing itself into objects to check their _estimator_type attribute. This seems wasteful to me.

I am trying to think of a better solution. Suggestions and thoughts welcome.

[aoifecahill]

Couldn't we do something a little less sophisticated and just hard code the string names of the regressors and classifiers wrapped by skll? We do that in other places for other reasons.

It wouldn't work for a custom learner though.

[desilinguist]

So, then we have another place where we have to update the names of learners when we add new ones? I am not a fan of that approach - what if someone forgets to update the list? Then we need to add more tests for that scenario and basically increase the complexity of the codebase. Plus it wouldn't work for a custom learner like you said.

May be we should just not have a default objective at all? If you don't specify an objective in the config file, you get an early error during config parsing itself. This way, you don't get an incomplete experiment and the responsibility of the objective is on the user itself.

It would break backward compatibility though for old config files which is not ideal either.

[aoifecahill]

Well if we don't want to add complexity to the config reading by doing the checks there, then I think the next best thing is to not have a default, but I suppose that would have to be part of a 2.0 release if it's breaking backwards compatibility.

[desilinguist]

Agreed. Let's do that in v2.0.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[desilinguist]

We will fix this in v2.0. Keep it open.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[desilinguist]

Keep it open for v2.0

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[aoifecahill]

keep it open

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[desilinguist]

keep it open please

[desilinguist]

As part of this issue, I am also going to get rid of the objective field and keep only the objectives field. This would simplify the code a lot and also improve consistency with the metrics option (both being lists of scoring functions).

[desilinguist]

Addressed by #458.