Need support for reading instance weights from feature files

[doxav]

I noticed that instance weights are not supported by Skll readers and writers. It is a key feature to deal with unbalanced data (or cost-sensitive learning additionally to matrix cost). ARFF supports it (http://weka.wikispaces.com/ARFF+(stable+version)#Instance weights in ARFF files ) and some Sklearn classifiers.

[dan-blanchard]

It is true that we don't support that in our ARFF reader, but there are only 4 classifiers in scikit-learn supported by SKLL that support the class_weight parameter: SVC, LogisticRegression, LinearSVC and SGDClassifier.

The way we've been handling unbalanced data is by specifying the class_weight option to in fixed_parameters as explained here in our docs.

You'll have to search for "class_weight" because sphinx didn't generate an anchor for that particular note. We should really have better documentation for this.

[doxav]

So no chance, it'll change ? Your ml data converters could be useful for a broader use (more sklearn estimators or any ml lib). Skll could be a more general purpose library then. I currently use Skll only for file conversion and it is already a limiting factor. I also use sklearn but extend it to map other ML libs (Vowpal, MOA) as estimators, those libs use weights. So in the same principle, Skll could support much more estimators easily.

[dan-blanchard]

Your ml data converters could be useful for a broader use...

But the target format would also have to support instance weights, which I don't believe any of the others do.

Granted, we could just make it so ARFFReader adds multiple copies of any instances that have a weight greater than 1 to the resulting FeatureSet, but if the weights can be floats that obviously wouldn't work. It's not clear from the format description whether weights have to be integers or not.

As for supporting other ML libs, we've actually got a PR open (#183) to add support for this sort of thing. We just need to add some examples and tests for it.

[dan-blanchard]

Hmm... apparently scikit-learn supports instance weights and not just class weights. If we were going to support reading the instance weights from files, we'd need a way to do it across file formats. That would be pretty straightforward for all of the supported file types except for MegaM. Although, I guess we could add it to the comment that we're currently using for IDs, much like we do for LibSVM files.

Adding support for this would be a somewhat major change, so I'm going to slate this for the 1.1 release.

[desilinguist]

I think this still squarely belongs in the help-wanted category since we haven't seen a lot of demand for this internally.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[ghost]

Please keep it open.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[aoifecahill]

keep it open please

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[desilinguist]

I am going to close this since no one really seems to want this.