Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

ENH: Add LinearSVR #1867

Closed
wants to merge 17 commits into from

4 participants

@luoq

Fix #1473

sklearn/svm/classes.py
@@ -385,6 +385,104 @@ def __init__(self, nu=0.5, kernel='rbf', degree=3, gamma=0.0,
'nu_svc', kernel, degree, gamma, coef0, tol, 0., nu, 0., shrinking,
probability, cache_size, None, verbose, max_iter)
+
+class LinearSVR(BaseLibLinear,LinearRegressorMixin,SelectorMixin,SparseCoefMixin):
@jnothman Owner

Style: Insert spaces between all of the parent classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/classes.py
((82 lines not shown))
+ representation for the data that will incur a memory copy.
+
+ **References:**
+ `LIBLINEAR: A Library for Large Linear Classification
+ <http://www.csie.ntu.edu.tw/~cjlin/liblinear/>`__
+
+ See also
+ --------
+ SVR
+ Support Vector Machine for Regression implemented using libsvm.
+
+ """
+ def __init__(self, C=1.0, loss="l2", penalty="l2", epsilon=0.1, dual=True, tol=1e-1,
+ fit_intercept=True, intercept_scaling=1,
+ verbose=0, random_state=None):
+ super(LinearSVR,self).__init__(
@jnothman Owner

Style: Insert a space before self

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
@@ -680,7 +692,7 @@ def fit(self, X, y):
# LibLinear wants targets as doubles, even for classification
y = np.asarray(y, dtype=np.float64).ravel()
- self.raw_coef_ = train(X, y, self._get_solver_type(), self.tol,
+ self.raw_coef_ = train(X, y, self._get_solver_type(), self.tol, self.epsilon,
@jnothman Owner

Is it just me that finds it strange that tol and epsilon here become eps and p in the Cython and C code without any comment?

@amueller Owner

Unfortunately naming between cython and python and between dense and sparse versions is inconsistent :-/ I fix what I see.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/tests/test_svm.py
@@ -175,6 +175,21 @@ def test_svr():
clf.fit(diabetes.data, diabetes.target)
assert_greater(clf.score(diabetes.data, diabetes.target), 0.02)
+
+def test_liblinear_svr():
+ """
+ Test Liblinear SVR
+ """
+
+ diabetes = datasets.load_diabetes()
+ for clf in (svm.LinearSVR(C=1.0,epsilon=0.1),
+ svm.LinearSVR(C=10.,epsilon=0.1),
+ svm.LinearSVR(loss="l1",C=10.,epsilon=0.1),
+ svm.LinearSVR(loss="l1",C=100.,epsilon=0.1),
@jnothman Owner

Is it worth varying epsilon to ensure it is transmitted to liblinear?
Also, is it appropriate to check that the solutions/parameters found are close to those produced by SVR with a linear kernel? (or is that not guaranteed? or is that liblinear's problem, not ours?)
(There should also be spaces after commas here.)

@luoq
luoq added a note
@jnothman Owner
@amueller Owner

Are loss and penalty the same? I would have guessed that there is an l2 slack penalty in LinearSVR and a l1 lack penalty in SVR.

Maybe a quick sanity check of the results would be nice, but we can't really expect the results to be the same as SVR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman jnothman commented on the diff
sklearn/linear_model/base.py
((8 lines not shown))
+ """
+ def decision_function(self,X):
+ """Predict using the linear model
+
+ Parameters
+ ----------
+ X : {array-like, sparse matrix}, shape = [n_samples, n_features]
+
+ Returns
+ -------
+ array, shape = [n_samples]
+ Predicted target values per element in X.
+ """
+ X = atleast2d_or_csr(X)
+ scores = safe_sparse_dot(X, self.coef_.T, dense_output=True) + self.intercept_
+ return scores.ravel()
@jnothman Owner

The only differences between this class and LinearModel seem to be the ravel and constraining the X validation a bit more. I don't understand why this case needs to be distinguished, what cases require ravel, or why this class cannot merely wrap LinearModel to make the minor differences more apparent. Or is it more to do with this being a Mixin and that an ABC?

@luoq
luoq added a note
@luoq
luoq added a note

Maybe LinearModel should inherits from LinearRegressorMixin

@jnothman Owner
@jnothman Owner

@larsmans, what's the go on differentiating ABCs and Mixins?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman
Owner

Note you're failing a Travis test in the Doctest for sklearn.metrics.metrics.hinge_loss which doesn't expect the epsilon parameter to be printed; perhaps using # doctest: +ELLIPSIS there is advisable.

@luoq

+ELLIPSIS cannot remove the error. I will change the docstring to include epsilon

@jnothman
Owner
sklearn/metrics/metrics.py
@@ -246,9 +246,9 @@ def hinge_loss(y_true, pred_decision, pos_label=1, neg_label=-1):
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
- LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
- intercept_scaling=1, loss='l2', multi_class='ovr', penalty='l2',
- random_state=0, tol=0.0001, verbose=0)
+ LinearSVC(C=1.0, class_weight=None, dual=True, epsilon=0.1,
+ fit_intercept=True, intercept_scaling=1, loss='l2', multi_class='ovr',
+ penalty='l2', random_state=0, svr=False, tol=0.0001, verbose=0)
@larsmans Owner

It doesn't make sense to have a public svr attribute on a LinearSVC.

@amueller Owner

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
@@ -651,17 +660,20 @@ def fit(self, X, y):
self : object
Returns self.
"""
- self._enc = LabelEncoder()
- y = self._enc.fit_transform(y)
- if len(self.classes_) < 2:
- raise ValueError("The number of classes has to be greater than"
- " one.")
+ if not self.svr:
@larsmans Owner

This violates OO design principles. SVR logic should go in the LinearSVR class, classifier logic in LinearSVC. (Use the template method pattern.)

@jnothman Owner

linear_model.LogisticRegression also inherits from svm.BaseLibLinear so perhaps you're actually asking for the logic to be split between LinearSVR and a hypothetical svm.BaseLibLinearClassifier...

@larsmans Owner

Good point. I'd call it BaseLibLinearClf for short, but that is the way to go.

@luoq
luoq added a note

I don't think this is necessary. Currently BaseLibLinear handle both regression and classification like liblinear C library. We can view it as a direct python represetation of liblinear library. LogisticRegression, LinearSVC, LinearSVR just inhert from it and hide some parameters.

@jnothman Owner

Maybe this is why @larsmans singled out the data validation: it's not part of LibLinear.

Even so, the current design is unfortunate in storing attributes that are irrelevant (epsilon, class_weight, multi_class, perhaps loss). One solution to this -- cutting back BaseLibLinear's __init__ parameters to precisely the base parameters -- might involve the call to train looking more like:

train(X, y, eps=self.tol, bias=self._get_bias(), C=self.C,
        random_seed=rnd.randint(np.iinfo('i').max),
        **self._get_train_kwargs())

but I'm not convinced that's especially neat, and the real mess is in _get_solver_type and its shared error handling.

@jnothman Owner

In any case, self.class_weight_ should not be set on LinearSVR.

@larsmans Owner

We shouldn't simulate the LibLinear authors' architectural mistakes in our Python code. I know the LibSVM wrappers do that, but only because I never got round to refactoring that out.

@luoq
luoq added a note

Is it not acceptable to set a attribute not responding to a init argument? I'm not sure of this by reading Coding guidelines. It looks like setting a redundant ( add solver_type in BaseLibLinear) or unused (class_weight_ in LinearSVR and epsilon in LinearSVC) will not cause trouble for methods of BaseEstimator.

@amueller Owner

Parameters have to be init parameters for set_params and get_params to work.
I am not sure what you are trying to say, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@luoq

LogisticRegression, LinearSVC, LinearSVR are the top level classes. Parameter loss, epsilon, class_weight are not used in at least one class.

LogisticRegression : -loss -epsilon
LinearSVC : -epsilon
LinearSVR : -class_weight

sklearn/svm/base.py
@@ -612,8 +617,12 @@ def _get_solver_type(self):
if self.multi_class != 'ovr':
raise ValueError("`multi_class` must be one of `ovr`, "
"`crammer_singer`")
- solver_type = "P%s_L%s_D%d" % (
- self.penalty.upper(), self.loss.upper(), int(self.dual))
+ if self.svr:
@larsmans Owner

Same remark as below. This should be handled in subclasses; a base class should not be aware of this kind of details.

@luoq
luoq added a note

The name BaseLibLinear is misleading because it's an estimator in itself currently. In fact, we can expose it to toplevel like LinearSVR. Maybe the name LibLinear is more appropriate.

@luoq
luoq added a note

If we just want BaseLibLinear to be base, we need to move _get_solver_type, code for data validation to subclass and leave only fit(). We also need to handle epsilon, class_weight specially in different subclasses. Maybe we can build a dict containing all parameters in fit() of subclass and pass X,y,solver_type,dict to fit() in BaseLibLinear.

@jnothman Owner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
@@ -612,8 +617,12 @@ def _get_solver_type(self):
if self.multi_class != 'ovr':
raise ValueError("`multi_class` must be one of `ovr`, "
"`crammer_singer`")
- solver_type = "P%s_L%s_D%d" % (
- self.penalty.upper(), self.loss.upper(), int(self.dual))
+ if self.svr:
+ solver_type = "P%s_L%sR_D%d" % (
+ self.penalty.upper(), self.loss.upper(), int(self.dual))
+ else:
+ solver_type = "P%s_L%s_D%d" % (
+ self.penalty.upper(), self.loss.upper(), int(self.dual))
if not solver_type in self._solver_type_dict:
@luoq
luoq added a note

Is it ok to add an attribute self.solver_type_ here ?

@jnothman Owner

Yes, perhaps it's a useful attribute for users who might want reference to the Liblinear API....? In any case, I would think it's better to add it within fit().

@luoq
luoq added a note

solver_type is determinated by loss, penalty, dual, multi_class and is independent of data. Besides we want to check if the solver is implemented in liblinear at init.

@jnothman Owner

Even if so, you still need to check if it is implemented in fit: sklearn assumes parameters can change between construction and fit.

@luoq
luoq added a note

Is this done in BaseEstimator.set_params? If so, setting solver_type in init is inappropriate because set_params will not update it. Checking solver_type is also inappropriate in init because no check is done when changing parameters. Maybe we should only check type in fit.

@larsmans Owner

Yes, validation should be done in fit (sorry, hadn't seen this yet).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@luoq

@jnothman How would I incorporate your Pull Request #1873 ? I'm not clear about the workflow.

@jnothman
Owner

Well, I'm not certain it'll get accepted. It may be best just to ignore it for the moment, and when it (or this PR) gets accepted, the later-accepted change will have to run git rebase master and sort out any conflicts introduced in the meantime.

@larsmans
Owner

@luoq Do you keep force-pushing commits? I get remarks by email but I can't find them in the GitHub web ui.

@luoq

No,I haven't used push -f .

@larsmans
Owner

Ah, I see now, the comments were on the complete diff.

@amueller
Owner

@luoq yes, chaning the tests for input validation seems the right thing to do. The reason is that when using grid search, the parameter is not passed through init.
I'm not sure what github messed up this time but I can't see any comments on the diff :-/

@luoq

@amueller From e-mail notice, you have posted 5 comments. Only one is displayed in github web. Others seem talking about old commit that has been updated.

penalty and loss are different. From example in lasso we have l2 loss and l1 penalty. penalty means regularization here.

sklearn/svm/base.py
((8 lines not shown))
Returns
-------
self : object
Returns self.
"""
- self._enc = LabelEncoder()
- y = self._enc.fit_transform(y)
- if len(self.classes_) < 2:
- raise ValueError("The number of classes has to be greater than"
- " one.")
-
- X = atleast2d_or_csr(X, dtype=np.float64, order="C")
+ X, y, class_weight_ = self._transform_data(X, y)
@jnothman Owner

This is relevant to the classifier subclasses only, and class_weight doesn't need an underscore on the end. The underscore on class attributes is used to indicate output from the fit process (and thus, you should still be setting self.class_weight_ for users to access).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
((11 lines not shown))
self.raw_coef_ = train(X, y, self._get_solver_type(), self.tol,
+ # LinearSVC, LogisticRegression have no epsilon attribute
+ self.epsilon if hasattr(self, 'epsilon') else 0.1,
@jnothman Owner

You can use getattr(self, 'epsilon', 0.1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
@@ -698,6 +636,46 @@ def fit(self, X, y):
return self
+ def _get_bias(self):
+ if self.fit_intercept:
+ return self.intercept_scaling
+ else:
+ return -1.0
+
+
+class LibLinearRegressorMixin(LibLinearMixin):
+ def _transform_data(self, X, y):
+ X = atleast2d_or_csr(X, dtype=np.float64, order="C")
@jnothman Owner

This is also validation. Why not just one _validate_data if this is the way to go about it?

Personally, I think this is not a good use of the "template method" pattern (@larsmans?): the subclasses should just pass their superclass X and y validated for their purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman jnothman commented on the diff
sklearn/linear_model/logistic.py
@@ -99,10 +99,27 @@ def __init__(self, penalty='l2', dual=False, tol=1e-4, C=1.0,
fit_intercept=True, intercept_scaling=1, class_weight=None,
random_state=None):
- super(LogisticRegression, self).__init__(
- penalty=penalty, dual=dual, loss='lr', tol=tol, C=C,
- fit_intercept=fit_intercept, intercept_scaling=intercept_scaling,
- class_weight=class_weight, random_state=None)
+ self.penalty = penalty
+ self.dual = dual
@jnothman Owner

I'm not sure what I think of this. It's a bit too much the opposite of what you first proposed: all the liblinear estimators have parameters tol, C, fit_intercept, intercept_scaling, random_state. This implementation would suggest they share nothing, and then when you come to LibLinearMixin.fit it assumes the presence of all of these attributes, but you have to look in a completely different place to see that they were set. I think that's also an indicator of bad OOP design.

But I think @larsmans should comment on design patterns.

@luoq
luoq added a note

LibLinearMixin is used just to reuse the code of fit method for LinearSVR, LinearSVC, LogisticRegression. We can set common attributes in LibLinearMixin, and then add other attribute in subclass. But I think that's quite unclear because we need to manually calculate what is common and what need to be added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
sklearn/svm/base.py
((6 lines not shown))
print('[LibLinear]', end='')
- # LibLinear wants targets as doubles, even for classification
- y = np.asarray(y, dtype=np.float64).ravel()
+ # epsilon, tol are named p, eps respectively in struct parameter of C source code
self.raw_coef_ = train(X, y, self._get_solver_type(), self.tol,
@jnothman Owner

I think using something like self._train_kwargs() to get a dict of the estimator-specific arguments would be neater; you would then put all the default values in the pyx functions.

@luoq
luoq added a note

If we return all parameters for liblinear in _train_kwargs which is neat, then there will be much duplication of code in subclass, like _get_bias got called in every subclass.

We can also only return( or not return) parameters that need special treatment, which I think is what you suggest. But I think it is dirty to treat parameters unequally in the first place.

Of cause, current implementation is also dirty in the same sense. And return class_weight_ in _transform_data is dirty too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@amueller
Owner

@luoq That is why I said slack penalty ;) Maybe a bad wording in context of the liblinear parameters. Anyhow, what I meant was that liblinear uses the squared hinge loss by default and libsvm the standard hinge-loss. I was wondering whether something similar holds true for the epsilon insensitive case.

@luoq

liblinear do not have a default for regression separately. The default is just for SVC. It looks like l1 loss SVR in liblinear and epsilon-SVR have the same formulation.

@luoq

For LinearSVR, LinearSVC, LogisticRegression, there should be no unrelevant( epsilon for LinearSVC) and redundant( loss in LogisticRegression) __init__ parameters. Currently the parameters in __init__almost satisfy this condition. Except in LinearSVC when multi_class is crammer_singer, penalty, loss, dual is unrelevant.

In the other end, to call underlying liblinear library. We must

  1. map some attributes to solver_type
  2. convert some attributes to parameters needed by liblinear. This is complicated by some factors
    1. different name( epsilon -> p) and different representation ( fit_intercept=False -> bias<0)
    2. handle unrelevant parameters( add p in LinearSVC).
    3. For classification, calculate class_weight_ from class_weight and y
  3. convert X,y to double

I think the sole point of using inheritance here is to put common code in base class, and the conversions in the list before in subclass. As for OOP design, I don't have much pratice in it. I suspect somewhere is always dirty when you want to write less code in this case.

@jnothman
Owner

You can rebase on master now if you want to take advantage of the pyx refactor.

luoq added some commits
@luoq luoq ENH: Add LinearSVR
Fix #1473
8e9dabf
@luoq luoq remove class_weight parameter 7ffddc8
@luoq luoq fix random_seed in test ddbfb7d
@luoq luoq format cleaning and comment for different parameter name e627453
@luoq luoq remove unnecessary ravel 1935cad
@luoq luoq update docstring for hinge_loss 6e5993f
@luoq luoq hide epsilon and svr in LinearSVC e575007
@luoq luoq Revert "update docstring for hinge_loss"
This reverts commit f7ef3b9.
3c70250
@luoq luoq test against original liblinear 7957d26
@luoq luoq REFACTOR liblinear based estimators
BaseLibLinear -> LibLinearMixin
b95c67f
@luoq luoq comment for parameters not shared by all 7fbd581
@luoq luoq LibLinearMixin should inherit from object 113c121
@luoq luoq solver_type check for liblinear based estimators
1. solver_type check is done only in fit
2. more informative error message
3. add test for parameter combinations for LinearSVR
4. clean parameter combination test for LinearSVC and LogisticRegression
52371fb
@luoq luoq mv _solver_type_dict to subclass
Before this, you can set loss = "lr", "l1r", "lr2" in LinearSVC to do
logistic regression and linear svr.
35c552b
@luoq luoq clean preparation for fitting
1. combine _transfrom_data and _validate_data to _prepare_fit
2. rely on self.class_weight_ for class weight
b2bf3a9
@luoq luoq fix shape of LinearSVR prediction c739043
@luoq luoq Add blank lines ab2bb60
@jnothman
Owner

You can simplify the call to liblinear.train_wrap just a little by moving the default values for to the pyx... This should make it possible for the Python side to be more neatly object-oriented.

@luoq

I am not clear with your point here.
If the call to `train_wrap' is maded in LibLinearMixin, we will always need to check if the object has epsilon, class_weight or not. And the default values set in getattr are just placehoiders, you can change them to any value with the same type.
And the other default values are all set in LinearSVR, LinearSVC, LogisticRegression. Currently there are no default values in train_wrap in liblinear.pyx.

Maybe you can provide a patch to explain your point.

@larsmans
Owner

Closing in favor of #3877; sorry @luoq!

@larsmans larsmans closed this
@luoq luoq deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 30, 2013
  1. @luoq

    ENH: Add LinearSVR

    luoq authored
    Fix #1473
  2. @luoq

    remove class_weight parameter

    luoq authored
  3. @luoq

    fix random_seed in test

    luoq authored
  4. @luoq
  5. @luoq

    remove unnecessary ravel

    luoq authored
  6. @luoq

    update docstring for hinge_loss

    luoq authored
  7. @luoq

    hide epsilon and svr in LinearSVC

    luoq authored
  8. @luoq

    Revert "update docstring for hinge_loss"

    luoq authored
    This reverts commit f7ef3b9.
  9. @luoq

    test against original liblinear

    luoq authored
  10. @luoq

    REFACTOR liblinear based estimators

    luoq authored
    BaseLibLinear -> LibLinearMixin
  11. @luoq
  12. @luoq
  13. @luoq

    solver_type check for liblinear based estimators

    luoq authored
    1. solver_type check is done only in fit
    2. more informative error message
    3. add test for parameter combinations for LinearSVR
    4. clean parameter combination test for LinearSVC and LogisticRegression
  14. @luoq

    mv _solver_type_dict to subclass

    luoq authored
    Before this, you can set loss = "lr", "l1r", "lr2" in LinearSVC to do
    logistic regression and linear svr.
  15. @luoq

    clean preparation for fitting

    luoq authored
    1. combine _transfrom_data and _validate_data to _prepare_fit
    2. rely on self.class_weight_ for class weight
  16. @luoq

    fix shape of LinearSVR prediction

    luoq authored
  17. @luoq

    Add blank lines

    luoq authored
Something went wrong with that request. Please try again.