Linear Selection Method fails with Python 3.12

**Context**:
With Python 3.12, following error is encountered while running the Linear method.

```
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/sample.py", line 13, in <module>
    selector = Selective(SelectionMethod.Linear(num_features=3, regularization="none"))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/Documents/projects/selective/feature/selector.py", line 532, in __init__
    self._imp = _Linear(self.seed, self.selection_method.num_features,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/Documents/projects/selective/feature/linear.py", line 32, in __init__
    "classification_none": LogisticRegression(random_state=self.seed,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LogisticRegression.__init__() got an unexpected keyword argument 'multi_class'
```

This is because the latest scikit-learn (1.9.0) does not suppor the argument 'multi_class' for LogisticRegression.

**Replication:**
1) Create a new conda environment with Python 3.12 and install the requirements using the requirements.txt file from selective Github page
2) Run the following sample code which is taken from the selective Github page:

```
from feature.utils import get_data_label
from feature.selector import Selective, SelectionMethod

# Data
data, label = get_data_label(fetch_california_housing())

# Feature selectors from simple to more complex
selector = Selective(SelectionMethod.Linear(num_features=3, regularization="none"))

# Feature reduction
subset = selector.fit_transform(data, label)
print("Reduction:", list(subset.columns))
print("Scores:", list(selector.get_absolute_scores()))
```

The above error will be encountered.

**Fix1:**
To fix the above error, update feature/linear.py to instantiate LogisticRegression without argument 'multi_class'.
The updated block of code should be like this:
```
# Implementor factory
self.factory = {"regression_none": LinearRegression(),
                "regression_lasso": Lasso(random_state=self.seed),
                "regression_ridge": Ridge(random_state=self.seed),
                "classification_none": LogisticRegression(random_state=self.seed, solver="liblinear"),
                "classification_lasso": LogisticRegression(random_state=self.seed, penalty='l1',
                                                           solver="liblinear"),
                "classification_ridge": RidgeClassifier(random_state=self.seed)}
```

**Tests:**
The above fix is backward compatible to Python 3.10. After applying the above fix, run the above sample code in the virtual environment with Python 3.10 and also in virtual environment with Python 3.12. You should see identical results. 

Output of above sample script using Python 3.10:
Reduction: ['MedInc', 'AveBedrms', 'Longitude']
Scores: [np.float64(0.4366932931343246), np.float64(0.009435778033238064), np.float64(0.10732204139090426), np.float64(0.6450656935198131), np.float64(3.9763894212480055e-06), np.float64(0.003786542654971019), np.float64(0.4213143775271428), np.float64(0.43451375467477704)]

Output of above sample script using Python 3.12:
Reduction: ['MedInc', 'AveBedrms', 'Longitude']
Scores: [np.float64(0.43669329313432437), np.float64(0.009435778033238086), np.float64(0.10732204139090418), np.float64(0.6450656935198129), np.float64(3.9763894212358625e-06), np.float64(0.003786542654970882), np.float64(0.4213143775271443), np.float64(0.43451375467477743)]


A slight change in precision might be possible because the two environments have different scikit-learn and numpy versions.


If we only remove the argument, then all test cases pass with Python 3.10 but 4 test cases fail with Python 3.12 as shown below:

======================================================================
```
ERROR: test_linear_classif_top_k (tests.test_linear.TestLinear.test_linear_classif_top_k)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_linear.py", line 69, in test_linear_classif_top_k
    selector.fit(data, label)
  File "/home/rbaral/Documents/projects/selective/feature/selector.py", line 561, in fit
    self._imp.fit(data, labels)
  File "/home/rbaral/Documents/projects/selective/feature/linear.py", line 49, in fit
    self.imp.fit(X=data, y=labels)
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py", line 1488, in fit
    raise ValueError(
ValueError: The 'liblinear' solver does not support multiclass classification (n_classes >= 3). Either use another solver or wrap the estimator in a OneVsRestClassifier to keep applying a one-versus-rest scheme.

======================================================================
ERROR: test_linear_classif_top_k_all (tests.test_linear.TestLinear.test_linear_classif_top_k_all)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_linear.py", line 106, in test_linear_classif_top_k_all
    selector.fit(data, label)
  File "/home/rbaral/Documents/projects/selective/feature/selector.py", line 561, in fit
    self._imp.fit(data, labels)
  File "/home/rbaral/Documents/projects/selective/feature/linear.py", line 49, in fit
    self.imp.fit(X=data, y=labels)
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py", line 1488, in fit
    raise ValueError(
ValueError: The 'liblinear' solver does not support multiclass classification (n_classes >= 3). Either use another solver or wrap the estimator in a OneVsRestClassifier to keep applying a one-versus-rest scheme.

======================================================================
ERROR: test_linear_classif_top_percentile (tests.test_linear.TestLinear.test_linear_classif_top_percentile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_linear.py", line 81, in test_linear_classif_top_percentile
    selector.fit(data, label)
  File "/home/rbaral/Documents/projects/selective/feature/selector.py", line 561, in fit
    self._imp.fit(data, labels)
  File "/home/rbaral/Documents/projects/selective/feature/linear.py", line 49, in fit
    self.imp.fit(X=data, y=labels)
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py", line 1488, in fit
    raise ValueError(
ValueError: The 'liblinear' solver does not support multiclass classification (n_classes >= 3). Either use another solver or wrap the estimator in a OneVsRestClassifier to keep applying a one-versus-rest scheme.

======================================================================
ERROR: test_linear_classif_top_percentile_all (tests.test_linear.TestLinear.test_linear_classif_top_percentile_all)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_linear.py", line 93, in test_linear_classif_top_percentile_all
    selector.fit(data, label)
  File "/home/rbaral/Documents/projects/selective/feature/selector.py", line 561, in fit
    self._imp.fit(data, labels)
  File "/home/rbaral/Documents/projects/selective/feature/linear.py", line 49, in fit
    self.imp.fit(X=data, y=labels)
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py", line 1488, in fit
    raise ValueError(
ValueError: The 'liblinear' solver does not support multiclass classification (n_classes >= 3). Either use another solver or wrap the estimator in a OneVsRestClassifier to keep applying a one-versus-rest scheme.

======================================================================
ERROR: test_process_category_data (tests.test_text.TestText.test_process_category_data)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_text.py", line 58, in test_process_category_data
    matrix = process_category_data(data, categories)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/Documents/projects/selective/feature/text_based.py", line 650, in process_category_data
    .groupby(level=0, axis=0)
     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/pandas/util/_decorators.py", line 336, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
TypeError: DataFrame.groupby() got an unexpected keyword argument 'axis'

======================================================================
FAIL: test_benchmark_classification (tests.test_benchmark.TestBenchmark.test_benchmark_classification)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 164, in test_benchmark_classification
    self.assertListAlmostEqual([0.28992981466266715, 0.5607438535573831, 0.2622507287680856, 0.04272068858604694],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28992981466266715 != 0 within 0.01 delta (0.28992981466266715 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_benchmark.TestBenchmark.test_benchmark_classification_cv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 204, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.22327603204146848, 0.03543066514916661, 0.26254667473769594, 0.506591069316828],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.22327603204146848 != 0.0 within 0.01 delta (0.22327603204146848 difference)

======================================================================
FAIL: test_benchmark_classification (tests.test_parallel.TestParallel.test_benchmark_classification)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 89, in test_benchmark_classification
    self.assertListAlmostEqual([0.289930, 0.560744, 0.262251, 0.042721],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28993 != 0 within 0.01 delta (0.28993 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_parallel.TestParallel.test_benchmark_classification_cv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 154, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.223276, 0.035431, 0.262547, 0.506591],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.223276 != 0.0 within 0.01 delta (0.223276 difference)

----------------------------------------------------------------------
Ran 115 tests in 287.773s

FAILED (failures=4, errors=5)
```

**Fix2**: Using solver lbfgs instead of liblinear in features/linear.py:

```
# Implementor factory
self.factory = {"regression_none": LinearRegression(),
                "regression_lasso": Lasso(random_state=self.seed),
                "regression_ridge": Ridge(random_state=self.seed),
                "classification_none": LogisticRegression(random_state=self.seed, solver="lbfgs"),
                "classification_lasso": LogisticRegression(random_state=self.seed, penalty='l1',
                                                           solver="lbfgs"),
                "classification_ridge": RidgeClassifier(random_state=self.seed)}
```

The Python 3.10 results in 4 test cases failure:

```
======================================================================
FAIL: test_benchmark_classification (tests.test_benchmark.TestBenchmark)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 164, in test_benchmark_classification
    self.assertListAlmostEqual([0.28992981466266715, 0.5607438535573831, 0.2622507287680856, 0.04272068858604694],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28992981466266715 != 5.4262150328554526e-15 within 0.01 delta (0.2899298146626617 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_benchmark.TestBenchmark)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 204, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.22327603204146848, 0.03543066514916661, 0.26254667473769594, 0.506591069316828],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.22327603204146848 != 5.329070518200751e-15 within 0.01 delta (0.22327603204146315 difference)

======================================================================
FAIL: test_benchmark_classification (tests.test_parallel.TestParallel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 89, in test_benchmark_classification
    self.assertListAlmostEqual([0.289930, 0.560744, 0.262251, 0.042721],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28993 != 5.4262150328554526e-15 within 0.01 delta (0.2899299999999946 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_parallel.TestParallel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 154, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.223276, 0.035431, 0.262547, 0.506591],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.223276 != 5.329070518200751e-15 within 0.01 delta (0.22327599999999467 difference)

----------------------------------------------------------------------
Ran 115 tests in 373.279s

FAILED (failures=4)

```

The Python 3.12 also shows 4 test cases failure:

```
======================================================================
ERROR: test_process_category_data (tests.test_text.TestText.test_process_category_data)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_text.py", line 58, in test_process_category_data
    matrix = process_category_data(data, categories)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/Documents/projects/selective/feature/text_based.py", line 650, in process_category_data
    .groupby(level=0, axis=0)
     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rbaral/miniconda3/envs/env_selective12/lib/python3.12/site-packages/pandas/util/_decorators.py", line 336, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
TypeError: DataFrame.groupby() got an unexpected keyword argument 'axis'

======================================================================
FAIL: test_benchmark_classification (tests.test_benchmark.TestBenchmark.test_benchmark_classification)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 164, in test_benchmark_classification
    self.assertListAlmostEqual([0.28992981466266715, 0.5607438535573831, 0.2622507287680856, 0.04272068858604694],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28992981466266715 != 5.4262150328554526e-15 within 0.01 delta (0.2899298146626617 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_benchmark.TestBenchmark.test_benchmark_classification_cv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_benchmark.py", line 204, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.22327603204146848, 0.03543066514916661, 0.26254667473769594, 0.506591069316828],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.22327603204146848 != 5.329070518200751e-15 within 0.01 delta (0.22327603204146315 difference)

======================================================================
FAIL: test_benchmark_classification (tests.test_parallel.TestParallel.test_benchmark_classification)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 89, in test_benchmark_classification
    self.assertListAlmostEqual([0.289930, 0.560744, 0.262251, 0.042721],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.28993 != 5.4262150328554526e-15 within 0.01 delta (0.2899299999999946 difference)

======================================================================
FAIL: test_benchmark_classification_cv (tests.test_parallel.TestParallel.test_benchmark_classification_cv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbaral/Documents/projects/selective/tests/test_parallel.py", line 154, in test_benchmark_classification_cv
    self.assertListAlmostEqual([0.223276, 0.035431, 0.262547, 0.506591],
  File "/home/rbaral/Documents/projects/selective/tests/test_base.py", line 23, in assertListAlmostEqual
    self.assertAlmostEqual(val, list2[index], delta=0.01)
AssertionError: 0.223276 != 5.329070518200751e-15 within 0.01 delta (0.22327599999999467 difference)

----------------------------------------------------------------------
Ran 115 tests in 405.695s

FAILED (failures=4, errors=1)

```


Hence, we also need to update the test cases. @takojunior @skadio 

Navigation Menu

Uh oh!

Linear Selection Method fails with Python 3.12 #24

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Linear Selection Method fails with Python 3.12 #24

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions