`IterativeImputer` skip iterative part if `keep_empty_features` is set to `True`

[xuefeng-xu]

Describe the bug

The mask is set to all True, so that the iterative imputation will be skipped.

scikit-learn/sklearn/impute/_iterative.py

Lines 649 to 651 in a4ebe19

	# mark empty features as not missing and keep the original
	# imputation
	mask_missing_values[:, valid_mask] = True

Steps/Code to Reproduce

import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

imp = IterativeImputer(keep_empty_features=True)
X = [[np.nan, 0, 1], [2, np.nan, 3], [4, 5, np.nan]]
print(imp.fit_transform(X))
print(imp.imputation_sequence_)

Expected Results

Should act the same as keep_empty_features=False

Actual Results

# transformed X
[[3.  0.  1. ]
 [2.  2.5 3. ]
 [4.  5.  2. ]]

# imputation_sequence_
[]

Versions

System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:38:11)  [Clang 14.0.6 ]
executable: /Users/xxf/miniconda3/envs/sklearn-env/bin/python
   machine: macOS-14.5-arm64-arm-64bit

Python dependencies:
      sklearn: 1.6.dev0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: 3.0.8
       pandas: 2.1.0
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/xxf/miniconda3/envs/sklearn-env/lib/libopenblas.0.dylib
        version: 0.3.23
threading_layer: openmp
   architecture: VORTEX

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/xxf/miniconda3/envs/sklearn-env/lib/libomp.dylib
        version: None

[Anurag-Varma]

/take

Can any member check this and remove needs triage.
Then assign me the issue.

[Anurag-Varma]

/take

[Anurag-Varma]

@xuefeng-xu

When keep_empty_features=False
The output is:

# transformed X
[[0.03425361 0.         1.        ]
 [2.         2.47849272 3.        ]
 [4.         5.         5.03476137]]
 
# imputation_sequence_
[_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2], dtype=int64), estimator=BayesianRidge()), _ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1], dtype=int64), estimator=BayesianRidge())]

But what is the expected result when keep_empty_features=True ?

According to your output, it is replacing the nan values as:

# transformed X
[[3.  0.  1. ]
 [2.  2.5 3. ]
 [4.  5.  2. ]]
 
# imputation_sequence_
[]

[xuefeng-xu]

@Anurag-Varma I think the output should be the same as keep_empty_features=False.

[Anurag-Varma]

@xuefeng-xu I think it's working as required

keep_empty_featuresbool, default=False

#Test this example

import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# Example data with a feature that has all missing values
X = [[np.nan, 0, 1], [np.nan, 2, 3], [np.nan, 4, 5]]

imp = IterativeImputer(keep_empty_features=True)
X_imputed = imp.fit_transform(X)

print(X_imputed)

You can test this code for keep_empty_features= True and False

When True:

#Output is
[[0. 0. 1.]
 [0. 2. 3.]
 [0. 4. 5.]]

When False:

#Output is:
[[0. 1.]
 [2. 3.]
 [4. 5.]]

[xuefeng-xu]

@Anurag-Varma For data you provided, there's no need to fit any estimater since it just need to discard the first column or replace it with zeros.

For the data sample I provided, do you think IterativeImputer should behave the same?

[Anurag-Varma]

@xuefeng-xu I understand it now.

So incase features that consist exclusively of missing values then it will return 0 if its set True.

Incase of False, it removes that feature and the returned array size may vary. But if True then it returns the same size but with 0 in the example which i provided above.

But for your example, it's returning different output when its set to True.

I think, it should return the same output as when it was set to False.

[xuefeng-xu]

@Anurag-Varma Thank you! I currently don't have time to fix this, if you want to do it, feel free to open a PR.

[adrinjalali]

so for completeness, this is current behavior:

In [8]: import numpy as np
   ...: from sklearn.experimental import enable_iterative_imputer
   ...: from sklearn.impute import IterativeImputer
   ...: 
   ...: imp = IterativeImputer(keep_empty_features=True)
   ...: X = [[np.nan, 0, 1], [2, np.nan, 3], [4, 5, np.nan]]
   ...: print(imp.fit_transform(X))
   ...: print(imp.imputation_sequence_)
[[3.  0.  1. ]
 [2.  2.5 3. ]
 [4.  5.  2. ]]
[]

In [9]: import numpy as np
   ...: from sklearn.experimental import enable_iterative_imputer
   ...: from sklearn.impute import IterativeImputer
   ...: 
   ...: imp = IterativeImputer(keep_empty_features=False)
   ...: X = [[np.nan, 0, 1], [2, np.nan, 3], [4, 5, np.nan]]
   ...: print(imp.fit_transform(X))
   ...: print(imp.imputation_sequence_)
[[0.03425361 0.         1.        ]
 [2.         2.47849272 3.        ]
 [4.         5.         5.03476137]]
[_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=0, neighbor_feat_idx=array([1, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=1, neighbor_feat_idx=array([0, 2]), estimator=BayesianRidge()), 
_ImputerTriplet(feat_idx=2, neighbor_feat_idx=array([0, 1]), estimator=BayesianRidge())]

And I agree it seems they need to be the same.

A PR is welcome.