ENH Use scipy Yeo-Johnson implementation in PowerTransformer for scipy >= 1.9 by yaichm · Pull Request #31227 · scikit-learn/scikit-learn

yaichm · 2025-04-19T12:21:19Z

Reference Issues/PRs

This PR is finalizing #27818 to close #26308 and also fix the warning issue reported in #23319 (comment)

Close #27818

What does this implement/fix? Explain your changes.

Use scipy.stats.yeojohnson instead of our own implementation as @lorentzenchr suggested.

Any other comments?

github-actions · 2025-04-19T12:22:17Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: ab2d918. Link to the linter CI: here}

lorentzenchr

@yaichm @smarie @eroussama @Dlimim @HamzaLuffy @AmineHannoun @xuefeng-xu thanks for this nice PR. Seems solid. Only a few nitpicks.

lorentzenchr · 2025-04-21T11:46:13Z

        return scipy.sparse.linalg.cg(A, b, **kwargs)


+#TODO : remove this when minimum version of scipy >= 1.9.0


Suggested change

#TODO : remove this when minimum version of scipy >= 1.9.0

# TODO: Remove this when minimum version of scipy >= 1.9.0, also remove

# sklearn.preprocessing._yeo_johnson_optimize and use scipy.stats.yeojohnson instead.

I guess we still need _yeo_johnson_optimize since we need to get rid of the nan, see also _box_cox_optimize

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

…angelog

lorentzenchr · 2025-04-25T10:49:33Z

@smarie Could you instruct how to use linting tools and the CI reports?

smarie · 2025-04-28T12:06:08Z

@smarie Could you instruct how to use linting tools and the CI reports?

Yes, good catch thanks @lorentzenchr . We reviewed the pre-commit tool together from https://pre-commit.com/#4-optional-run-against-all-the-files

ogrisel

LGTM, just a nitpick.

ogrisel

Actually, there are other small details to fix:

smarie · 2025-05-05T07:28:22Z

@@ -0,0 +1,5 @@
+- Fixed numerical stability (mostly overflows) of the Yeo-Johnson transform with


@ogrisel @lorentzenchr I suggest to explicitly tell about the replacement of implem, what do you think ? Like this:

Suggested change

- Fixed numerical stability (mostly overflows) of the Yeo-Johnson transform with

- Now using ``scipy.stats.yeojohnson`` instead of our own implementation of the Yeo-Johnson transform. Fixed numerical stability (mostly overflows) with

We all agree, but I do not have the permission to accept this suggestion myself. @yaichm has to do it I guess.

ogrisel · 2025-05-05T13:00:05Z

Once done, we can merge.

lesteve · 2025-05-05T14:20:57Z

+  Initial PR by :user: `Xuefeng Xu <@xuefeng-xu>` completed by `Mohamed Yaich<@yaichm>`,
+  `Oussama Er-rabie<@eroussama>`, `Mohammed Yaslam Dlimi<@Dlimim>`,
+  `Hamza Zaroual<@HamzaLuffy>`, `Amine Hannoun<@AmineHannoun>` and `Sylvain Marié<@smarie>`.


I think you need something like this, please double-check the rendering in the changelog by clicking on "check the rendered doc" CI status at the bottom of the PR.

Suggested change

Initial PR by :user: `Xuefeng Xu <@xuefeng-xu>` completed by `Mohamed Yaich<@yaichm>`,

`Oussama Er-rabie<@eroussama>`, `Mohammed Yaslam Dlimi<@Dlimim>`,

`Hamza Zaroual<@HamzaLuffy>`, `Amine Hannoun<@AmineHannoun>` and `Sylvain Marié<@smarie>`.

Initial PR by :user:`Xuefeng Xu <xuefeng-xu>` completed by :user:`Mohamed Yaich <yaichm>`,

:user:`Oussama Er-rabie <eroussama>`, :user:`Mohammed Yaslam Dlimi <Dlimim>`,

:user:`Hamza Zaroual <HamzaLuffy>`, :user:`Amine Hannoun <AmineHannoun>` and :user:`Sylvain Marié <smarie>`.

The current rendering is slightly off, see this

Thanks, the changelog rendering looks good now see this

lesteve · 2025-05-05T15:57:00Z

Thanks for your PR, merging!

smarie · 2025-05-05T20:55:07Z

Congratulations, team ! And thanks to the reviewers

Final changes in yeojohnson and its tests

9dcc8de

github-actions Bot added module:preprocessing module:utils labels Apr 19, 2025

Merge branch 'scikit-learn:main' into feature/yeo_johnson1

3ec18c1

lorentzenchr approved these changes Apr 21, 2025

View reviewed changes

lorentzenchr added this to the 1.7 milestone Apr 21, 2025

xuefeng-xu reviewed Apr 21, 2025

View reviewed changes

Comment thread doc/whats_new/upcoming_changes/sklearn.utils/27818.fix.rst Outdated

yaichm and others added 5 commits April 21, 2025 22:56

Merge branch 'scikit-learn:main' into feature/yeo_johnson1

572b552

Update sklearn/utils/fixes.py

accf689

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

Apply suggestions from code review

ea3b804

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

Merge branch 'scikit-learn:main' into feature/yeo_johnson1

32b9a61

skip test_power_transformer_no_warnings for scipy>=1.9, and update ch…

006e592

…angelog

lorentzenchr reviewed Apr 25, 2025

View reviewed changes

Comment thread doc/whats_new/upcoming_changes/sklearn.utils/27818.fix.rst Outdated

yaichm and others added 3 commits April 25, 2025 21:54

Merge branch 'scikit-learn:main' into feature/yeo_johnson1

2b50f38

Move changelog entry to correct folder and skip test if SciPy < 1.9

ca119fc

Fix lint errors and rename changelog file to match PR number

795c31d

smarie reviewed Apr 28, 2025

View reviewed changes

Comment thread doc/whats_new/upcoming_changes/sklearn.preprocessing/31227.fix.rst Outdated

smarie reviewed Apr 28, 2025

View reviewed changes

Comment thread sklearn/utils/fixes.py Outdated

smarie reviewed Apr 28, 2025

View reviewed changes

Comment thread sklearn/preprocessing/tests/test_data.py Outdated

yaichm and others added 2 commits April 29, 2025 00:54

Merge branch 'main' into feature/yeo_johnson1

2b5fcf5

Improve changelog and skip the test if scipy < 1.12

e8f2145

ogrisel approved these changes Apr 29, 2025

View reviewed changes

Comment thread sklearn/utils/fixes.py Outdated

Comment thread sklearn/utils/fixes.py Outdated

ogrisel reviewed Apr 29, 2025

View reviewed changes

Comment thread sklearn/preprocessing/tests/test_data.py Outdated

Comment thread doc/whats_new/upcoming_changes/sklearn.preprocessing/31227.fix.rst Outdated

Mohamed Yaich and others added 4 commits April 29, 2025 22:26

Update SciPy version and fix warning check

67305ce

Merge branch 'main' into feature/yeo_johnson1

1beacf3

Merge branch 'main' into feature/yeo_johnson1

a713b07

Fix contributor names in Yeo-Johnson PowerTransformer changelog

838c27c

smarie reviewed May 5, 2025

View reviewed changes

Merge branch 'main' into feature/yeo_johnson1

f63cc95

Update changelog

309a425

lesteve reviewed May 5, 2025

View reviewed changes

Update changelog

ab2d918

lesteve changed the title ~~ENH: Finalize Yeo-Johnson implementation with scipy.stats in PowerTransformer (#27818)~~ ENH Use scipy Yeo-Johnson implementation in PowerTransformer for scipy >= 1.9 (#27818) May 5, 2025

lesteve changed the title ~~ENH Use scipy Yeo-Johnson implementation in PowerTransformer for scipy >= 1.9 (#27818)~~ ENH Use scipy Yeo-Johnson implementation in PowerTransformer for scipy >= 1.9 May 5, 2025

lesteve enabled auto-merge (squash) May 5, 2025 15:56

lesteve merged commit 6c1d33f into scikit-learn:main May 5, 2025

lesteve mentioned this pull request May 5, 2025

Yeo-Johnson Power Transformer gives Numpy warning (and raises scipy.optimize._optimize.BracketError in some cases) #23319

Closed

pattplatt mentioned this pull request Jun 24, 2025

[MNT] Update scikit-learn requirement from <1.7.0,>=1.0.0 to >=1.0.0,<1.8.0 in the python-packages group aeon-toolkit/aeon#2907

Merged

		return scipy.sparse.linalg.cg(A, b, **kwargs)


		#TODO : remove this when minimum version of scipy >= 1.9.0

	#TODO : remove this when minimum version of scipy >= 1.9.0
	# TODO: Remove this when minimum version of scipy >= 1.9.0, also remove
	# sklearn.preprocessing._yeo_johnson_optimize and use scipy.stats.yeojohnson instead.

		@@ -0,0 +1,5 @@
		- Fixed numerical stability (mostly overflows) of the Yeo-Johnson transform with

	- Fixed numerical stability (mostly overflows) of the Yeo-Johnson transform with
	- Now using ``scipy.stats.yeojohnson`` instead of our own implementation of the Yeo-Johnson transform. Fixed numerical stability (mostly overflows) with

Uh oh!

Uh oh!

Conversation

yaichm commented Apr 19, 2025 • edited by lesteve Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

github-actions Bot commented Apr 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lorentzenchr Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

xuefeng-xu Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lorentzenchr commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

smarie commented Apr 28, 2025

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

smarie May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel May 5, 2025

Choose a reason for hiding this comment

Uh oh!

ogrisel commented May 5, 2025

Uh oh!

lesteve May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lesteve May 5, 2025

Choose a reason for hiding this comment

Uh oh!

lesteve commented May 5, 2025

Uh oh!

smarie commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yaichm commented Apr 19, 2025 •

edited by lesteve

Loading

github-actions Bot commented Apr 19, 2025 •

edited

Loading

smarie May 5, 2025 •

edited

Loading

lesteve May 5, 2025 •

edited

Loading