Skrub release 0.7.0

[rcap107]

Release 0.7.0

✨ Highlights

• Data Ops can now be tuned with Optuna.
• It is now possible to pass extra named arguments to an estimator through DataOps.skb.apply.
• The TableReport now supports numpy arrays.
• The minimum supported version of Python has been increased to 3.10, and the minimum supported versions of scikit-learn and requests are now 1.4.2 and 2.27.1 respectively.
• Added support for the upcoming Pandas 3.0.
• A lot of bugs have been fixed.

16 new contributors participated in this release 🎉

New features

• It is now possible to tune the choices in a DataOp with Optuna. Allow setting dataop choices with optuna #1661 by @jeromedockes.
• DataOp.skb.apply now allows passing extra named arguments to the estimator's methods through the parameters fit_kwargs, predict_kwargs etc. allow passing kwargs in .skb.apply() #1642 by @jeromedockes.
• TableReport now displays the mean statistic for boolean columns. Add Mean to boolean-type columns in Table Report #1647 by @abenechehab.
• DataOp.skb.get_vars allows inspecting all the variables, or all the named dataops, in a DataOp. This lets us easily know what keys should be present in the environment dictionary we pass to DataOp.skb.eval or to SkrubLearner.fit, SkrubLearner.predict, etc. add .skb.get_vars() #1646 by @jeromedockes.
• DataOp.skb.iter_cv_splits iterates over the training and testing environments produced by a CV splitter -- similar to DataOp.skb.train_test_split but for multiple cross-validation splits. add .skb.iter_cv_splits() #1653 by @jeromedockes.
• TableReport now supports np.array. FEAT - add support for numpy arrays to the TableReport #1676 by @Nismamjad1.
• DataOp.skb.full_report now accepts a new parameter, title, that is displayed in the html report. Feat: Add title to report #1654 by @MarieSacksick.
• TableReport now includes the open_tab parameter, which lets the user select which tab should be opened when the TableReport is rendered. ENH - Adding a parameter to select the default TableReport tab to show #1737 by @rcap107.

Changes

• The minimum supported version of Python has been increased to 3.10. Additionally, the minimum supported versions of scikit-learn and requests are 1.4.2 and 2.27.1 respectively. Support for python 3.14 has been added. MAINT - Bumping minimum python and sklearn requirements #1572 by @rcap107.
• The DataOp.skb.full_report method now deletes reports created with output_dir=None after 7 days. FEAT - Pruning skrub-data folder to remove old data ops execution reports #1657 by @dierickxsimon
• The tabular_pipeline uses a SquashingScaler instead of a StandardScaler for centering and scaling numerical features when linear models are used. Change StandardScaler to the SquashScaler for the tabular_pipline #1644 by @dierickxsimon.
• The transformer ToFloat, previously called ToFloat32, is now public. feat: add a public to float #1687 by @MarieSacksick.
• Improved the error message raised when a Polars lazyframe is passed to TableReport, clarifying that .collect() must be called first. FIX - Improving the error message raised when the TableReport is called on a lazy polars dataframe #1767 by @fatiben2002.
• Computing the associations in TableReport is now deterministic and can be controlled by the new parameter subsampling_seed of the global configuration. FEAT - Use sampling seed in summarize_dataframe for reproducibility #1775 by @thomass-dev.
• Added cast_to_str parameter to Cleaner to prevent unintended conversion of list/object-like columns to strings unless explicitly enabled. Add add_tostr flag to Cleaner to control ToStr transformation #1789 by @PilliSiddharth.

Bugfixes

• The skrub.cross_validate function now raises a specific exception if the wrong variable type is passed. Added type-specific error message to cross_validate function #1799 by @emassoulie.
• Fixed various issues with some transformers by adding get_feature_names_out to all single column transformers. FIX - Implementing the default get_feature_names_out method #1666 by @rcap107.
• Issues occurring when DataOp.skb.apply was passed a DataOp as the estimator have been fixed in sanitize behavior when estimator passed to apply() is a dataop #1671 by @jeromedockes.
• TableReport could raise an error while trying to check if Polars columns with some dtypes (lists, structs) are sorted. It would not indicate Polars columns sorted in descending order. Fixed in fix tablereport for columns for which order is not defined #1673 by @jeromedockes.
• Fixed nightly checks and added support for upcoming library versions, including Pandas v3.0. MAINT - Using pixi for nightlies and fixing failing nightly tests #1664 by @auguste-probabl and @rcap107.
• Fixed the use of TableReport and Cleaner with Polars dataframes containing a column with empty string as name. Fix: Cleaner (and TableVectorizer) fail when there is a polars empty column name #1722 by @MarieSacksick.
• Fixed an issue where TableReport would fail when computing associations for Polars dataframes if PyArrow was not installed. FIX - Adding condition and test for Polars dataframes without PyArrow installed in TableReport #1742 by @rcap107.
• Fixed an issue in the Data Ops report generation in cases where the DataOp contained escape characters or were spanning multiple lines. FIX - Proper escaping of special characters for data ops tooltips #1764 by @rcap107.
• Added get_feature_names_out to Cleaner for consistency with the TableVectorizer and other transformers. FIX - adding get_feature_names_out to Cleaner #1762 by @rcap107.
• Improve error message when TextEncoder is used without the optional transformers dependencies. FIX - Improve the error message when the TextEncoder is fitted without installing the additional dependencies #1769 by @fxzhou22.
• Accessing .skb.applied_estimator on a DataOp after calling .skb.set_name(), .skb.set_description(), .skb.mark_as_X() or .skb.mark_as_y() used to raise an error, this has been fixed in fix 1781: attribute missing from dataop returned by set_name() #1782 by @jeromedockes.
• Fixed potential issues that could arise in ParamSearch.plot_results when NaN values were present in the cross-validation results. BUG - Fixing a few edge cases in the generation of the parallel coordinate plot #1800 by @rcap107.

New Contributors

• @csejourne made their first contribution in add sizes of dataframes and total sizes on disk #1503
• @divakaivan made their first contribution in DOC - Add SquashingScaler to the homepage #1598
• @star1327p made their first contribution in DOC - Add link to pixi in CONTRIBUTING.rst #1599
• @abenechehab made their first contribution in Add Mean to boolean-type columns in Table Report #1647
• @dierickxsimon made their first contribution in Change StandardScaler to the SquashScaler for the tabular_pipline #1644
• @kudos07 made their first contribution in DOC: clarify behavior of 'cols' parameter in ApplyToCols and ApplyToFrame #1670
• @DimitriPapadopoulos made their first contribution in MAINT - PEP 639 compliance #1692
• @auguste-probabl made their first contribution in MAINT - Using pixi for nightlies and fixing failing nightly tests #1664
• @amirakahub made their first contribution in shuffle the df before generating the html doc for reporting #1715
• @Nismamjad1 made their first contribution in FEAT - add support for numpy arrays to the TableReport #1676
• @Alispirale made their first contribution in [Issue 1616] Fix examples numbering #1717
• @basile-desjuzeur made their first contribution in enhancement: clarify TableReport behavior and improve user guidance when plotting is skipped due to max_plot_columns #1685
• @fatiben2002 made their first contribution in FIX - Improving the error message raised when the TableReport is called on a lazy polars dataframe #1767
• @fxzhou22 made their first contribution in FIX - Improve the error message when the TextEncoder is fitted without installing the additional dependencies #1769
• @PilliSiddharth made their first contribution in Add add_tostr flag to Cleaner to control ToStr transformation #1789
• @emassoulie made their first contribution in Added type-specific error message to cross_validate function #1799

Full Changelog: 0.6.2...0.7.0

---

This discussion was created from the release Skrub release 0.7.0.