Fix dependency: Joblib 1.4.0 breaks PyCaret, so vendorize compatibility routines

[amotl]

Problem

PyCaret uses the private function joblib.memory._format_load_msg from Joblib, which is no longer present in Joblib 1.4.

• [BUG]: Can't use RegressionExperiment with the latest version of joblib (1.4.0) #3959

Solution

In order to fix the problem without much efforts, this patch vendorizes the corresponding function from Joblib without much ado.

Evaluation

It is advisable to release this update on behalf of a bugfix release, so that downstream users will get this issue resolved quickly.

Install

In order to use the patch instantly before upstream maintainers ran a new bugfix release, if it's important to you, you may want to use this pip package specification to install the package in an ad hoc manner, including the fixes for both dependency flaws.

pip install --upgrade 'pycaret[parallel] @ git+https://github.com/crate-workbench/pycaret@fix-joblib-vendorize'

It also works within package metadata specifications like setup.py, setup.cfg, pyproject.toml, or requirements.txt files, with recent versions of pip, when the git command is available on the installation environment. This satisfies most workstation installations, the Google Colab runtime environment, and probably a few others where git is provided.

Trivia

Please note this patch is not sufficient to make PyCaret work on Python 3.11.9, which also has been released recently ¹, and introduces another incompatibility. You will additionally need that patch to make it work:

• Fix dependency: Python 3.11.9 breaks Dask #3963

However, this does not impact users who did not update their Python 3.11 version yet, and others who might still be on Python 3.10 or earlier.

Footnotes

1. That's the reason why CI still fails, see [META]: Joblib 1.4.0 breaks PyCaret. Python 3.11.9 breaks Dask. #3966, which summarizes both issues. ↩

[celestinoxp]

@amotl tests are failing... you must fix the remaining errors...

[amotl]

@amotl tests are failing... you must fix the remaining errors...

Thanks for noticing. We did not see that error before, apologies. We will look into it.

AttributeError: 'FastMemorizedFunc' object has no attribute '_get_output_identifiers'

-- https://github.com/pycaret/pycaret/actions/runs/8676442808/job/23790959421?pr=3965#step:6:1611

NB: Will you consider merging GH-3963, in order to unblock CI running Python 3.11.9 already?

[celestinoxp]

ping @Yard1

[amotl]

It looks like the error around FastMemorizedFunc might be yet another thing, only happening on Python 3.9 and 3.10. The error on Python 3.11 will be remedied by GH-3963. We will look into where the new flaw may be coming from.

• Python 3.9: AttributeError: 'FastMemorizedFunc' object has no attribute '_get_output_identifiers'
  https://github.com/pycaret/pycaret/actions/runs/8676442808/job/23790959421?pr=3965#step:6:2968

• Python 3.10: AttributeError: 'FastMemorizedFunc' object has no attribute '_get_output_identifiers'
  https://github.com/pycaret/pycaret/actions/runs/8676442808/job/23790959545?pr=3965#step:6:2968

• Python 3.11: TypeError: descriptor '__call__' for 'type' objects doesn't apply to a 'property' object
  https://github.com/pycaret/pycaret/actions/runs/8676442808/job/23790959722?pr=3965#step:6:78

[amotl]

Hi again,

it looks like the errors with Python 3.9 and 3.10 are also related to Joblib 1.4.0, so we are abstaining from vendorizing, and favor downgrading as a quick measure.

In this spirit, we diverted this into another patch...

• Fix dependency: Joblib 1.4.0 breaks PyCaret, so downgrade to <1.4 #3967

... and will close this one again.

With kind regards,
Andreas.