unmaintained dependency matrixprofile makes tsfresh uninstallable on python 3.10

[mmp3]

The problem:

tsfresh cannot be installed on python 3.10 because it has matrixprofile as a dependency, yet matrixprofile cannot be installed on python 3.10 and is no longer maintained.

Anything else we need to know?:

matrixprofile is superseded by stumpy.

Environment:

• Python version: 3.10

[arturdaraujo]

This package is stuck on Python 3.8 until solving this bug, we are all waiting for this fix. This issue was posted before on #972 and #984

[arturdaraujo]

Let's help @kempa-liehr fix this.

SOLUTIONS:

1. Make matrixprofile package's installation optional using extras_require in the setup.py
2. Implement stumpy instead of matrixprofile.

Anyone familiar on how to use extras_require in the setup.py to make matrixprofile optional? Anyone able to post a code that would replace matrixprofile functions on tsfresh? I will have a go and post it here

@erezinman @aiwalter @PJPRoche @williamgilpin @enrique-wesper

[arturdaraujo]

Making matrixprofile package optional:

• Changes on: setup.py

For making matrixprofile optional, the setup.py solution is like this:

import sys

from setuptools import find_packages, setup

with open("README.md") as f:
    long_description = f.read()

with open("requirements.txt") as f:
    requirements = [line for line in f if not line.startswith("#")]

needs_sphinx = {"build_sphinx", "upload_docs"}.intersection(sys.argv)
sphinx = ["sphinx", "sphinx_rtd_theme"] if needs_sphinx else []

simple_requirements = [r for r in requirements if r != "matrixprofile>=1.1.10,<2.0.0"]
advanced_requirements = [r for r in requirements if r == "matrixprofile>=1.1.10,<2.0.0"]

setup(
    use_scm_version=True,
    long_description=long_description,
    long_description_content_type="text/markdown",
    setup_requires=["setuptools_scm"] + sphinx,
    packages=find_packages(exclude=["tests.*", "tests"]),
    extras_require={
        "advanced": advanced_requirements,  # Advanced features
        "simple": simple_requirements,  # Basic features
    }
)

• Changes on: feature_extraction/feature_calculators.py

Note: "matrix_profile" is a tsfresh function, while "matrixprofile" is the outdated package in question.
The function "matrix_profile" from feature_calculators.py script will still try to import it. The solution is to check if matrixprofile package is installed before importing it. I suggest making the following changes.

import importlib.util

package = "matrixprofile"
is_present = importlib.util.find_spec(package)

if is_present is not None:
    import matrixprofile as mp

And

if is_present is not None: 
    @set_property("fctype", "combiner")
    def matrix_profile(x, param):
    """
     Calculates the 1-D Matrix Profile[1] and returns... 
    """

     [...]
    
        return [(key, value) for key, value in res.items()]

• Changes on: across tsfresh package

After the implementation on feature_calculators.py, the only thing needed would be to place a if is_present is not None: on where matrix_profile is called across tsfresh package.
Calling importlib.util.find_spec(package) is fast but you could make it faster by simply using from ...feature_extraction.feature_calculators import is_present, this will help scale on big data. and will call importlib.util.find_spec(package) only once.

I hope this answer is clear enough to give a sense of how to fix this. If it is too convoluted or it could be improved please point it out.
@kempa-liehr @erezinman @aiwalter @PJPRoche @williamgilpin @enrique-wesper @mmp3 @bvanelli

References:

• https://stackoverflow.com/questions/62713525/declare-optional-dependencies-within-extras-require
• https://www.codespeedy.com/how-to-check-if-a-python-package-is-installed/#:~:text=Using%20the%20import%20keyword%20in,to%20do%20the%20same%20thing.
• https://stackoverflow.com/questions/19289171/importing-a-variable-from-one-python-script-to-another

[erezinman]

@arturdaraujo Just separate the matrixprofile requirement from the rest. I'll try creating a PR if I'll get to it later. Otherwise, good job!

Also, you could simply import the package normally, but wrap this with try: ... except ImportError: ... without using importlib. Again, I'll try to do it later on.

[arturdaraujo]

I also think try: ... except ImportError: ... is a much better solution now that you mentioned it, no need for using importlib.

Thanks for the reply

[arturdaraujo]

I just realized that there is a PR #985
Can you guys also check if that's adequate? It seems pretty good to me

@kempa-liehr @erezinman @aiwalter @PJPRoche @williamgilpin @enrique-wesper @mmp3 @bvanelli @michaelosthege

[nils-braun]

PR has been merged and a new version 0.20.0 released :)
(if you still see issues, please let us know by!)
Thank you for your time and effort!

[aiwalter]

@nils-braun great! should we also update it here?

tsfresh/setup.cfg

Lines 10 to 30 in d059eec

	classifier =
	Development Status :: 4 - Beta
	Programming Language :: Python
	Programming Language :: Python :: 3
	Programming Language :: Python :: 3.7
	Programming Language :: Python :: 3.8
	Programming Language :: Python :: 3.9
	Programming Language :: Python :: 3 :: Only
	Operating System :: Unix
	Operating System :: Microsoft :: Windows
	Operating System :: MacOS
	Intended Audience :: Science/Research
	Topic :: Scientific/Engineering
	Topic :: Software Development
	[options]
	python_requires = >= 3.7
	[entry_points]
	# Add here console scripts like:
	# console_scripts =

[nils-braun]

Thanks @aiwalter ! I have added py 3.10 as a classifier. I have not added an upper bound to the python_requires as the ´tsfresh´ package itself has no limitation on it (some dependencies might have though).
Also, I have not added a 3.11 classifier as currently the dependencies will not allow tsfresh to be installed on py 3.11.

[ElreboCM]

@nils-braun I still cannot use tsfresh with python 3.10.9. I am getting the following error:
ModuleNotFoundError: TSFreshFeatureExtractor requires python version to be <3.10, but system python version is 3.10.9 (v3.10.9:1dd9be6584, Dec 6 2022, 14:37:36) [Clang 13.0.0 (clang-1300.0.29.30)].

I am using the main branch with latest commit 2e49614

[nils-braun]

Hi @ElreboCM - it looks like you are not using tsfresh directly, but maybe via the aeon toolkit by @aiwalter? Because TSFreshFeatureExtractor sounds like it. So maybe there is another layer which prevents you from using it? The version constraint in tsfresh is removed though.

[ElreboCM]

Indeed I was importing from sktime

from sktime.transformations.panel.tsfresh import TSFreshFeatureExtractor