Test suite execution broken in 5.7.2 with Python 3.7 due to joblib update

[dhgutteridge]

The updated version of joblib packaged in 5.7.2 has broken test suite execution with Python 3.7. In the same environment, 5.7.0 and 5.7.1 manage the test executions successfully, where 5.7.2 immediately fails with the following trace.

cloudpickle_fast.py is a new implementation picked up in the joblib update. types.CellType is new with Python 3.8.

running build_ext
Traceback (most recent call last):
  File "setup.py", line 71, in <module>
    include_package_data=True,
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/pkg/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/pkg/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/pkg/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 237, in run
    self.run_tests()
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 259, in run_tests
    exit=False,
  File "/usr/pkg/lib/python3.7/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/pkg/lib/python3.7/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/pkg/lib/python3.7/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/usr/pkg/lib/python3.7/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 406, in _find_tests
    full_path, pattern, namespace)
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/pkg/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/pkg/lib/python3.7/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/disciple/pkgsrc/wip_dhg/py-music21.bak/work/music21-5.7.2/music21/ext/joblib/externals/cloudpickle/cloudpickle_fast.py", line 388, in <module>
    class CloudPickler(Pickler):
  File "/home/disciple/pkgsrc/wip_dhg/py-music21.bak/work/music21-5.7.2/music21/ext/joblib/externals/cloudpickle/cloudpickle_fast.py", line 410, in CloudPickler
    dispatch[types.CellType] = _cell_reduce
AttributeError: module 'types' has no attribute 'CellType'
*** Error code 1

[mscuthbert]

THanks -- will jump on this

[dhgutteridge]

It wasn't immediately obvious to me why that was happening, as there is a Python version check in place that's intended to only use cloudpickle_fast.py with Python 3.8. But it seems that's still getting pulled in separately with Python <3.8 somewhere, which triggers that exception.

(Anyway, not the most burning issue, I'm sure. I run "test" for regression testing downstream package updates.)

[dhgutteridge]

This is fixed, from my perspective. 6.1.0's test suite execution works for me with Python 3.7 again.

[mscuthbert]

Thanks! Hopefully bundling with particular requirements.txt will reduce the frequency that this happens.