Test failures with NumPy 1.16.3+

[QuLogic]

Minimal, reproducible code sample, a copy-pastable example if possible

$ pytest

Problem description

Tests are failing with NumPy 1.16.3 due to a change in the default pickle loading settings:

_________________________ test_backwards_compatibility _________________________
    def test_backwards_compatibility():
        codec = Categorize(labels=labels, dtype='<U4', astype='u1')
        check_backwards_compatibility(Categorize.codec_id, arrays, [codec],
                                      prefix='U')
        codec = Categorize(labels=labels, dtype=object, astype='u1')
        check_backwards_compatibility(Categorize.codec_id, arrays_object,
>                                     [codec], prefix='O')
../../../BUILDROOT/python-numcodecs-0.6.3-2.fc31.x86_64/usr/lib64/python3.7/site-packages/numcodecs/tests/test_categorize.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../BUILDROOT/python-numcodecs-0.6.3-2.fc31.x86_64/usr/lib64/python3.7/site-packages/numcodecs/tests/common.py:190: in check_backwards_compatibility
    arr = np.load(arr_fn)
/usr/lib64/python3.7/site-packages/numpy/lib/npyio.py:447: in load
    pickle_kwargs=pickle_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
fp = <_io.BufferedReader name='fixture/categorize/O/array.03.npy'>
allow_pickle = False, pickle_kwargs = {'encoding': 'ASCII', 'fix_imports': True}
    def read_array(fp, allow_pickle=False, pickle_kwargs=None):
        """
        Read an array from an NPY file.
    
        Parameters
        ----------
        fp : file_like object
            If this is not a real file object, then this may take extra memory
            and time.
        allow_pickle : bool, optional
            Whether to allow writing pickled data. Default: False
    
            .. versionchanged:: 1.16.3
                Made default False in response to CVE-2019-6446.
    
        pickle_kwargs : dict
            Additional keyword arguments to pass to pickle.load. These are only
            useful when loading object arrays saved on Python 2 when using
            Python 3.
    
        Returns
        -------
        array : ndarray
            The array from the data on disk.
    
        Raises
        ------
        ValueError
            If the data is invalid, or allow_pickle=False and the file contains
            an object array.
    
        """
        version = read_magic(fp)
        _check_version(version)
        shape, fortran_order, dtype = _read_array_header(fp, version)
        if len(shape) == 0:
            count = 1
        else:
            count = numpy.multiply.reduce(shape, dtype=numpy.int64)
    
        # Now read the actual data.
        if dtype.hasobject:
            # The array contained Python objects. We need to unpickle the data.
            if not allow_pickle:
>               raise ValueError("Object arrays cannot be loaded when "
                                 "allow_pickle=False")
E               ValueError: Object arrays cannot be loaded when allow_pickle=False
/usr/lib64/python3.7/site-packages/numpy/lib/format.py:692: ValueError

The same traceback occurs for test_backwards_compatibility in:

• numcodecs/tests/test_json.py
• numcodecs/tests/test_msgpacks.py
• numcodecs/tests/test_pickles.py
• numcodecs/tests/test_vlen_array.py
• numcodecs/tests/test_vlen_bytes.py
• numcodecs/tests/test_vlen_utf8.py

Version and installation information

• Value of numcodecs.__version__: 0.6.3
• Version of Python interpreter: 3.7.3
• Operating system (Linux/Windows/Mac): Fedora Rawhide
• How NumCodecs was installed (e.g., "using pip into virtual environment", or "using conda"): Building a package; not yet installed.

[QuLogic]

Ping?

[jakirkham]

Would you like to send a PR?

[QuLogic]

Depends how you want to fix it.

[QuLogic]

I made a PR to just enable pickle loading, but that is not a long-term fix. The security concerns are still there though and a malicious actor could PR some innocuous thing with some conditional bad code hidden in the pickle.

[jakirkham]

Yeah we encourage users to try JSON or MsgPack as some alternatives. However Pickle has been around longer.

[jakirkham]

Thanks for the test fix. If you want to raise the pickle issue separately, would be happy to discuss further.