Further test_zip failures with Python 3.13: `_compresslevel` vs. `compress_level`

[mgorny]

$ pytest fsspec/implementations/tests/test_zip.py -vv
========================================================= test session starts =========================================================
platform linux -- Python 3.13.0rc2, pytest-8.3.3, pluggy-1.5.0 -- /tmp/filesystem_spec/.venv/bin/python3.13
cachedir: .pytest_cache
rootdir: /tmp/filesystem_spec
configfile: pyproject.toml
collected 17 items                                                                                                                    

fsspec/implementations/tests/test_zip.py::test_info PASSED                                                                      [  5%]
fsspec/implementations/tests/test_zip.py::test_fsspec_get_mapper PASSED                                                         [ 11%]
fsspec/implementations/tests/test_zip.py::test_not_cached PASSED                                                                [ 17%]
fsspec/implementations/tests/test_zip.py::test_root_info PASSED                                                                 [ 23%]
fsspec/implementations/tests/test_zip.py::test_write_seek PASSED                                                                [ 29%]
fsspec/implementations/tests/test_zip.py::test_rw PASSED                                                                        [ 35%]
fsspec/implementations/tests/test_zip.py::test_mapper PASSED                                                                    [ 41%]
fsspec/implementations/tests/test_zip.py::test_zip_glob_star PASSED                                                             [ 47%]
fsspec/implementations/tests/test_zip.py::test_append PASSED                                                                    [ 52%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_true FAILED                                  [ 58%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_false PASSED                                 [ 64%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_true_include_dirs FAILED                     [ 70%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_false_include_dirs PASSED                    [ 76%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_path_set PASSED                                     [ 82%]
fsspec/implementations/tests/test_zip.py::test_find_with_and_without_slash_should_return_same_result PASSED                     [ 88%]
fsspec/implementations/tests/test_zip.py::test_find_should_return_file_if_exact_match PASSED                                    [ 94%]
fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_recursion_depth_set PASSED                          [100%]

============================================================== FAILURES ===============================================================
____________________________________________ test_find_returns_expected_result_detail_true ____________________________________________

zip_file = PosixPath('/tmp/pytest-of-mgorny/pytest-3/test_find_returns_expected_res0/test.zip')

    def test_find_returns_expected_result_detail_true(zip_file):
        zip_file_system = ZipFileSystem(zip_file)
    
        result = zip_file_system.find("/", detail=True)
    
        expected_result = {
            "dir2/file3.txt": {
                "orig_filename": "dir2/file3.txt",
                "filename": "dir2/file3.txt",
                "date_time": (2024, 8, 16, 10, 46, 18),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 260,
                "CRC": 2636827734,
                "compress_size": 8,
                "file_size": 6,
                "_raw_time": 21961,
                "_end_offset": 312,
                "name": "dir2/file3.txt",
                "size": 6,
                "type": "file",
            },
            "file1.txt": {
                "orig_filename": "file1.txt",
                "filename": "file1.txt",
                "date_time": (2024, 8, 16, 10, 46, 18),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 139,
                "CRC": 3964322768,
                "compress_size": 15,
                "file_size": 13,
                "_raw_time": 21961,
                "_end_offset": 193,
                "name": "file1.txt",
                "size": 13,
                "type": "file",
            },
            "file2.txt": {
                "orig_filename": "file2.txt",
                "filename": "file2.txt",
                "date_time": (2024, 8, 16, 10, 46, 18),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 193,
                "CRC": 1596576865,
                "compress_size": 28,
                "file_size": 26,
                "_raw_time": 21961,
                "_end_offset": 260,
                "name": "file2.txt",
                "size": 26,
                "type": "file",
            },
        }
    
>       _assert_all_except_context_dependent_variables(result, expected_result)

fsspec/implementations/tests/test_zip.py:271: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

result = {'dir2/file3.txt': {'CRC': 2636827734, '_end_offset': 312, '_raw_time': 43777, 'comment': b'', ...}, 'dir2startwithsam...7, 'comment': b'', ...}, 'file2.txt': {'CRC': 1596576865, '_end_offset': 191, '_raw_time': 43777, 'comment': b'', ...}}
expected_result = {'dir2/file3.txt': {'CRC': 2636827734, '_compresslevel': None, '_end_offset': 312, '_raw_time': 21961, ...}, 'file1.tx...e': 21961, ...}, 'file2.txt': {'CRC': 1596576865, '_compresslevel': None, '_end_offset': 260, '_raw_time': 21961, ...}}

    def _assert_all_except_context_dependent_variables(result, expected_result):
        for path in expected_result.keys():
            assert result[path]
            result_without_date_time = result[path].copy()
            result_without_date_time.pop("date_time")
            result_without_date_time.pop("_raw_time")
            result_without_date_time.pop("external_attr")
            result_without_date_time.pop("create_system")
            result_without_date_time.pop("_end_offset", None)
            result_without_date_time.pop("header_offset", None)
    
            expected_result_without_date_time = expected_result[path].copy()
            expected_result_without_date_time.pop("date_time")
            expected_result_without_date_time.pop("_raw_time")
            expected_result_without_date_time.pop("external_attr")
            expected_result_without_date_time.pop("create_system")
            expected_result_without_date_time.pop("_end_offset", None)
            expected_result_without_date_time.pop("header_offset", None)
>           assert result_without_date_time == expected_result_without_date_time
E           AssertionError: assert {'orig_filename': 'dir2/file3.txt', 'filename': 'dir2/file3.txt', 'compress_type': 8, 'compress_level': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 2636827734, 'compress_size': 8, 'file_size': 6, 'name': 'dir2/file3.txt', 'size': 6, 'type': 'file'} == {'orig_filename': 'dir2/file3.txt', 'filename': 'dir2/file3.txt', 'compress_type': 8, '_compresslevel': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 2636827734, 'compress_size': 8, 'file_size': 6, 'name': 'dir2/file3.txt', 'size': 6, 'type': 'file'}
E             
E             Common items:
E             {'CRC': 2636827734,
E              'comment': b'',
E              'compress_size': 8,
E              'compress_type': 8,
E              'create_version': 20,
E              'extra': b'',
E              'extract_version': 20,
E              'file_size': 6,
E              'filename': 'dir2/file3.txt',
E              'flag_bits': 0,
E              'internal_attr': 0,
E              'name': 'dir2/file3.txt',
E              'orig_filename': 'dir2/file3.txt',
E              'reserved': 0,
E              'size': 6,
E              'type': 'file',
E              'volume': 0}
E             Left contains 1 more item:
E             {'compress_level': None}
E             Right contains 1 more item:
E             {'_compresslevel': None}
E             
E             Full diff:
E               {
E                   'CRC': 2636827734,
E             -     '_compresslevel': None,
E                   'comment': b'',
E             +     'compress_level': None,
E                   'compress_size': 8,
E                   'compress_type': 8,
E                   'create_version': 20,
E                   'extra': b'',
E                   'extract_version': 20,
E                   'file_size': 6,
E                   'filename': 'dir2/file3.txt',
E                   'flag_bits': 0,
E                   'internal_attr': 0,
E                   'name': 'dir2/file3.txt',
E                   'orig_filename': 'dir2/file3.txt',
E                   'reserved': 0,
E                   'size': 6,
E                   'type': 'file',
E                   'volume': 0,
E               }

fsspec/implementations/tests/test_zip.py:182: AssertionError
_____________________________________ test_find_returns_expected_result_detail_true_include_dirs ______________________________________

zip_file = PosixPath('/tmp/pytest-of-mgorny/pytest-3/test_find_returns_expected_res2/test.zip')

    def test_find_returns_expected_result_detail_true_include_dirs(zip_file):
        zip_file_system = ZipFileSystem(zip_file)
    
        result = zip_file_system.find("/", detail=True, withdirs=True)
        expected_result = {
            "dir1": {
                "orig_filename": "dir1/",
                "filename": "dir1/",
                "date_time": (2024, 8, 16, 10, 54, 24),
                "compress_type": 0,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 1106051088,
                "header_offset": 0,
                "CRC": 0,
                "compress_size": 0,
                "file_size": 0,
                "_raw_time": 22220,
                "_end_offset": 35,
                "name": "dir1",
                "size": 0,
                "type": "directory",
            },
            "dir2": {
                "orig_filename": "dir2/",
                "filename": "dir2/",
                "date_time": (2024, 8, 16, 10, 54, 24),
                "compress_type": 0,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 1106051088,
                "header_offset": 35,
                "CRC": 0,
                "compress_size": 0,
                "file_size": 0,
                "_raw_time": 22220,
                "_end_offset": 70,
                "name": "dir2",
                "size": 0,
                "type": "directory",
            },
            "dir2/file3.txt": {
                "orig_filename": "dir2/file3.txt",
                "filename": "dir2/file3.txt",
                "date_time": (2024, 8, 16, 10, 54, 24),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 260,
                "CRC": 2636827734,
                "compress_size": 8,
                "file_size": 6,
                "_raw_time": 22220,
                "_end_offset": 312,
                "name": "dir2/file3.txt",
                "size": 6,
                "type": "file",
            },
            "file1.txt": {
                "orig_filename": "file1.txt",
                "filename": "file1.txt",
                "date_time": (2024, 8, 16, 10, 54, 24),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 139,
                "CRC": 3964322768,
                "compress_size": 15,
                "file_size": 13,
                "_raw_time": 22220,
                "_end_offset": 193,
                "name": "file1.txt",
                "size": 13,
                "type": "file",
            },
            "file2.txt": {
                "orig_filename": "file2.txt",
                "filename": "file2.txt",
                "date_time": (2024, 8, 16, 10, 54, 24),
                "compress_type": 8,
                "_compresslevel": None,
                "comment": b"",
                "extra": b"",
                "create_system": 3,
                "create_version": 20,
                "extract_version": 20,
                "reserved": 0,
                "flag_bits": 0,
                "volume": 0,
                "internal_attr": 0,
                "external_attr": 2175008768,
                "header_offset": 193,
                "CRC": 1596576865,
                "compress_size": 28,
                "file_size": 26,
                "_raw_time": 22220,
                "_end_offset": 260,
                "name": "file2.txt",
                "size": 26,
                "type": "file",
            },
        }
    
>       _assert_all_except_context_dependent_variables(result, expected_result)

fsspec/implementations/tests/test_zip.py:425: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

result = {'dir1': {'CRC': 0, '_end_offset': 35, '_raw_time': 43777, 'comment': b'', ...}, 'dir2': {'CRC': 0, '_end_offset': 70,....}, 'dir2startwithsamename.txt': {'CRC': 1089626562, '_end_offset': 260, '_raw_time': 43777, 'comment': b'', ...}, ...}
expected_result = {'dir1': {'CRC': 0, '_compresslevel': None, '_end_offset': 35, '_raw_time': 22220, ...}, 'dir2': {'CRC': 0, '_compress...2220, ...}, 'file1.txt': {'CRC': 3964322768, '_compresslevel': None, '_end_offset': 193, '_raw_time': 22220, ...}, ...}

    def _assert_all_except_context_dependent_variables(result, expected_result):
        for path in expected_result.keys():
            assert result[path]
            result_without_date_time = result[path].copy()
            result_without_date_time.pop("date_time")
            result_without_date_time.pop("_raw_time")
            result_without_date_time.pop("external_attr")
            result_without_date_time.pop("create_system")
            result_without_date_time.pop("_end_offset", None)
            result_without_date_time.pop("header_offset", None)
    
            expected_result_without_date_time = expected_result[path].copy()
            expected_result_without_date_time.pop("date_time")
            expected_result_without_date_time.pop("_raw_time")
            expected_result_without_date_time.pop("external_attr")
            expected_result_without_date_time.pop("create_system")
            expected_result_without_date_time.pop("_end_offset", None)
            expected_result_without_date_time.pop("header_offset", None)
>           assert result_without_date_time == expected_result_without_date_time
E           AssertionError: assert {'orig_filename': 'dir1/', 'filename': 'dir1/', 'compress_type': 0, 'compress_level': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 0, 'compress_size': 0, 'file_size': 0, 'name': 'dir1', 'size': 0, 'type': 'directory'} == {'orig_filename': 'dir1/', 'filename': 'dir1/', 'compress_type': 0, '_compresslevel': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 0, 'compress_size': 0, 'file_size': 0, 'name': 'dir1', 'size': 0, 'type': 'directory'}
E             
E             Common items:
E             {'CRC': 0,
E              'comment': b'',
E              'compress_size': 0,
E              'compress_type': 0,
E              'create_version': 20,
E              'extra': b'',
E              'extract_version': 20,
E              'file_size': 0,
E              'filename': 'dir1/',
E              'flag_bits': 0,
E              'internal_attr': 0,
E              'name': 'dir1',
E              'orig_filename': 'dir1/',
E              'reserved': 0,
E              'size': 0,
E              'type': 'directory',
E              'volume': 0}
E             Left contains 1 more item:
E             {'compress_level': None}
E             Right contains 1 more item:
E             {'_compresslevel': None}
E             
E             Full diff:
E               {
E                   'CRC': 0,
E             -     '_compresslevel': None,
E                   'comment': b'',
E             +     'compress_level': None,
E                   'compress_size': 0,
E                   'compress_type': 0,
E                   'create_version': 20,
E                   'extra': b'',
E                   'extract_version': 20,
E                   'file_size': 0,
E                   'filename': 'dir1/',
E                   'flag_bits': 0,
E                   'internal_attr': 0,
E                   'name': 'dir1',
E                   'orig_filename': 'dir1/',
E                   'reserved': 0,
E                   'size': 0,
E                   'type': 'directory',
E                   'volume': 0,
E               }

fsspec/implementations/tests/test_zip.py:182: AssertionError
======================================================= short test summary info =======================================================
FAILED fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_true - AssertionError: assert {'orig_filename': 'dir2/file3.txt', 'filename': 'dir2/file3.txt', 'compress_type': 8, 'compress_level': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 2636827734, 'compress_size': 8, 'file_size': 6, 'name': 'dir2/file3.txt', 'size': 6, 'type': 'file'} == {'orig_filename': 'dir2/file3.txt', 'filename': 'dir2/file3.txt', 'compress_type': 8, '_compresslevel': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 2636827734, 'compress_size': 8, 'file_size': 6, 'name': 'dir2/file3.txt', 'size': 6, 'type': 'file'}
  
  Common items:
  {'CRC': 2636827734,
   'comment': b'',
   'compress_size': 8,
   'compress_type': 8,
   'create_version': 20,
   'extra': b'',
   'extract_version': 20,
   'file_size': 6,
   'filename': 'dir2/file3.txt',
   'flag_bits': 0,
   'internal_attr': 0,
   'name': 'dir2/file3.txt',
   'orig_filename': 'dir2/file3.txt',
   'reserved': 0,
   'size': 6,
   'type': 'file',
   'volume': 0}
  Left contains 1 more item:
  {'compress_level': None}
  Right contains 1 more item:
  {'_compresslevel': None}
  
  Full diff:
    {
        'CRC': 2636827734,
  -     '_compresslevel': None,
        'comment': b'',
  +     'compress_level': None,
        'compress_size': 8,
        'compress_type': 8,
        'create_version': 20,
        'extra': b'',
        'extract_version': 20,
        'file_size': 6,
        'filename': 'dir2/file3.txt',
        'flag_bits': 0,
        'internal_attr': 0,
        'name': 'dir2/file3.txt',
        'orig_filename': 'dir2/file3.txt',
        'reserved': 0,
        'size': 6,
        'type': 'file',
        'volume': 0,
    }
FAILED fsspec/implementations/tests/test_zip.py::test_find_returns_expected_result_detail_true_include_dirs - AssertionError: assert {'orig_filename': 'dir1/', 'filename': 'dir1/', 'compress_type': 0, 'compress_level': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 0, 'compress_size': 0, 'file_size': 0, 'name': 'dir1', 'size': 0, 'type': 'directory'} == {'orig_filename': 'dir1/', 'filename': 'dir1/', 'compress_type': 0, '_compresslevel': None, 'comment': b'', 'extra': b'', 'create_version': 20, 'extract_version': 20, 'reserved': 0, 'flag_bits': 0, 'volume': 0, 'internal_attr': 0, 'CRC': 0, 'compress_size': 0, 'file_size': 0, 'name': 'dir1', 'size': 0, 'type': 'directory'}
  
  Common items:
  {'CRC': 0,
   'comment': b'',
   'compress_size': 0,
   'compress_type': 0,
   'create_version': 20,
   'extra': b'',
   'extract_version': 20,
   'file_size': 0,
   'filename': 'dir1/',
   'flag_bits': 0,
   'internal_attr': 0,
   'name': 'dir1',
   'orig_filename': 'dir1/',
   'reserved': 0,
   'size': 0,
   'type': 'directory',
   'volume': 0}
  Left contains 1 more item:
  {'compress_level': None}
  Right contains 1 more item:
  {'_compresslevel': None}
  
  Full diff:
    {
        'CRC': 0,
  -     '_compresslevel': None,
        'comment': b'',
  +     'compress_level': None,
        'compress_size': 0,
        'compress_type': 0,
        'create_version': 20,
        'extra': b'',
        'extract_version': 20,
        'file_size': 0,
        'filename': 'dir1/',
        'flag_bits': 0,
        'internal_attr': 0,
        'name': 'dir1',
        'orig_filename': 'dir1/',
        'reserved': 0,
        'size': 0,
        'type': 'directory',
        'volume': 0,
    }
==================================================== 2 failed, 15 passed in 0.08s =====================================================

This is with b842cf4.

[martindurant]

OK, thanks for reporting. I suppose we should be testing only against the few keys we know should be there rather than excluding problematic ones one-by-one as we discover changes.

[martindurant]

Is 3.13 already available as a package?

[mgorny]

I'm sorry but I don't sure I understand your question.

3.13.0 final has been deferred one week, so the newest version right now is rc3. It's available from "pre-releases" in https://www.python.org/downloads/source/. Some Linux distributions also ship it (Debian 13, Fedora 39+, Gentoo, nix, openSUSE, Ubuntu), though may need some distro-specific switches to access it. It's also available in AUR, MacPorts, deadsnakes, GitHub CI…

[martindurant]

This was the answer I was after:

Some Linux distributions also ship it (Debian 13, Fedora 39+, Gentoo, nix, openSUSE, Ubuntu), though may need some distro-specific switches to access it.

to give me a sense of urgency in this issue. The CI here runs on conda, which doesn't support 3.13 yet.

[mgorny]

https://conda-forge.org/blog/2024/09/26/python-313/ suggests it does.

[martindurant]

Indeed, my conda people just said the same - you need the dev label. I am trying to find out when that goes away. I can make the tests you note pass either way, though.