Test failures with PyPy3.10 7.3.12rc2

[mgorny]

These seem new, compared to PyPy3.9:

$ pypy3.10 --version
Python 3.10.11 (07561e2940ea7838339a8d615f9f314a0e83bbc0, May 31 2023, 07:28:45)
[PyPy 7.3.12-candidate2 with GCC 12.2.1 20230428]
$ pip install -r tests/requirements.txt && pip install -e .
[...]
$ python -m pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.10.11[pypy-7.3.12-candidate], pytest-7.2.1, pluggy-1.0.0
rootdir: /tmp/pydantic-core, configfile: pyproject.toml, testpaths: tests
plugins: pretty-1.1.1, speed-0.3.5, examples-0.0.4, mock-3.10.0, timeout-2.1.0, hypothesis-6.63.0
timeout: 30.0s
timeout method: signal
timeout func_only: False
collected 3656 items                                                                                                                  

tests/test_build.py .................                                                                                           [  0%]
tests/test_config.py .............                                                                                              [  0%]
tests/test_docstrings.py ..........................................................                                             [  2%]
tests/test_errors.py .......................................................................................................... [  5%]
.....................                                                                                                           [  5%]
tests/test_garbage_collection.py xx                                                                                             [  5%]
tests/test_hypothesis.py ..........                                                                                             [  6%]
tests/test_isinstance.py .....                                                                                                  [  6%]
tests/test_json.py .........................................                                                                    [  7%]
tests/test_misc.py ...........                                                                                                  [  7%]
tests/test_schema_functions.py .....................................................................                            [  9%]
tests/test_strict.py ....................................                                                                       [ 10%]
tests/test_typing.py ..........                                                                                                 [ 10%]
tests/test_validation_context.py ...........                                                                                    [ 11%]
tests/benchmarks/test_complete_benchmark.py ....s..s.s.                                                                         [ 11%]
tests/benchmarks/test_micro_benchmarks.py s.s...s.s...........sss.s.s.s.s...s..s..s..s.s..s..ss.s............s.s.s.s........... [ 13%]
............................s.....                                                                                              [ 14%]
tests/benchmarks/test_serialization_micro.py s.s.s...........ss......s.....                                                     [ 15%]
tests/serializers/test_any.py ...............................s..F                                                               [ 16%]
tests/serializers/test_any.py:129 test_any_json_decode_error[\x81] - SystemError: PY_SSIZE_T_CLEAN macro must be defined fo…
tests/serializers/test_any.py F                                                                                                 [ 16%]
tests/serializers/test_any.py:129 test_any_json_decode_error[value1] - SystemError: PY_SSIZE_T_CLEAN macro must be defined …
tests/serializers/test_any.py ..............................................                                                    [ 17%]
tests/serializers/test_bytes.py ..F                                                                                             [ 17%]
tests/serializers/test_bytes.py:31 test_bytes_invalid_cpython - SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#'…
tests/serializers/test_bytes.py ...........                                                                                     [ 18%]
tests/serializers/test_dataclasses.py .....                                                                                     [ 18%]
tests/serializers/test_datetime.py .................                                                                            [ 18%]
tests/serializers/test_definitions.py .......                                                                                   [ 19%]
tests/serializers/test_definitions_recursive.py .....                                                                           [ 19%]
tests/serializers/test_dict.py ...............................                                                                  [ 19%]
tests/serializers/test_format.py ...............                                                                                [ 20%]
tests/serializers/test_functions.py ..............................s.                                                            [ 21%]
tests/serializers/test_generator.py ......                                                                                      [ 21%]
tests/serializers/test_json.py ....                                                                                             [ 21%]
tests/serializers/test_json_or_python.py .                                                                                      [ 21%]
tests/serializers/test_list_tuple.py .......................................................................                    [ 23%]
tests/serializers/test_literal.py ....                                                                                          [ 23%]
tests/serializers/test_misc.py ..                                                                                               [ 23%]
tests/serializers/test_model.py ................................................                                                [ 25%]
tests/serializers/test_model_root.py .....                                                                                      [ 25%]
tests/serializers/test_none.py .............................                                                                    [ 25%]
tests/serializers/test_nullable.py .                                                                                            [ 25%]
tests/serializers/test_other.py ......                                                                                          [ 26%]
tests/serializers/test_set_frozenset.py .......                                                                                 [ 26%]
tests/serializers/test_simple.py ......................                                                                         [ 26%]
tests/serializers/test_string.py .........                                                                                      [ 27%]
tests/serializers/test_timedelta.py ...                                                                                         [ 27%]
tests/serializers/test_typed_dict.py ..........................                                                                 [ 27%]
tests/serializers/test_union.py ..................                                                                              [ 28%]
tests/serializers/test_url.py ......                                                                                            [ 28%]
tests/validators/test_arguments.py ..............s..s....s.........s.sssssss........ssssss.........sss......s..s.............s. [ 31%]
..sss.....s...s......ssss.....sss...sss.....sssss........                                                                       [ 32%]
tests/validators/test_bool.py .......................................                                                           [ 33%]
tests/validators/test_bytes.py ............................                                                                     [ 34%]
tests/validators/test_call.py ..........................                                                                        [ 35%]
tests/validators/test_callable.py ..............                                                                                [ 35%]
tests/validators/test_chain.py ............                                                                                     [ 35%]
tests/validators/test_custom_error.py ......                                                                                    [ 36%]
tests/validators/test_dataclasses.py .......s..sss........sssss......s......................................................... [ 38%]
..................................                                                                                              [ 39%]
tests/validators/test_date.py ..................................................................................                [ 41%]
tests/validators/test_datetime.py ............................................................................................. [ 44%]
...                                                                                                                             [ 44%]
tests/validators/test_definitions.py ..........                                                                                 [ 44%]
tests/validators/test_definitions_recursive.py .................................                                                [ 45%]
tests/validators/test_dict.py ..............................                                                                    [ 46%]
tests/validators/test_float.py ................................................................................................ [ 48%]
.............................................                                                                                   [ 50%]
tests/validators/test_frozenset.py .......................................................................                      [ 52%]
tests/validators/test_function.py ........................................................                                      [ 53%]
tests/validators/test_generator.py ......................................                                                       [ 54%]
tests/validators/test_int.py .................................................................................................. [ 57%]
...........................                                                                                                     [ 58%]
tests/validators/test_is_instance.py ................................                                                           [ 59%]
tests/validators/test_is_subclass.py .............                                                                              [ 59%]
tests/validators/test_json.py ...............................                                                                   [ 60%]
tests/validators/test_json_or_python.py .                                                                                       [ 60%]
tests/validators/test_lax_or_strict.py ..                                                                                       [ 60%]
tests/validators/test_list.py .............................................................................................     [ 62%]
tests/validators/test_literal.py ............................................                                                   [ 64%]
tests/validators/test_model.py ....................................                                                             [ 65%]
tests/validators/test_model_fields.py ......................................................................................... [ 67%]
.....................................................................................................................           [ 70%]
tests/validators/test_model_init.py ..........                                                                                  [ 70%]
tests/validators/test_model_root.py ......                                                                                      [ 71%]
tests/validators/test_none.py ..                                                                                                [ 71%]
tests/validators/test_nullable.py ..                                                                                            [ 71%]
tests/validators/test_set.py .........................................................................                          [ 73%]
tests/validators/test_string.py ....................................................................                            [ 75%]
tests/validators/test_tagged_union.py .................................................................                         [ 76%]
tests/validators/test_time.py ..................................................................................                [ 79%]
tests/validators/test_timedelta.py ........................................................................                     [ 81%]
tests/validators/test_tuple.py ......................................................................................           [ 83%]
tests/validators/test_typed_dict.py ........................................................................................... [ 85%]
..............................................................                                                                  [ 87%]
tests/validators/test_union.py ..........................................                                                       [ 88%]
tests/validators/test_url.py .................................................................................................. [ 91%]
............................................................................................................................... [ 94%]
..................................................................................................                              [ 97%]
tests/validators/test_with_default.py ........................................................................................  [100%]

============================================================== FAILURES ===============================================================
__________________________________________________ test_any_json_decode_error[\x81] ___________________________________________________

any_serializer = SchemaSerializer(serializer=Any(
    AnySerializer,
), definitions=[]), value = b'\x81'

    @pytest.mark.parametrize('value', [b'\x81', bytearray(b'\x81')])
    def test_any_json_decode_error(any_serializer, value):
        assert any_serializer.to_python(value) == value
    
        msg = 'Error serializing to JSON: invalid utf-8 sequence of 1 bytes from index 0'
        with pytest.raises(PydanticSerializationError, match=msg):
            any_serializer.to_json(value)
    
        with pytest.raises(ValueError):
>           any_serializer.to_python(value, mode='json')
E           SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

tests/serializers/test_any.py:139: SystemError
_________________________________________________ test_any_json_decode_error[value1] __________________________________________________

any_serializer = SchemaSerializer(serializer=Any(
    AnySerializer,
), definitions=[]), value = bytearray(b'\x81')

    @pytest.mark.parametrize('value', [b'\x81', bytearray(b'\x81')])
    def test_any_json_decode_error(any_serializer, value):
        assert any_serializer.to_python(value) == value
    
        msg = 'Error serializing to JSON: invalid utf-8 sequence of 1 bytes from index 0'
        with pytest.raises(PydanticSerializationError, match=msg):
            any_serializer.to_json(value)
    
        with pytest.raises(ValueError):
>           any_serializer.to_python(value, mode='json')
E           SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

tests/serializers/test_any.py:139: SystemError
_____________________________________________________ test_bytes_invalid_cpython ______________________________________________________

    def test_bytes_invalid_cpython():
        # PyO3/pyo3#2770 is now fixed
        s = SchemaSerializer(core_schema.bytes_schema())
    
        with pytest.raises(UnicodeDecodeError, match="'utf-8' codec can't decode byte 0x81 in position 0: invalid utf-8"):
>           s.to_python(b'\x81', mode='json')
E           SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

tests/serializers/test_bytes.py:37: SystemError
                                                     Summary of Failures                                                     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃  File                             ┃  Function                            ┃  Function Line  ┃  Error Line  ┃  Error        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│  tests/serializers/test_any.py    │  test_any_json_decode_error[\x81]    │  130            │  139         │  SystemError  │
│  tests/serializers/test_any.py    │  test_any_json_decode_error[value1]  │  130            │  139         │  SystemError  │
│  tests/serializers/test_bytes.py  │  test_bytes_invalid_cpython          │  32             │  37          │  SystemError  │
└───────────────────────────────────┴──────────────────────────────────────┴─────────────────┴──────────────┴───────────────┘
Results (19.41s):
         3 failed
      3562 passed
        89 skipped
         2 xfailed

Unfortunately, Rust extensions are outside my expertise, so I can't tell if it's something pydantic-core is doing, a bug in PyO3 or a bug in PyPy3.10. If you could help me determine that, I can pass it on to the correct project.

[mgorny]

On a semi-related note, if I build the extension in dev profile for debugging, it crashes on all PyPy versions, though on something else: https://foss.heptapod.net/pypy/pypy/-/issues/3948

[samuelcolvin]

@davidhewitt any idea if this is a pyo3 issue?

[davidhewitt]

Very likely. We haven't yet attempted to add pypy 3.10 dev builds to our CI, I'll try to check tomorrow if it's possible in GitHub actions, and if so, we can get a 0.19.1 out.

[mgorny]

They are listed on https://downloads.python.org/pypy/versions.json, so it probably should be possible. You may need to somehow indicate that you want -dev versions, I guess.

[mgorny]

Gentle ping. Does anyone happen to know if there's any progress here?

[davidhewitt]

Since #731 we now have the upstream support from PyO3 for PyPy 3.10 added in here.

Also see PyO3/pyo3#2781 (reply in thread) - @adamreichold identified some possible issues related to stack overflows. I'm not sure anything was actioned here about that.

[adamreichold]

I'm not sure anything was actioned here about that.

Was that diff included here? If not, there are still two independent recursion limits used and both are too large for debug builds on PyPy.

[mgorny]

I was asking about these errors here:

SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

I can still reproduce them with d08d269 (which is using PyO3 0.19.1. Debug builds are not a critical feature for us.

[davidhewitt]

Right, thanks for reraising that one. I've debugged this morning and I believe PyO3/pyo3#3297 will resolve.

[mgorny]

Thanks a lot!