Pypy support

[samuelcolvin]

@tiangolo asked about pypy support. @messense do you have input?

Looks like pyo3 does support pypy, see here and PyO3/rust-numpy#219.

But I know pydantic-core uses some non abi3 methods, maybe we use stuff that would cause problems with pypy, we should probably try it sooner rather than later.

[messense]

PyPy support is ready in both pyo3 and maturin. PyPy doesn’t support abi3 yet so we’ll need to build one wheel per PyPy version for each OS and architecture.

I’ll work on it when I have time after releasing maturin 0.13.

[samuelcolvin]

Sounds good, my only concern would be

https://github.com/samuelcolvin/pydantic-core/blob/697b98f310a0055bc163d2a6b981550807c5d3c0/src/validators/model_class.rs#L109-L152

I wonder if that might need to be edited for pypy?

[messense]

I’m on mobile so I can’t say I can answer that. Maybe @davidhewitt can shed some light on this.

[davidhewitt]

GitHub actions supports PyPy so I would recommend adding to your build matrix asap and giving it a shot; for the most part hopefully you should find it just works.

TBH the failure mode for PyO3 is intended to be that if PyPy doesn't support it then compile will fall, either with rustc errors or linker errors.

The snippet above looks like it should be fine as long as the matching ffi calls are implemented.

There have been multiple rounds of investment in PyPy support over the years and afaik it's generally working well for most. I suggest start by adding it to GitHub CI, and if there's a problem I'd be happy to help resolve!

[samuelcolvin]

Thanks so much, will do.

[samuelcolvin]

Well, I've had a go, see #159.

The only feature which pydantic-core uses and isn't available in pypy is get_item_unchecked for list and tuple. I've resolved that with pyo3-build-config.

Two remaining problems:

Build on my m1 mac works, but when I try to run tests I get

ImportError while loading conftest '/Users/samuel/code/pydantic-core/tests/conftest.py'.
tests/conftest.py:11: in <module>
    from pydantic_core import SchemaValidator
pydantic_core/__init__.py:1: in <module>
    from ._pydantic_core import SchemaError, SchemaValidator, ValidationError, __version__
E   ImportError: dlopen(/Users/samuel/code/pydantic-core/pydantic_core/_pydantic_core.pypy39-pp73-darwin.so, 0x0006): tried: '/Users/samuel/code/pydantic-core/pydantic_core/_pydantic_core.pypy39-pp73-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Build on CI is passing and tests are running however, pypy is crashing with:

Fatal Python error: Aborted
Stack (most recent call first, approximate line numbers):
  File "/home/runner/work/pydantic-core/pydantic-core/tests/validators/test_typed_dict.py", line 787 in test_from_attributes_extra

See https://github.com/samuelcolvin/pydantic-core/runs/7253126026?check_suite_focus=true

Some other tests are failing, but I can't get the traceback due to the fatal error.

Any ideas? Hard for me to debug for the next few weeks until I get back to my desktop with x86.

[messense]

pypy does not support arm64 macOS yet, so you need to build for x86_64 with maturin build —-target x86_64-apple-darwin

[davidhewitt]

Hmm I'll try to give it a whirl later

[messense]

Here is a lldb backtrace I got from arm64 macOS: https://gist.github.com/messense/9c7aa0e0ed1b5ae06f010df576d43375

[davidhewitt]

Yep I got a similar one. (Note that to run maturin build --target you need to use maturin 0.13 prerelease.)

I get the impression is that the crashes are caused by deep recursive call stacks which causes PyPy to run out of stack space, which isn't being handled very gracefully. Being not super familiar with the codebase I'm unsure how hard it would be to convert recursive function calls to something like a loop with the recursion tracked on the heap instead to avoid this.

If I run with pytest --capture=no I see errors like:

Fatal error in cpyext, CPython compatibility layer, calling PyDict_New
Either report a bug or consider not using this particular extension
<StackOverflow object at 0x1112d20c8>
RPython traceback:
  File "pypy_module_cpyext_1.c", line 1065, in wrapper_second_level__star_0_11
  File "pypy_module_cpyext_1.c", line 39078, in make_ref
  File "pypy_module_cpyext_2.c", line 19756, in create_ref
  File "implement.c", line 37440, in dispatcher_56
  File "rpython_rlib.c", line 12862, in stack_check_slowpath

I opened #161 which limits the recursion on PyPy, fixing some of the crashes. I had to add skips to three functions, which we can investigate further.

[messense]

pypy does not support arm64 macOS yet, so you need to build for x86_64 with maturin build —-target x86_64-apple-darwin

FYI, maturin develop in the upcoming v0.13.0 release will do this automatically on macOS when default Rust target doesn't match the architecture of Python interpreter: PyO3/maturin#980

[messense]

I got a different error sometimes:

Error in cpyext, CPython compatibility layer:
The function PyErr_Print was not supposed to fail
Fatal error in cpyext, CPython compatibility layer, calling PyErr_Print
Either report a bug or consider not using this particular extension
<SystemError object at 0x10ec47cb8>
RPython traceback:
  File "pypy_module_cpyext.c", line 12321, in wrapper_second_level__star_0_1
  File "pypy_module_cpyext_1.c", line 38697, in not_supposed_to_fail
Fatal Python error: Segmentation fault

[davidhewitt]

Yes, my theory there is that a different PyPy API call fails with a stack overflow, and then returns null, which leads pyo3 to try to call PyErr_Print but there's no error set so things die. Not sure.