dev call 20220407

Dev Call: 7 April 2022

Present

Antonio, Simon, Matti, Ronan, Florian, Tim, Stepan

Agenda

• HPy release 0.0.4
• Autogen: separate API and ABI (context)

• HPy binary compatibility

  • Assuming that our ABI will always be backwards compatible is a bit naïve. We should start thinking about how to deal with that.
  • Inspired by SDL2 paper: https://www.reddit.com/r/linux_gaming/comments/1upn39/sdl2_adds_dynamic_api_magic_to_allow_updating_it/ (sorry, couldn't find the original paper)
  • We should think about a policy for supported HPy ABI versions

• Debug mode: The possibility to decide per module makes implementation complicated. Maybe a global decision would be enough (i.e. all contexts are debug contexts or none).
• HPyType_FromSpec: disallow setting any tp_flags not supported by HPy https://github.com/hpyproject/hpy/issues/305
• Remove support for Python 3.6 https://github.com/hpyproject/hpy/issues/307

• NumPy Progress and Plans

  • NpyIter
  • UFunc
  • Wiki with some migration notes: https://github.com/hpyproject/hpy/wiki/Numpy-HPy-migration-notes:-blockers-and-concerns

• Open HPy PRs

  • Capsule PR: https://github.com/hpyproject/hpy/pull/308
  • exact type check: https://github.com/hpyproject/hpy/pull/309
  • detect HPy_Close/return of context constants in debug: https://github.com/hpyproject/hpy/pull/303

HPy Release 0.0.4

• HPy release 0.0.4: no objections; just do it.
• PyPy also started to integrate HPy 0.0.4 already.
• GraalPython is almost ready. Next official release will be in July but we provide snapshot builds for experimenting.
• We ignored (for now) that HPyGlobals should be able to be shared between multiple (sub-)interpreters.
• Reason: The GraalPython team migrated MPL and Kiwisolver to HPy and wants to make those public.

Autogen: separate API and ABI (context) -----------------------------* Autogen API/ABI: we should have a real public header that contains the official API. * People agree, that separation basically makes sense. * Concerns are that API/ABI might run out of sync. * Create and issue and continue discussion there: https://github.com/hpyproject/hpy/issues/311

HPy binary compatibility

• There is some discussion in the Python community. So maybe it's a good time to think about it and make first steps.
• Our binary stability is that we have the context and if we don't change the functions and just add at the end, we are fine. But that's probably not future-proof.
• The HPy context already has a version.
• Rough idea from Tim/Florian how to do it:
  • The extension somehow needs to request the required context version.
  • In multi-phase init, the module can request the context version in the spec.
  • In single-phase init, we need some _protocol to request the HPy context in the right version.
• Idea by Antonio: The extension could just provide some symbol (much like HPyInit_*) that would specify the context. We can generate the symbol in macro HPy_MODINIT.
• What if an extension calls function of different extension and passes the context. They could both require a different context version. We will need some _trampoline or some converter function to call C functions (with HPyContext) of a different extension.
• The CPython approach is: if you can load and call a function that raises an exception, then the extension can at least handle the incompatibility nicely (without an immediate crash).
• Tim and Florian prefer the approach of being able to request a specific context version because that would allow to emulate obsolete/removed ABI functions (maybe at the cost of performance).
• Simon: how hard would it be for impls support older ABI?

Misc

• HPyGlobal: we are seeing a lot of load global - call API fun - close handle and these could maybe be done more efficiently by having a dedicated API for it. For example, in our NumPy
• Simon: talked to two students that will do some HPy benchmarking later this year. We invite them to IRC, mailing list, and dev calls with next month.
• Cython: Florian didn't address the last review comments yet. The PR is still open but priorities are now on Numpy migration. He will eventually finish that ASAP.