dev call 20230202

Dev Call: 2 February 2023

Present

Petr V., Florian, Stepan, Antonio, Tim, Du Toit, Matti, Simon

Agenda

It's (again) all about the next release 0.9.

• Merged PRs:

  • Multiphase module initialization
  • Use fixed-width integers and clarify encoding of C strings.
  • Improve HPy documentation
  • Minor stylistic improvements to the top-level README
  • Small documentation improvements & quickstart guide

• We need to make decisions for a few things:

  • How to resolve Equivalent of PyObject_TypeCheck
  • Do we want an Eval API? If so, which one?

• Some important PRs still need to be reviewed and merged.

  • Support vectorcall protocol
  • Debug mode: Check correct context usage
  • Exact type checks with HPy_TypeCheck

• Numpy/HPy

  • There is a list of API functions we've introduced in our branch that are not yet on master and maybe will never be:

  • Less controversial ones:

    int HPyType_IsSubtype(HPyContext *ctx, HPy sub, HPy type);
    const char *HPyType_GetName(HPyContext *ctx, HPy type);
    HPy HPySeqIter_New(HPyContext *ctx, HPy seq);
    HPy HPyUnicode_FromEncodedObject(HPyContext *ctx, HPy obj, const char *encoding, const char *errors);
    HPy HPyUnicode_Substring(HPyContext *ctx, HPy obj, HPy_ssize_t start, HPy_ssize_t end);
    HPy HPyDict_Keys(HPyContext *ctx, HPy h);
    int HPySlice_Unpack(HPyContext *ctx, HPy slice, HPy_ssize_t *start, HPy_ssize_t *stop, HPy_ssize_t *step);
    HPy HPyContextVar_New(HPyContext *ctx, const char *name, HPy default_value);
    int HPyContextVar_Get(HPyContext *ctx, HPy context_var, HPy default_value, HPy *result);
    HPy HPyContextVar_Set(HPyContext *ctx, HPy context_var, HPy value);
    

  • Controversial ones:

    HPy HPy_MaybeGetAttr_s(HPyContext *ctx, HPy obj, const char *name); /* more like HPy_GetAttr_NoError */
    int HPy_SetType(HPyContext *ctx, HPy obj, HPy type);               /* should eventually go away */
    HPy HPyDict_GetItem(HPyContext *ctx, HPy op, HPy key);             /* Simon prefers to avoid that and have only the generic API; Antonio argues that it's used a lot. Simon wants HPyDict_GetItemWithError  */
    int HPyType_CheckSlot(HPyContext *ctx, HPy type, HPyDef *value);   /* CPython is up to introduce
    HPy HPyUnicode_InternFromString(HPyContext *ctx, const char *str); /* let's leave that out */
    int HPySequence_Check(HPyContext *ctx, HPy h);
    

• Misc

  • Module state: use separate calling convention? Call for better macro names. https://github.com/hpyproject/hpy/issues/286#issuecomment-1413418275
  • Use fixed width integers (at least in ABI): I'm working on using int64_t for HPy_ssize_t but that's a bit cumbersome. We assume that sizeof(HPy_ssize_t) == sizeof(Py_ssize_t) which could be violated.
  • More doc is needed. In particular API function documentation. If anyone has spare time, please help.
  • We need a few more API functions (or replacements) for Numpy/HPy. I've updated the branches and this is the current diff.

Decisions for release

How to resolve Equivalent of PyObject_TypeCheck

Let's go for option 1 as described in the issue.

Do we want an Eval API? If so, which one?

Florian suggests to implement very basic functions like CPython's Py_CompileString and PyEval_EvalCode. Everyone agrees with that.

Misc

• Use fixed-width integers and clarify encoding of C strings.: Let's not waste time on that and just use ssize_t for HPy_ssize_t.
• Support vectorcall protocol
  • Simon is mostly concerned about having so many different but somehow similar calling conventions.
  • Florian suggests that we could align them. E.g. the varargs convention already uses CPython's fastcall convention but we could also use the vectorcall convention.
  • Antonio suggested to make vectorcall the default (so, maybe no tp_call at all).
  • Stepan asks Petr for the reason of vectorcall; this is for better performance. There are several sources of better performance depending on what is being called. First because tuple's don't need to be created/unpacked. For methods, it's the extra slot in the args array for self. etc.
  • Given that we already use fastcall internally, does vectorcall make sense at all? Petr answers yes because fastcall doesn't address all problems.
  • Stepan suggests that we should think of vectorcall as a slot and slots have different calling conventions.
  • Antonio points out that CPython has also METH_FASTCALL | METH_KEYWORDS, which is the same as vectorcall and HPy could take use of that. Maybe use that instead of the other kwargs calling conventions that HPy has? Petr pointed out that sometimes it is preferable to take keyword args as dictionary if you pass it to some other functions.

Numpy/HPy

• Discussion of controversial API functions used in the Numpy/HPy port:

HPy_SetType

Setting the type is only a temporary thing and eventually go away.

HPyDict_GetItem

Simon prefers to avoid such functions and have only the generic API. Antonio argues that it's used a lot. It's not clear why this function is used. Simon argues that if this is for performance, then again: the runtime should do it (having shortcuts internally instead of doing the full dispatch). If it's because of the different error behavior, then Simon rather prefers to have a HPy(Dict)_GetItemNoError.

HPy_MaybeGetAttr_s

Similar to HpyDict_GetItem: should it be HPy_GetAttr_NoError. Florian suggests to create an issue for further discussion.

HPyType_CheckSlot

CPython is up to introduce something similar. It is therefore also a possible way for us. Petr asks, where Numpy is using that since he wants to track that a bit.

HPyUnicode_InternFromString

It's unclear what guarantees are given with string interning. Matti says, this is a perf optimization. That's something the runtime should do. We won't implement that but use HPyUnicode_FromString instead.

HPySequence_Check

We may need something like this since semantics is derived from slots. Florian will analyze usage of it in Numpy and maybe that helps to make a decision on if and how we should implement that.

Conferences / Sprint

We also quickly talked about conferences.

• Matti will certainly attend SciPy 2023
• Antonio (and Simon?) will certainly be at PyCon US 2023
• One of Tim, Stepan, Florian will try to be at EuroPython 2023
• There is also a Python core dev sprint in autumn 2023 (Brno, CZ)
• We could co-locate an HPy sprint with that.