function annotation for builtin and C function

[bhy]

BPO	3208
Nosy	@amauryfa, @vstinner, @larryhastings, @benjaminp

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-06-26.13:16:17.368>
labels = ['type-feature']
title = 'function annotation for builtin and C function'
updated_at = <Date 2013-12-04.01:45:05.212>
user = 'https://bugs.python.org/bhy'

bugs.python.org fields:

activity = <Date 2013-12-04.01:45:05.212>
actor = 'larry'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2008-06-26.13:16:17.368>
creator = 'bhy'
dependencies = []
files = []
hgrepos = []
issue_num = 3208
keywords = []
message_count = 18.0
messages = ['68783', '69629', '69678', '70104', '70167', '70170', '70178', '70179', '70186', '70192', '70194', '70195', '70196', '70422', '163251', '163252', '205184', '205186']
nosy_count = 7.0
nosy_names = ['amaury.forgeotdarc', 'vstinner', 'larry', 'richardb', 'benjamin.peterson', 'bhy', 'Ramchandra Apte']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3208'
versions = ['Python 3.4']

[bhy]

It is better if the function annotation(PEP-3107) can be supported by
built-in function and C function writtin in extension module, just like
the __doc__ attribute.

[avassalotti]

Extension modules can use PyFunction_GetAnnotations() to access and
modify the annotations dictionary. In addition,
PyFunction_SetAnnotations() can be used to add annotations.

I added some documentation for these functions in r64934.

[bhy]

Sorry I haven't state the issue clearly. For this issue I mean the
built-in function should able to define an __annotations__ attribute,
just like the __doc__ attribute, but not to access it in extension module.

[amauryfa]

PyCFunctionObject has indeed no way to store annotations. This could be
useful for extension module writers.

The PyMethodDef structure could grow a "ml_annotations" member. A patch
is welcome!

[bhy]

By considering the implementing, some problems emerged.

First of all, as we know, all CFunctionObject and their attributes are
imutable, but the __annotations__ attribute should be a dict, and dict
is mutable. So how to solve this?

Secondly, the annotation value can be abitrary expression, and then,
for extension module, would it be reasonable to restrict these value to
string?

Thanks!

[amauryfa]

• A immmutable object may contain mutable members. Try with a tuple
  containing a list.
  Then, I don't think that something says that CFunctionObjects are
  immutable. They don't have any modifiable attribute, until today!

• (Did I say "string"?) The new PyMethodDef::ml_annotations would not be
  a char*, but a PyObject* member. If it is not possible to set it in the
  static array, one could update the array in the module init function.

Anyway, for a SWIG module I think the best is to set the __annotations__
in the shadow python file. It seems more practical to build the dict there.

[bhy]

I think there is reason that CFunctionObjects are immutable: single
CFunctionObject is shared by mutiple Python interpreters, so any change
of CFunctionObject would affect other Python interpreters. Is that
right?

If it should be immutable, then we should use something like static
array to assign annotations to CFunctionObject, and the value also
should be immutable, that means the value couldn't be abitrary
PyObject. (by value I mean the value of every __annotations__ dict
items.)

For SWIG, there's a way to bypass the Python side proxy, eg. for a
simple C function, in the shadow module we directly
let 'func=_cmod.func', where _cmod is the C DLL module. So the
annotation information would be lost if we can't directly assign
annotation to C function.

[benjaminp]

There never should be multiple Python interpreters running in the same
process, though.

[bhy]

As I understand, at least C extension modules, which built as shared
library, would be shared among Python interpreter in different process
space. Is that correct?

[benjaminp]

On Wed, Jul 23, 2008 at 8:44 PM, Haoyu Bai <report@bugs.python.org> wrote:

Haoyu Bai <divinekid@gmail.com> added the comment:

As I understand, at least C extension modules, which built as shared
library, would be shared among Python interpreter in different process
space. Is that correct?

The operating system should provide memory protection between processes.

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3208\>

---

[amauryfa]

Shared libraries share code, not memory.

But were you talking about sub-interpreters?
http://docs.python.org/dev/c-api/init.html#Py_NewInterpreter
mod_python uses them, but see the "Caveats" section of the doc.

[bhy]

I found the explanation of why buitl-ins are immutable:

For the curious: there are two reasons why changing built-in classes is
disallowed. First, it would be too easy to break an invariant of a
built-in type that is relied upon elsewhere, either by the standard
library, or by the run-time code. Second, when Python is embedded in
another application that creates multiple Python interpreters, the
built-in class objects (being statically allocated data structures) are
shared between all interpreters; thus, code running in one interpreter
might wreak havoc on another interpreter, which is a no-no.

(From http://www.python.org/download/releases/2.2.3/descrintro/)

Is the statement still valid for current version of Python?

[amauryfa]

The "First" argument does not apply here, we could just say "annotations
are not a function invariant", but the "Second" argument is valid to me.

A solution would be a global (or interpreter-local if we really want to
support sub-interpreters) registry that stores annotations. The index
could not be the PyCFunctionObject (since it is different for every
bound method), but the address of the PyMethodDef entry.

[richardb]

I don't think it's reasonable not to support multiple interpreters in a
single process - they're quite widely used by mod_python and mod_wsgi,
and probably by others. I'm not sure whether that's a problem here or
not, though.

If we need to allow function annotations to be arbitrary PyObjects,
these PyObject pointers can't (in general) refer to statically allocated
python objects, so some extension modules will have to allocate them in
the module initialisation function (and presumably deallocate them again
when the module is unloaded).

I would have thought that any such PyObjects are going to be valid only
from within a single interpreter. Perhaps I'm wrong.

Certainly it would be unpleasant if a change to one of the objects in
one interpreter was reflected in other interpreters, but if that didn't
risk causing a crash due to the memory allocation going wrong, or
something equally nasty, it might be acceptable.

[RamchandraApte]

What is the status of this bug?

[benjaminp]

Awaiting a patch.

[vstinner]

This issue has been addressed by the PEP-436 (Argument Clinic) which supports annotation per parameter and annotation on the return type. This PEP has been implemented in Python 3.4.

I suggest to close the issue, but I would prefer that Larry closes the issue instead of me, he wrote the PEP.

[larryhastings]

Argument Clinic theoretically could support annotations for builtins, though it's never been tested. I don't know if it makes sense to close this bug yet.