Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use PEP 484 type annotations for static type declarations #1672

Closed
ghost opened this issue Apr 11, 2017 · 24 comments
Closed

Use PEP 484 type annotations for static type declarations #1672

ghost opened this issue Apr 11, 2017 · 24 comments

Comments

@ghost
Copy link

ghost commented Apr 11, 2017

Python already has static typing via the typing module. It should be possible to use this rather than cdef.

@robertwb
Copy link
Contributor

Though I (and others) have thought about doing this, the implementation wouldn't be as straightforward as it seems. For example, would a type of int be a C int or a Python int? PEP 484-style types are supposed to have no runtime effects; would a .py file compiled with Cython that has type annotations violate this? Cython's type system is neither a subset of superset of the typing module's system, and it's not clear the best way to extend it. (E.g. would List[str] require runtime checks on getting and setting items?) Also, cdef declares more than just type, it's also used to declare scope (e.g. cdef, cpdef, and def functions cannot be inferred from the argument/return types, and we need modifiers on class attribute visibility).

That being said, I would like to see this fleshed out into a full proposal (maybe on the wiki) that addresses these issues.

@ghost
Copy link
Author

ghost commented Apr 12, 2017

Type definitions for functions should remain the same. I hadn't thought of this initially, but using decorators for the functions rather than the cdef would probably be better as well. I'll start a wiki page on this sometime soon.

@robertwb
Copy link
Contributor

That'd be great. See also http://cython.readthedocs.io/en/latest/src/tutorial/pure.html

@ghost
Copy link
Author

ghost commented Apr 14, 2017

@datnamer
Copy link

@JukkaL and @ilevkivskyi had similiar ideas.

@ilevkivskyi
Copy link

Note that PEP 526 syntax for variable annotation (works in Python 3.6) will be also useful.

Concerning Python int vs C int, my idea was to create an additional module (maybe ctyping?) that will contain more precise low-level types for annotations, plus few special constructs that will be interpreted specially, e.g.

from ctyping import int_8, float_32, Struct

class Data(Struct):
    number: int_8
    value: float_32

(compare this to how typing.NamedTuple is implemented and interpret by tools like mypy)

@robertwb
Copy link
Contributor

Thanks. I expanded the wiki some.

Ideally variables like int_8 could be (c)imported from their definitions in libc.stdint, so we only need the fundamental C types to come from somewhere.

Interesting idea inheriting from Struct (and by analogy Union, Enum). I had thought of using decorators to annotate classes like this instead. I wonder if cdef extern blocks could be phrased as Python-valid blocks as well (with statements? If they lived in a namespace, a class would be natural).

@ghost
Copy link
Author

ghost commented Apr 20, 2017

The proposal that I had suggested using ctypes for type annotations. There's no need to bring in a custom module when python already has all of the c types builtin.

@shalabhc
Copy link
Contributor

Cython's pure mode already has objects to represent some C types. Could we just use that E.g.

@cython.func
def c_compare(a:cython.int, b:cython.int) -> cython.int:
    return a == b

MyStruct = cython.struct(x=cython.int, y=cython.int, data=cython.double)

@cython.func
def get_sum(s: MyStruct) -> cython.int:
   return s.x + s.y

Cython already works with these types, so only the method of annotation changes.

PEP 484-style types are supposed to have no runtime effects; would a .py file compiled with Cython that has type annotations violate this?

I think in general this principle shouldn't be violated. However if the functions are wrapped with a decorator as above, then it is clear that the function is being transformed, and the expectation is not violated IMO.

@ilevkivskyi
Copy link

PEP 484-style types are supposed to have no runtime effects; would a .py file compiled with Cython that has type annotations violate this?

This part of the PEP only applies to default behaviour of Python implementations. There are already runtime type checkers that use decorators, import hooks, etc. to enforce type hints.
I think it is perfectly OK if Cython will "enforce" annotations provided there is control over this, like a compilation flag, per-file import, or per-function decorator.

@mrkaspa
Copy link

mrkaspa commented May 5, 2017

This is a great idea and will provide more concistency for using python and cython together

@techdragon
Copy link

@ilevkivskyi has a very important point. Cython, especially when compiling rather than interpreting, which is already a non-standard way of running python. Should be free to use the type annotations as necessary to get the most value for Cython. The only qualification I can see is that it would be best to make sure that how the type hints are used, is designed so that it provides maximum compatibility with non-cython use.

@scoder
Copy link
Contributor

scoder commented Aug 9, 2017

It appears that typing has some support for user defined types, which should allow redeclaring Cython's type system using the typing module. Basically, it would mean rewriting most of what the Cython Shadow.py module defines for pure Python mode, based on the type classes exported by typing.

Pull requests for that are welcome. Mapping that to the corresponding Cython types during compilation should then be rather straight forward.

@scoder
Copy link
Contributor

scoder commented Aug 21, 2017

I just remembered that I had disabled the annotation typing by default since it was unclear at the time how it would interact with the typing annotations. With the stubs in #1826 and the directive annotation_typing=True, I think it should be possible to write Python code with static Cython annotations in PEP 484 syntax, pretty much as in #1672 (comment)

Here's how I originally intended it to be used:
https://github.com/cython/cython/blob/0.26/tests/run/annotation_typing.pyx
It seems that we should now rather strip that down to what PEP 484 actually allows, i.e. remove the dict and string expression support.

@scoder
Copy link
Contributor

scoder commented Aug 22, 2017

FYI, I tried using types like cython.int or SomeStruct through annotations and the compiler picks them up but then fails with a type lookup error afterwards. Should be fixable, though. I'll take a look, but it doesn't have a high priority for me currently.

@scoder
Copy link
Contributor

scoder commented Sep 2, 2017

Implemented in latest master as of 1504019. Please give it a try and comment.

@scoder scoder added this to the 0.27 milestone Sep 2, 2017
@stsievert
Copy link

stsievert commented Sep 2, 2017

Perfect! Thanks for this. This is a great first step.

In 1504019:

Only Cython types (e.g. cython.int) and Python builtin types are
currently considered as type declarations. Everything else is ignored [...]

Is there support for np.ndararay too?

@ilevkivskyi
Copy link

Implemented in latest master as of 1504019. Please give it a try and comment.

Great! I don't have time right now, will try later. But the first impression is similar, it would be great to support some basic generics like List[int] and numpy.ndarray. (There are actually some basic stubs for the latter, see https://github.com/machinalis/mypy-data/tree/master/numpy-mypy, so that mypy partially works with it, more support is planned later for fixed size/shape arrays, see python/mypy#3345 and python/mypy#3540.)

@scoder
Copy link
Contributor

scoder commented Sep 2, 2017

Supporting ndarray in pure mode feels like a separate ticket to me. The crux there is to support it in both Python interpreted and Cython compiled code. It has nothing to do with the annotation syntax specifically.

When I wrote that "everything else is ignored", I meant the typing expressions like List[int] or Optional[Tuple[...]]. While some of them could be used as Cython type declarations, supporting them can happy come later. They are not critical and most of them are not even useful (especially the standard container types aren't).

@scoder
Copy link
Contributor

scoder commented Sep 2, 2017

I also got structs working now:

MyStruct = cython.struct(x=cython.int, y=cython.int, data=cython.double)

def struct_io(s : MyStruct) -> MyStruct:
    """
    >>> d = struct_io(dict(x=1, y=2, data=3))
    >>> sorted(d.items())
    [('data', 3.0), ('x', 2), ('y', 1)]
    """
    t = s
    t.x, t.y = s.y, s.x
    return t

There are probably other issues with other kinds of type declarations hidden here and there. Testing is really the only way to uncover them. Pull requests welcome for more tests in the pep526_variable_annotations.pyx and annotation_typing.pyx files.

@scoder
Copy link
Contributor

scoder commented Sep 20, 2017

Any more comments on this? I've pushed a beta release for testing.
https://github.com/cython/cython/releases/tag/0.27b1

@scoder scoder changed the title Possible to use Python static typing? Use PEP 484 type annotations for static type declarations Sep 20, 2017
@scoder
Copy link
Contributor

scoder commented Sep 21, 2017

Since the most relevant parts of this are implemented, I'll close this ticket. Further improvements can be done in the form of pull requests and/or new tickets.

@scoder scoder closed this as completed Sep 21, 2017
@lxkain
Copy link

lxkain commented Nov 15, 2017

If I understand correctly, we can now write:

import cython

@cython.cfunc  # cdef functions are faster but not callable from python
def fun(x: float) -> float:  
    return x**2-x

@cython.locals(i=cython.int, N=cython.int)  # better integration soon
@cython.cdivision(True)  # remove divide by zero protection
def integrate_f(a: float, b: float, N: int):
    s : float = 0
    dx : float = (b - a) / N
    for i in range(N):
        s += fun(a + i * dx)
    return s * dx

Question: what would it take, and would it be possible, to (1) have python's int type default to cython.int in the function annotation for variable N, and (2) to be able to declare or infer that loop variable i is cython.int?

@scoder
Copy link
Contributor

scoder commented Nov 15, 2017

(1) would be unsafe since a Python int has arbitrary size and C int is limited to 32bit max and wraps around. When people use int as an annotation, they may actually mean any integer value and not necessarily C int (or C long).

(2) you already did that with the @cython.locals() decorator, but you can also type it before the loop with PEP 526 syntax.

That being said, I think this ticket is closed for good. For further questions, please ask on the cython-users mailing list.

@cython cython locked and limited conversation to collaborators Nov 15, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants