-
Notifications
You must be signed in to change notification settings - Fork 993
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
typing: Add typing module #584
base: master
Are you sure you want to change the base?
Conversation
There's also a pretty good / simple typing module here: https://github.com/pfalcon/pycopy-lib/tree/master/typing |
I also want to add, thanks for starting this! While there are a few stub / hack methods of working around typing enough to use autocomplete in IDE's, there's value in a more featured typing module for use with mypy and related tooling |
yes but seemed fairly old and had less types than I found in CPython so I just started from scratch |
Would love this. I didn't even realize until just now that there isn't a typing module till I tried to add type hints for an optional parameter just now. Thanks you! |
Thanks for the reminder @andrewleech. Some thoughts....
So... could we get most of what people need by having Furthermore, could we take this one step further and implement a special case in the compiler to just ignore any import statements from Additionally, as far as I understand Python 3.9 now mostly makes importing stuff from the |
I like most of what's proposed here, in particular having a minimal on-board "stub" to reduce runtime costs. Even with the newer syntax not needing eg. List there's still a bunch of typing only "constructs" like Union or Optional which I use a lot, though maybe there's new syntax to replace that too... I'll have to take a look. After that there's an even shorter list of functions in typing such as NewType and cast which are used outside type annotations, so would crash out as a None. Maybe instead of returning None the getattr could return something callable like
or a similar lambda or similar? |
I think this should suffice, and then this would be my preferred way: no |
…fügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…fügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…fügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…fügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock Type benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock benutzt. CUST-234
…ugefügt Micropython bietet kein typing Modul (ist in Arbeit, siehe micropython/micropython-lib#584). Stattdessen wird ein Mock benutzt. CUST-234
I had a go at implementing a "simplified" on-device stub version based on a It's common practice when using type hinting to create type aliases of complex types for reuse, eg. from typing import Union, Tuple
LedLevels: TypeAlias = Tuple[Union[int, float, None], Union[int, float, None], Union[int, float, None]] While it might just be a case of "you can't do this with microypthon" it's a bit of a shame... it works with the existing stub library here. This slightly bigger on-device stub does seem to work with my above use case and any similar "real" use of typing features: class _TypeIgnore:
def __init__(*args, **kwargs):
pass
def __call__(self, *args, **kwargs):
return self
def __getitem__(self, attr):
return self
_type_ignore = _TypeIgnore()
def __getattr__(attr):
return _type_ignore |
|
||
class ClassVar: | ||
pass | ||
|
||
|
||
class Final: | ||
pass | ||
|
||
|
||
class Hashable: | ||
pass | ||
|
||
|
||
class IO: | ||
pass | ||
|
||
|
||
class NoReturn: | ||
pass | ||
|
||
|
||
class Sized: | ||
pass | ||
|
||
|
||
class SupportsInt: | ||
pass | ||
|
||
|
||
class SupportsFloat: | ||
pass | ||
|
||
|
||
class SupportsComplex: | ||
pass | ||
|
||
|
||
class SupportsBytes: | ||
pass | ||
|
||
|
||
class SupportsIndex: | ||
pass | ||
|
||
|
||
class SupportsAbs: | ||
pass | ||
|
||
|
||
class SupportsRound: | ||
pass | ||
|
||
|
||
class TextIO: | ||
pass | ||
|
||
|
||
AnyStr = str | ||
Text = str | ||
Pattern = str | ||
Match = str | ||
TypedDict = dict | ||
|
||
AbstractSet = _Subscriptable | ||
AsyncContextManager = _Subscriptable | ||
AsyncGenerator = _Subscriptable | ||
AsyncIterable = _Subscriptable | ||
AsyncIterator = _Subscriptable | ||
Awaitable = _Subscriptable | ||
Callable = _Subscriptable | ||
ChainMap = _Subscriptable | ||
Collection = _Subscriptable | ||
Container = _Subscriptable | ||
ContextManager = _Subscriptable | ||
Coroutine = _Subscriptable | ||
Counter = _Subscriptable | ||
DefaultDict = _Subscriptable | ||
Deque = _Subscriptable | ||
Dict = _Subscriptable | ||
FrozenSet = _Subscriptable | ||
Generator = _Subscriptable | ||
Generic = _Subscriptable | ||
Iterable = _Subscriptable | ||
Iterator = _Subscriptable | ||
List = _Subscriptable | ||
Literal = _Subscriptable | ||
Mapping = _Subscriptable | ||
MutableMapping = _Subscriptable | ||
MutableSequence = _Subscriptable | ||
MutableSet = _Subscriptable | ||
NamedTuple = _Subscriptable | ||
Optional = _Subscriptable | ||
OrderedDict = _Subscriptable | ||
Sequence = _Subscriptable | ||
Set = _Subscriptable | ||
Tuple = _Subscriptable | ||
Type = _Subscriptable | ||
Union = _Subscriptable | ||
|
||
TYPE_CHECKING = False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def cast(type, val): | |
return val | |
def get_origin(type): | |
return None | |
def get_args(type): | |
return () | |
def no_type_check(arg): | |
return arg | |
def overload(func): | |
return None | |
class _AnyCall: | |
def __init__(*args, **kwargs): | |
pass | |
def __call__(*args, **kwargs): | |
pass | |
_anyCall = _AnyCall() | |
class _SubscriptableType: | |
def __getitem__(self, arg): | |
return _anyCall | |
_Subscriptable = _SubscriptableType() | |
def TypeVar(type, *types): | |
return None | |
def NewType(name, type): | |
return type | |
class Any: | |
pass | |
class BinaryIO: | |
pass | |
class ClassVar: | |
pass | |
class Final: | |
pass | |
class Hashable: | |
pass | |
class IO: | |
pass | |
class NoReturn: | |
pass | |
class Sized: | |
pass | |
class SupportsInt: | |
pass | |
class SupportsFloat: | |
pass | |
class SupportsComplex: | |
pass | |
class SupportsBytes: | |
pass | |
class SupportsIndex: | |
pass | |
class SupportsAbs: | |
pass | |
class SupportsRound: | |
pass | |
class TextIO: | |
pass | |
AnyStr = str | |
Text = str | |
Pattern = str | |
Match = str | |
TypedDict = dict | |
AbstractSet = _Subscriptable | |
AsyncContextManager = _Subscriptable | |
AsyncGenerator = _Subscriptable | |
AsyncIterable = _Subscriptable | |
AsyncIterator = _Subscriptable | |
Awaitable = _Subscriptable | |
Callable = _Subscriptable | |
ChainMap = _Subscriptable | |
Collection = _Subscriptable | |
Container = _Subscriptable | |
ContextManager = _Subscriptable | |
Coroutine = _Subscriptable | |
Counter = _Subscriptable | |
DefaultDict = _Subscriptable | |
Deque = _Subscriptable | |
Dict = _Subscriptable | |
FrozenSet = _Subscriptable | |
Generator = _Subscriptable | |
Generic = _Subscriptable | |
Iterable = _Subscriptable | |
Iterator = _Subscriptable | |
List = _Subscriptable | |
Literal = _Subscriptable | |
Mapping = _Subscriptable | |
MutableMapping = _Subscriptable | |
MutableSequence = _Subscriptable | |
MutableSet = _Subscriptable | |
NamedTuple = _Subscriptable | |
Optional = _Subscriptable | |
OrderedDict = _Subscriptable | |
Sequence = _Subscriptable | |
Set = _Subscriptable | |
Tuple = _Subscriptable | |
Type = _Subscriptable | |
Union = _Subscriptable | |
TYPE_CHECKING = False | |
from micropython import const | |
TYPE_CHECKING = const(False) | |
if TYPE_CHECKING: | |
# Type Checkers on PC will "override" this dynamically to turn on TYPE_CHECKING and include below | |
def TypeVar(type, *types): | |
return None | |
def NewType(name, type): | |
return type | |
def cast(type, val): | |
return val | |
def get_origin(type): | |
return None | |
def get_args(type): | |
return () | |
def no_type_check(arg): | |
return arg | |
def overload(func): | |
return None | |
class _AnyCall: | |
def __init__(*args, **kwargs): | |
pass | |
def __call__(*args, **kwargs): | |
pass | |
_anyCall = _AnyCall() | |
class _SubscriptableType: | |
def __getitem__(self, arg): | |
return _anyCall | |
_Subscriptable = _SubscriptableType() | |
class Any: | |
pass | |
class BinaryIO: | |
pass | |
class ClassVar: | |
pass | |
class Final: | |
pass | |
class Hashable: | |
pass | |
class IO: | |
pass | |
class NoReturn: | |
pass | |
class Sized: | |
pass | |
class SupportsInt: | |
pass | |
class SupportsFloat: | |
pass | |
class SupportsComplex: | |
pass | |
class SupportsBytes: | |
pass | |
class SupportsIndex: | |
pass | |
class SupportsAbs: | |
pass | |
class SupportsRound: | |
pass | |
class TextIO: | |
pass | |
AnyStr = str | |
Text = str | |
Pattern = str | |
Match = str | |
TypedDict = dict | |
AbstractSet = _Subscriptable | |
AsyncContextManager = _Subscriptable | |
AsyncGenerator = _Subscriptable | |
AsyncIterable = _Subscriptable | |
AsyncIterator = _Subscriptable | |
Awaitable = _Subscriptable | |
Callable = _Subscriptable | |
ChainMap = _Subscriptable | |
Collection = _Subscriptable | |
Container = _Subscriptable | |
ContextManager = _Subscriptable | |
Coroutine = _Subscriptable | |
Counter = _Subscriptable | |
DefaultDict = _Subscriptable | |
Deque = _Subscriptable | |
Dict = _Subscriptable | |
FrozenSet = _Subscriptable | |
Generator = _Subscriptable | |
Generic = _Subscriptable | |
Iterable = _Subscriptable | |
Iterator = _Subscriptable | |
List = _Subscriptable | |
Literal = _Subscriptable | |
Mapping = _Subscriptable | |
MutableMapping = _Subscriptable | |
MutableSequence = _Subscriptable | |
MutableSet = _Subscriptable | |
NamedTuple = _Subscriptable | |
Optional = _Subscriptable | |
OrderedDict = _Subscriptable | |
Sequence = _Subscriptable | |
Set = _Subscriptable | |
Tuple = _Subscriptable | |
Type = _Subscriptable | |
Union = _Subscriptable | |
else: | |
# On-device when compiled to mpy the above should be stripped out leaving just this section. | |
class _TypeIgnore: | |
def __init__(*args, **kwargs): | |
pass | |
def __call__(self, *args, **kwargs): | |
return self | |
def __getitem__(self, attr): | |
return self | |
_type_ignore = _TypeIgnore() | |
def __getattr__(attr): | |
return _type_ignore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how well it works suggesting this inline, but this is my attempt at providing an on-device simplified runtime stub, but keeping the "full" details for PC based type checking. mpy-cross
does strip out the bulk of it, compiled to mpy
beforehand was about 1.7KB, with this change it brings it down to 289 bytes for me.
If the stub/mock works then what would still be the benefit of keeping the rest? Catching import errors like typos? Would also be a bit tedious to have to manually change the TYPE_CHECKING constant. |
Yeah I'm a little conflicted about keeping the rest... having it there can help autocomplete / linting in some tools. You don't need to manually change the const, the tools generally do that dynamically themselves. However I'm working on getting full mypy working with https://github.com/Josverl/micropython-stubs and the typing stub modules there, so this copy isn't actually used on PC in that case. |
If you're linting, you're using CPython or similar anyway which has its own proper typing module (well, I guess, see question below). For autocomplete it might indeed help, on the other hand I'm not sure if it's really such an issue if it doesn't (especially if you're used writing micropython-based code already) since it's mostly on a 'first time it is needed' basis anyway: once you've added a couple of typing statements in a project any sane autocomplete will pick them up.
Why does it need a typing stub, can't it not use CPython's typing module? (with the benefit that a linter would then effectively interpret the types correctly, not sure if it can do that based on a stub?). |
There's a few weird things about the real cpython typing module, for example it pulls in a few packages to create type aliases like importing collections.defaultdict to alias to DefaultDict. If I'm trying to use micropython specific stubs for built-in packages there's a bunch of these that get broken. If I let it use cpython from the dependent imports, it ends up being a long thread of various not-quite-compatible stuff that results in more and more errors. So it ends up being a trimmed / modified typing module being used in the stubs package. To be clear the typing module I'm taking about is part of the stdlib stubs package and is based on the real cpython one, it's not really a stub as such but is named pyi, it's intended to be consumed by tools rather than executed I guess. https://pypi.org/project/micropython-stdlib-stubs I've patched this more for mypy compat but haven't managed to figure out how to PR my patches upstream yet. This doesn't really explain why I should keep any of the extra detail in this installable typing module... maybe there really isn't any reason at all. |
@andrewleech lets chat,. I'm interested in what changes you made to get mypy working better. |
@Josverl yes I've been meaning to reach out / raise a bunch of merge requests, but each time I think I've got it all working I find another exclusion / override left in my pyproject.toml that, once reset to defaults, highlights a bunch of things still broken. My plan is to finish cleaning up basic support, then get changes into git, then slice and dice the changes into stdlib logical commits/pr and other things into the code generator stuff as appropriate. It's just been a slow part time project so far, but has already unearthed some bugs / incorrect code in my application codebase so showing its value! |
On the types needed by the stubs: Currently the types I use to build the stubs are the Uppercase variants in order to be compatible with older Python versions.
a few likely candidates for reduction are:
Micropython-stdlib has a bunch more - but perhaps that could be also trimmed down further. Would that help ? |
ref: micropython/micropython-lib#584 Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Ive been bundling the The error reported is on
I cannot find a reference of
what was the origin of your |
The AnyCall in this PR doesn't come from anywhere necessarily I don't think, it's just a minimal stub that implements It's then wrapped in simple Subscriptable type that's used by everything that should be used the way, it would be useful to see a bug report about what usage threw the error? |
See Josverl/micropython-stubs#755 (comment) for a specific example of the error. |
is this still something being considered, or is typing going to be ignored by the interpreter? |
@jack60612 For now you can add add this to your mcu or freeze it to shave off some memory. |
@Josverl @andrewleech we should move this forward; did you come to a conlusion about the various versions (typing.py doesn't have To summarize these seem to be the possibilities:
|
C / Python implementationIn my current testing I can compile the .py files down to approx 440 bytes, so if the C implementation takes less than that im all for it. Freezing may reduce that further. minimal compiled versions using
The .py version with some extra'sIf I add just the conditional on TYPE_CHECKINGThese are runtime modules , so
|
related: micropython/micropython-lib#584 Signed-off-by: Jos Verlinde <Jos.Verlinde@microsoft.com>
I had a proper look at this. My first idea was to ignore any typing imports but that makes things rather complicated because statements like 'from typing import List' are split in 'import typing' and 'lookup the List attribute on the module' but for that last one there should be an actual mp_obj_t available. I skipped on that and tried an alternative: make typing an actual but empty module with just an attr function which returns None for everything. Small in size, works ok for standard type hints, but this case immediately breaks
because typing has nothing in it,
because I made 'from typing import ...' return None. Also note that even if ignoring anything typing-related in C would work, the 2 above examples would also fail with that mechanism. So: unless I'm missing something, ignoring typing in C isn't going to work, and a working implementation would basically mean implementing pretty much everything the .py file does in C. Seeing it's not a lof of code after all, I wonder if that's going to be much smaller than freezing the module. |
It's likely not.. A C implementation of this module
is 436 bytes here. Freezing the Python module is 480 bytes. So an 'obscure' C implementation doesn't really seem worth it. Let's see if Andrew has a preference. |
Very interesting investigation about code options and size, thanks guys. To be honest I'm really conflicted about all this myself! It seems like it should be unnecessary to ever cost any on-device bytes for this sort of feature that's really just used by on-pc tooling. But then we have to consider there's two separate environments that need to share the one application codebase:
The cpython tooling should never see the fake typing stub being discussed here. Similarly I don't see any value in a more full featured typing library that runs on micropython - the extensive stubs from @Josverl fill this need and are installed into cpython venv, not the micropython application. But making the code that uses these cpython compatible typing features compatible enough to run in the micropython environment is obviously crucial. I do keep falling back on my first workaround; just wrapping any import and usage of typing with a try/except. This does not cost a module&class worth of code but it's ugly, has a very slight runtime overhead during an often-time-sensitive startup import phase, and doesn't really help if you want to reuse a cpython module without modifying stuff. I do like @jimmo's import skip suggestion but as you found @stinos it's more complicated that it seems, what with the from/import structure and potential use of imported objects outside of pure type hinting syntax. I did also think perhaps the typing import could be filtered out in the compiler, similar to how the type hints themselves already are. I've also thought of ways to make a C stub typing module that might be much smaller, but haven't written this yes to see if it comes out small enough to justify. Lastly there's the current python minimal version presented here. It has the advantage of flexibility - users can install it only if/ when it's needed by the users application. |
The advantage of a typing.py/.mpy module is that it
I'm not sure if we should consider any difference in runtime impact to memory allocation/fragmentation or processing overhead. |
Brought up a couple of times here and on the forums.
To implement this I just took most things from
dir(typing)
in CPython 3.8, removed what looked too esoteric and kept the rest. Alphabetically so it's easier to find things. Probably still too much, but I really cannot guess what is used and what is not.Closes #190.