typing: Add typing module

[stinos]

Brought up a couple of times here and on the forums.

To implement this I just took most things from dir(typing) in CPython 3.8, removed what looked too esoteric and kept the rest. Alphabetically so it's easier to find things. Probably still too much, but I really cannot guess what is used and what is not.

Closes #190.

[andrewleech]

There's also a pretty good / simple typing module here: https://github.com/pfalcon/pycopy-lib/tree/master/typing
I've got a rebased copy of this locally that I pulled out as part of my (unfinished) pdb work if it looks useful?

[andrewleech]

I also want to add, thanks for starting this!

While there are a few stub / hack methods of working around typing enough to use autocomplete in IDE's, there's value in a more featured typing module for use with mypy and related tooling

[stinos]

looks useful

yes but seemed fairly old and had less types than I found in CPython so I just started from scratch

[JayPalm]

Would love this. I didn't even realize until just now that there isn't a typing module till I tried to add type hints for an optional parameter just now. Thanks you!

[jimmo]

Thanks for the reminder @andrewleech. Some thoughts....

• Having a typing module on the device is obviously important if you want to be able to directly run code that has type annotations. However, I understand that most of this is just to make the from typing import List line work, because the actual annotations are ignored by the compiler (i.e. I can write def foo(x: abc) -> xyz:) and the compiler doesn't care if abc or xyz exist.
• However, it's a pretty unfortunate waste of flash+RAM to just make a no-op import work.

So... could we get most of what people need by having typing.py just implement a module-level __getattr__ that returns None to every attribute? Then you could write from typing import AnythingYouLike and the compiler would be happy.

Furthermore, could we take this one step further and implement a special case in the compiler to just ignore any import statements from typing ?

Additionally, as far as I understand Python 3.9 now mostly makes importing stuff from the typing module obsolete. You can write x: list[int] now (not List[int]). I guess code will continue to exist for a long time with typing imports.

[andrewleech]

I like most of what's proposed here, in particular having a minimal on-board "stub" to reduce runtime costs.

Even with the newer syntax not needing eg. List there's still a bunch of typing only "constructs" like Union or Optional which I use a lot, though maybe there's new syntax to replace that too... I'll have to take a look.

After that there's an even shorter list of functions in typing such as NewType and cast which are used outside type annotations, so would crash out as a None.

Maybe instead of returning None the getattr could return something callable like

def null(*args, **kwargs):
    return None

or a similar lambda or similar?

[stinos]

Furthermore, could we take this one step further and implement a special case in the compiler to just ignore any import statements from typing ?

I think this should suffice, and then this would be my preferred way: no typing module needed at all, all type annotations simply ignored.

[andrewleech]

I had a go at implementing a "simplified" on-device stub version based on a __getattr__ approach, immediately ran into a bunch of issues with my existing project codebase :-(

It's common practice when using type hinting to create type aliases of complex types for reuse, eg.

from typing import Union, Tuple
LedLevels: TypeAlias = Tuple[Union[int, float, None], Union[int, float, None], Union[int, float, None]]

While it might just be a case of "you can't do this with microypthon" it's a bit of a shame... it works with the existing stub library here.

This slightly bigger on-device stub does seem to work with my above use case and any similar "real" use of typing features:

class _TypeIgnore:
    def __init__(*args, **kwargs):
        pass

    def __call__(self, *args, **kwargs):
        return self

    def __getitem__(self, attr):
        return self

_type_ignore = _TypeIgnore()

def __getattr__(attr):
    return _type_ignore

[andrewleech]

Suggested change

	def cast(type, val):
	return val
	def get_origin(type):
	return None
	def get_args(type):
	return ()
	def no_type_check(arg):
	return arg
	def overload(func):
	return None
	class _AnyCall:
	def __init__(*args, **kwargs):
	pass
	def __call__(*args, **kwargs):
	pass
	_anyCall = _AnyCall()
	class _SubscriptableType:
	def __getitem__(self, arg):
	return _anyCall
	_Subscriptable = _SubscriptableType()
	def TypeVar(type, *types):
	return None
	def NewType(name, type):
	return type
	class Any:
	pass
	class BinaryIO:
	pass
	class ClassVar:
	pass
	class Final:
	pass
	class Hashable:
	pass
	class IO:
	pass
	class NoReturn:
	pass
	class Sized:
	pass
	class SupportsInt:
	pass
	class SupportsFloat:
	pass
	class SupportsComplex:
	pass
	class SupportsBytes:
	pass
	class SupportsIndex:
	pass
	class SupportsAbs:
	pass
	class SupportsRound:
	pass
	class TextIO:
	pass
	AnyStr = str
	Text = str
	Pattern = str
	Match = str
	TypedDict = dict
	AbstractSet = _Subscriptable
	AsyncContextManager = _Subscriptable
	AsyncGenerator = _Subscriptable
	AsyncIterable = _Subscriptable
	AsyncIterator = _Subscriptable
	Awaitable = _Subscriptable
	Callable = _Subscriptable
	ChainMap = _Subscriptable
	Collection = _Subscriptable
	Container = _Subscriptable
	ContextManager = _Subscriptable
	Coroutine = _Subscriptable
	Counter = _Subscriptable
	DefaultDict = _Subscriptable
	Deque = _Subscriptable
	Dict = _Subscriptable
	FrozenSet = _Subscriptable
	Generator = _Subscriptable
	Generic = _Subscriptable
	Iterable = _Subscriptable
	Iterator = _Subscriptable
	List = _Subscriptable
	Literal = _Subscriptable
	Mapping = _Subscriptable
	MutableMapping = _Subscriptable
	MutableSequence = _Subscriptable
	MutableSet = _Subscriptable
	NamedTuple = _Subscriptable
	Optional = _Subscriptable
	OrderedDict = _Subscriptable
	Sequence = _Subscriptable
	Set = _Subscriptable
	Tuple = _Subscriptable
	Type = _Subscriptable
	Union = _Subscriptable
	TYPE_CHECKING = False
	from micropython import const
	TYPE_CHECKING = const(False)
	if TYPE_CHECKING:
	# Type Checkers on PC will "override" this dynamically to turn on TYPE_CHECKING and include below
	def TypeVar(type, *types):
	return None
	def NewType(name, type):
	return type
	def cast(type, val):
	return val
	def get_origin(type):
	return None
	def get_args(type):
	return ()
	def no_type_check(arg):
	return arg
	def overload(func):
	return None
	class _AnyCall:
	def __init__(*args, **kwargs):
	pass
	def __call__(*args, **kwargs):
	pass
	_anyCall = _AnyCall()
	class _SubscriptableType:
	def __getitem__(self, arg):
	return _anyCall
	_Subscriptable = _SubscriptableType()
	class Any:
	pass
	class BinaryIO:
	pass
	class ClassVar:
	pass
	class Final:
	pass
	class Hashable:
	pass
	class IO:
	pass
	class NoReturn:
	pass
	class Sized:
	pass
	class SupportsInt:
	pass
	class SupportsFloat:
	pass
	class SupportsComplex:
	pass
	class SupportsBytes:
	pass
	class SupportsIndex:
	pass
	class SupportsAbs:
	pass
	class SupportsRound:
	pass
	class TextIO:
	pass
	AnyStr = str
	Text = str
	Pattern = str
	Match = str
	TypedDict = dict
	AbstractSet = _Subscriptable
	AsyncContextManager = _Subscriptable
	AsyncGenerator = _Subscriptable
	AsyncIterable = _Subscriptable
	AsyncIterator = _Subscriptable
	Awaitable = _Subscriptable
	Callable = _Subscriptable
	ChainMap = _Subscriptable
	Collection = _Subscriptable
	Container = _Subscriptable
	ContextManager = _Subscriptable
	Coroutine = _Subscriptable
	Counter = _Subscriptable
	DefaultDict = _Subscriptable
	Deque = _Subscriptable
	Dict = _Subscriptable
	FrozenSet = _Subscriptable
	Generator = _Subscriptable
	Generic = _Subscriptable
	Iterable = _Subscriptable
	Iterator = _Subscriptable
	List = _Subscriptable
	Literal = _Subscriptable
	Mapping = _Subscriptable
	MutableMapping = _Subscriptable
	MutableSequence = _Subscriptable
	MutableSet = _Subscriptable
	NamedTuple = _Subscriptable
	Optional = _Subscriptable
	OrderedDict = _Subscriptable
	Sequence = _Subscriptable
	Set = _Subscriptable
	Tuple = _Subscriptable
	Type = _Subscriptable
	Union = _Subscriptable
	else:
	# On-device when compiled to mpy the above should be stripped out leaving just this section.
	class _TypeIgnore:
	def __init__(*args, **kwargs):
	pass
	def __call__(self, *args, **kwargs):
	return self
	def __getitem__(self, attr):
	return self
	_type_ignore = _TypeIgnore()
	def __getattr__(attr):
	return _type_ignore

[andrewleech]

Not sure how well it works suggesting this inline, but this is my attempt at providing an on-device simplified runtime stub, but keeping the "full" details for PC based type checking. mpy-cross does strip out the bulk of it, compiled to mpy beforehand was about 1.7KB, with this change it brings it down to 289 bytes for me.

[stinos]

If the stub/mock works then what would still be the benefit of keeping the rest? Catching import errors like typos? Would also be a bit tedious to have to manually change the TYPE_CHECKING constant.

[andrewleech]

Yeah I'm a little conflicted about keeping the rest... having it there can help autocomplete / linting in some tools. You don't need to manually change the const, the tools generally do that dynamically themselves.

However I'm working on getting full mypy working with https://github.com/Josverl/micropython-stubs and the typing stub modules there, so this copy isn't actually used on PC in that case.

[stinos]

having it there can help autocomplete / linting in some tools

If you're linting, you're using CPython or similar anyway which has its own proper typing module (well, I guess, see question below). For autocomplete it might indeed help, on the other hand I'm not sure if it's really such an issue if it doesn't (especially if you're used writing micropython-based code already) since it's mostly on a 'first time it is needed' basis anyway: once you've added a couple of typing statements in a project any sane autocomplete will pick them up.

and the typing stub modules there

Why does it need a typing stub, can't it not use CPython's typing module? (with the benefit that a linter would then effectively interpret the types correctly, not sure if it can do that based on a stub?).

[andrewleech]

There's a few weird things about the real cpython typing module, for example it pulls in a few packages to create type aliases like importing collections.defaultdict to alias to DefaultDict.

If I'm trying to use micropython specific stubs for built-in packages there's a bunch of these that get broken.

If I let it use cpython from the dependent imports, it ends up being a long thread of various not-quite-compatible stuff that results in more and more errors.

So it ends up being a trimmed / modified typing module being used in the stubs package.

To be clear the typing module I'm taking about is part of the stdlib stubs package and is based on the real cpython one, it's not really a stub as such but is named pyi, it's intended to be consumed by tools rather than executed I guess.

https://pypi.org/project/micropython-stdlib-stubs

I've patched this more for mypy compat but haven't managed to figure out how to PR my patches upstream yet.

This doesn't really explain why I should keep any of the extra detail in this installable typing module... maybe there really isn't any reason at all.

[Josverl]

@andrewleech lets chat,. I'm interested in what changes you made to get mypy working better. micropython-stdlib-stubs is maintained manually , and I've got quality tests setup that can be extended to mypy, so making changes should be simpler with little risk of degradation

[andrewleech]

@Josverl yes I've been meaning to reach out / raise a bunch of merge requests, but each time I think I've got it all working I find another exclusion / override left in my pyproject.toml that, once reset to defaults, highlights a bunch of things still broken.

My plan is to finish cleaning up basic support, then get changes into git, then slice and dice the changes into stdlib logical commits/pr and other things into the code generator stuff as appropriate.

It's just been a slow part time project so far, but has already unearthed some bugs / incorrect code in my application codebase so showing its value!

[Josverl]

On the types needed by the stubs:

Currently the types I use to build the stubs are the Uppercase variants in order to be compatible with older Python versions.
Ive not researched it yet, but as typing is more and more part of the newer Python versions, I think that could probably be relaxed to Types that are already available without adding overhead
Below is the entire list currently explicitly used in micropython-stubs

TYPING_IMPORT: List[str] = [
    "from typing import IO, Any, Callable, Coroutine, Dict, Generator, Iterator, List, NoReturn, Optional, Tuple, Union, NamedTuple, TypeVar",
    "from _typeshed import Incomplete",
]

a few likely candidates for reduction are:

• Dict --> dict
• List --> list
• Iterator --> list
• Tuple --> tuple
• NamedTuple --> collections.namedtuple

Micropython-stdlib has a bunch more - but perhaps that could be also trimmed down further.

Would that help ?

[Josverl]

@andrewleech ,

Ive been bundling the typings.py in micropython-stubs , and one of the users, @ANogin, found a runtime issue when defining Generics.

The error reported is on class _AnyCall ,

TypeError: '_AnyCall' object isn't subscriptable
Josverl/micropython-stubs#755

I cannot find a reference of _AnyCall in pylance/mypy typing (Python 3.11), while I can find usage of _AnyCallable.

_AnyCallable: TypeAlias = Callable[..., object]
in functools.pyi

what was the origin of your _AnyCall and could / should we adopt it or replace it by _AnyCallable ?
Is that the same / similar ?

[andrewleech]

The AnyCall in this PR doesn't come from anywhere necessarily I don't think, it's just a minimal stub that implements __call__().

It's then wrapped in simple Subscriptable type that's used by everything that should be used the way, it would be useful to see a bug report about what usage threw the error?

[ANogin]

The AnyCall in this PR doesn't come from anywhere necessarily I don't think, it's just a minimal stub that implements __call__().

It's then wrapped in simple Subscriptable type that's used by everything that should be used the way, it would be useful to see a big report about what usage threw the error?

See Josverl/micropython-stubs#755 (comment) for a specific example of the error.

[jack60612]

is this still something being considered, or is typing going to be ignored by the interpreter?

[Josverl]

@jack60612 For now you can add add this to your mcu or freeze it to shave off some memory.
mpremote mip install github:josverl/micropython-stubs/mip/typing.mpy

• Docs

[stinos]

@Josverl @andrewleech we should move this forward; did you come to a conlusion about the various versions (typing.py doesn't have if TYPE_CHECKING for example)?

To summarize these seem to be the possibilities:

• jimmo's idea of making micropython ignore any typing import; would be my preferred solution, should I have a go at that or would an actual typing.py be preffered?
• andrew's typing.py with

class _TypeIgnore:
    def __init__(*args, **kwargs):
        pass

    def __call__(self, *args, **kwargs):
        return self

    def __getitem__(self, attr):
        return self

_type_ignore = _TypeIgnore()

def __getattr__(attr):
    return _type_ignore

• the above, but conditional on TYPE_CHECKING such that classes can still be defined if needed
• something else?

[Josverl]

C / Python implementation

In my current testing I can compile the .py files down to approx 440 bytes, so if the C implementation takes less than that im all for it. Freezing may reduce that further.
I don't know the other effects on memory - but creating objects either way is bound to use and cause some memory fragmentation.

minimal compiled versions using mpy-cross mp_abc_min.py -O3

stubtestprojects\rt_typing
30-08-2024  18:24               116 mp_abc_min.mpy
30-08-2024  18:24               324 mp_typing_min.mpy
               2 File(s)            440 bytes

The .py version with some extra's

If I add just the __getattr__(attr): I find I can omit most of the other classes , reducing the overall size significantly.
the class _TypeIgnore class is the same as the _AnyCall class , so 1 is sufficent , I do like the name

conditional on TYPE_CHECKING

These are runtime modules , so TYPE_CHECKING will always be False by definition.

from abc import ABC

I think we can expect that ABC will also be used, so I did a quick test on that as well.

"""
abc.py - Micropython runtime Abstract Base Classes module
"""
from typing import _any_call  # type: ignore
def abstractmethod(funcobj):
    return funcobj
def __getattr__(attr):
    return _any_call

That allows me to run though a a few test , but it is not perfect as it fails on : class MyABC(metaclass=ABCMeta): with a
TypeError: function doesn't take keyword arguments
Ive not found a way around this , also not sure how common this sample is.

Testing

I have made my current test in a jupyter notebook running against a board.
that is fine for prototyping , but we need a body of scripts / snippets to test against, most likely on the unix port for ease of testing.
I could extend some of my stub-quality tests, but only part of them were actually made to be actually run.

[stinos]

if the C implementation takes less

I had a proper look at this. My first idea was to ignore any typing imports but that makes things rather complicated because statements like 'from typing import List' are split in 'import typing' and 'lookup the List attribute on the module' but for that last one there should be an actual mp_obj_t available.

I skipped on that and tried an alternative: make typing an actual but empty module with just an attr function which returns None for everything. Small in size, works ok for standard type hints, but this case immediately breaks

from typing import *

Vector = List[float]

because typing has nothing in it, List isn't defined. Not sure how common that is though. This also doesn't work:

from typing import List

Vector = List[float]

because I made 'from typing import ...' return None. Also note that even if ignoring anything typing-related in C would work, the 2 above examples would also fail with that mechanism.

So: unless I'm missing something, ignoring typing in C isn't going to work, and a working implementation would basically mean implementing pretty much everything the .py file does in C. Seeing it's not a lof of code after all, I wonder if that's going to be much smaller than freezing the module.

[stinos]

Seeing it's not a lof of code after all, I wonder if that's going to be much smaller than freezing the module.

It's likely not.. A C implementation of this module

class _TypeIgnore:
    def __init__(*args, **kwargs):
        pass

    def __call__(self, *args, **kwargs):
        return self

    def __getitem__(self, attr):
        return self

_type_ignore = _TypeIgnore()

def __getattr__(attr):
    return _type_ignore

is 436 bytes here. Freezing the Python module is 480 bytes. So an 'obscure' C implementation doesn't really seem worth it. Let's see if Andrew has a preference.

[andrewleech]

Very interesting investigation about code options and size, thanks guys.

To be honest I'm really conflicted about all this myself!

It seems like it should be unnecessary to ever cost any on-device bytes for this sort of feature that's really just used by on-pc tooling.
Especially when most forms on type information is stripped out by the micropython computer, any extra typing support code just seems like wasted space.

But then we have to consider there's two separate environments that need to share the one application codebase:

• PC / cpython tooling (ruff, mypy, pyright / vscode etc)
• On-device micropython

The cpython tooling should never see the fake typing stub being discussed here. Similarly I don't see any value in a more full featured typing library that runs on micropython - the extensive stubs from @Josverl fill this need and are installed into cpython venv, not the micropython application.

But making the code that uses these cpython compatible typing features compatible enough to run in the micropython environment is obviously crucial.

I do keep falling back on my first workaround; just wrapping any import and usage of typing with a try/except. This does not cost a module&class worth of code but it's ugly, has a very slight runtime overhead during an often-time-sensitive startup import phase, and doesn't really help if you want to reuse a cpython module without modifying stuff.

I do like @jimmo's import skip suggestion but as you found @stinos it's more complicated that it seems, what with the from/import structure and potential use of imported objects outside of pure type hinting syntax.

I did also think perhaps the typing import could be filtered out in the compiler, similar to how the type hints themselves already are.
Perhaps any usage of any objects imported from typing could be similarly filtered out - but I haven't delved into the compiler code before to get a feel for how much work this is.

I've also thought of ways to make a C stub typing module that might be much smaller, but haven't written this yes to see if it comes out small enough to justify.

Lastly there's the current python minimal version presented here. It has the advantage of flexibility - users can install it only if/ when it's needed by the users application.
But as I started with, this is really just wasted space because the application isn't really using the typing code.
So yeah, I'm conflicted on the most appropriate way forward...

[Josverl]

The advantage of a typing.py/.mpy module is that it

• can be simply added to the source modules of anyone that wants to use these, and thus has no impact at all to the firmware size.
• is mip installable
• can be changed simpler compared to a C-implementation
  -can be frozen into custom firmware

I'm not sure if we should consider any difference in runtime impact to memory allocation/fragmentation or processing overhead.
On one hand I think both impacts are small, but on the other hand static typing is becoming more and more part of Python.
I Think it's worth clarifying such impact in advance, to help make a good implementation choice.