Discussion of Python 3.5 support

[dpgeorge]

MicroPython aims to implement the Python 3.x "standard". At the moment x is currently 4, ie we try to be compatible with CPython 3.4. It's new territory as to what to do when CPython evolves to larger version numbers. I would say we should try to follow the changes and implement them when possible/sensible.

Python 3.5 had a feature freeze on 24 May 2015 and is scheduled for final release on 13 September 2015. This ticket here is about discussing if, what and how we should upgrade uPy to version 3.5 of the language. It may be that some issues need to break off into separate tickets and that's fine but we should link to them from this one.

The PEP discussing the new features in 3.5: https://www.python.org/dev/peps/pep-0478/

A more friendly overview in the docs: https://docs.python.org/3.5/whatsnew/3.5.html

Below is a list of finalised/accepted PEPs for 3.5 grouped into their impact to MicroPython.

Extensions to the syntax:

• PEP 448 - additional unpacking generalizations; see Additional function arg unpacking (PEP 448) #5807
• PEP 465 - a new matrix multiplication operator; done in 2069c56
• PEP 492 - coroutines with async and await syntax

Extensions and changes to the runtime:

• PEP 461 - %-formatting for binary strings - tentatively done
• PEP 475 - retrying system calls that fail with EINTR - done in 9418611
• PEP 479 - change StopIteration handling inside generators - done in 3f6ffe0

Standard library changes:

• PEP 471 - os.scandir()
• PEP 485 - math.isclose(), a function for testing approximate equality - done in af5c998

Miscellaneous changes that are not relevant to MicroPython:

• PEP 441 - improved Python zip application support
• PEP 486 - make the Python Launcher aware of virtual environments
• PEP 484 - type hints (advisory only)
• PEP 488 - elimination of PYO files
• PEP 489 - redesigning extension module loading

Other Language Changes

• Added the "namereplace" error handlers. The "backslashreplace" error handlers now work with decoding and translating.
• Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings.
• Circular imports involving relative imports are now supported.

New modules

• typing
• zipapp

Changes to MicroPython built-in modules

• asyncio (many, may need another ticket)
• cmath - A new function isclose() provides a way to test for approximate equality.

• collections
  • The OrderedDict class is now implemented in C, which makes it 4 to 100 times faster.
  • OrderedDict.items(), OrderedDict.keys(), OrderedDict.values() views now support reversed() iteration.
  • The deque class now defines index(), insert(), and copy(), and supports the + and * operators.
  • Docstrings produced by namedtuple() can now be updated.
  • The UserString class now implements the __getnewargs__(), __rmod__(), casefold(), format_map(), isprintable(), and maketrans() methods to match the corresponding methods of str.

• heapq - Element comparison in merge() can now be customized by passing a key function in a new optional key keyword argument, and a new optional reverse keyword argument can be used to reverse element comparison
• io - A new BufferedIOBase.readinto1() method, that uses at most one call to the underlying raw stream's RawIOBase.read() or RawIOBase.readinto() methods.
• json - JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about the error.

• math
  • Two new constants have been added to the math module: inf and nan.
  • A new function isclose() provides a way to test for approximate equality.
  • A new gcd() function has been added. The fractions.gcd() function is now deprecated.
• os
  • The new scandir() function returning an iterator of DirEntry objects has been added.
  • The urandom() function now uses the getrandom() syscall on Linux 3.17 or newer, and getentropy() on OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file descriptor exhaustion.
  • New get_blocking() and set_blocking() functions allow getting and setting a file descriptor's blocking mode (O_NONBLOCK.)
  • There is a new os.path.commonpath() function returning the longest common sub-path of each passed pathname.
• re
  • References and conditional references to groups with fixed length are now allowed in lookbehind assertions.
  • The number of capturing groups in regular expressions is no longer limited to 100.
  • The sub() and subn() functions now replace unmatched groups with empty strings instead of raising an exception.
  • The re.error exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better context information about the error
• socket
  • Functions with timeouts now use a monotonic clock, instead of a system clock.
  • A new socket.sendfile() method allows sending a file over a socket by using the high-performance os.sendfile() function on UNIX, resulting in uploads being from 2 to 3 times faster than when using plain socket.send().
  • The socket.sendall() method no longer resets the socket timeout every time bytes are received or sent. The socket timeout is now the maximum total duration to send all data.
  • The backlog argument of the socket.listen() method is now optional. By default it is set to SOMAXCONN or to 128, whichever is less.
• ssl
  • Memory BIO Support
  • Application-Layer Protocol Negotiation Support
  • There is a new SSLSocket.version() method to query the actual protocol version in use.
  • The SSLSocket class now implements a SSLSocket.sendfile() method.
  • The SSLSocket.send() method now raises either the ssl.SSLWantReadError or ssl.SSLWantWriteError exception on a non-blocking socket if the operation would block. Previously, it would return 0.
  • The cert_time_to_seconds() function now interprets the input time as UTC and not as local time, per RFC 5280. Additionally, the return value is always an int.
  • New SSLObject.shared_ciphers() and SSLSocket.shared_ciphers() methods return the list of ciphers sent by the client during the handshake.
  • The SSLSocket.do_handshake(), SSLSocket.read(), SSLSocket.shutdown(), and SSLSocket.write() methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent.
  • The match_hostname() function now supports matching of IP addresses.
• sys
  • A new set_coroutine_wrapper() function allows setting a global hook that will be called whenever a coroutine object is created by an async def function. A corresponding get_coroutine_wrapper() can be used to obtain a currently set wrapper.
  • A new is_finalizing() function can be used to check if the Python interpreter is shutting down.

• time The monotonic() function is now always available.

(Changes to non-built-in modules will need to be documented elsewhere.)

The above list should be edited if/when progress is made on a given feature.

[pfalcon]

We "by definition" have "%-formatting for binary strings", I'm not sure however what details PEP 461 may have.

[pfalcon]

retrying system calls that fail with EINTR

is potentially interesting, because it's boring to handle it in each app, and actually nobody handles it in each app, unless faces issues, and then it's likely users, and not an author. So, doing that PEP kinda saves from such situations. But then again it may be just adding more bloat than necessary. Need to read PEP, and actually, all PEPs (in detail, again, or at all).

[dpgeorge]

I'm not sure however what details PEP 461 may have.

We are missing: "%b" for interpolating bytes objects, "%a" for repr of bytes, "%c" in uPy truncates to 7-bit char but should be full 8-bit char, and we don't support % operator on bytearray object.

[roger-]

Type hints sound like they could be useful.

Personally I'd like to see unpacking generalizations and async/await support.

[dpgeorge]

I'd like to see unpacking generalizations

I just had a read of that PEP and it looks like it might be some effort, and a decent increase in code size, to implement that. It allows things like lst = [*a, b, *c] and foo(*a, b, *c).

[pfalcon]

Running uPy testsuite against CPython3.5b3 produces few warnings/errors.

[dpgeorge]

Running uPy testsuite against CPython3.5b3 produces few warnings/errors.

A few too many, or few enough to easily fix them? Do you have any examples of failures?

[pfalcon]

I built 3.5b3, and that managed to install itself as system-wide python3, so I scratched ny head for a bit while testsuite fails. As soon as I figured that out, I switched it back and didn't look back, and don't think it's a priority before 3.5 release. So, just FYI ;-).

[pfalcon]

3.5.0 was released today.

[dpgeorge]

CPy 3.5 is now distributed with Arch Linux so that's what I'll be using. There are only a few errors (mostly related to new , * syntax).

[dpgeorge]

@pfalcon you'll be pleased to know that regex splitting with an empty regex is on track for deprecation. Running tests/extmod/ure_split.py with CPy 3.5 gives a FutureWarning for these cases. So this means the dirty magic is hopefully gone!

[dpgeorge]

Tests now pass with CPy 3.5; see 34f26ea.

[dpgeorge]

Evaluation order of dictionary key/value pairs in CPy 3.5 has changed. Key is now evaluated first in a dict literal.

[njouanin]

Hi,
Is Python 3.5 coroutine syntax (await / async) supported be uPy ?

[dpgeorge]

@njouanin await/async syntax is not (yet) supported. But the underlying "yield from" functionality is.

[pfalcon]

More specifically, await/async syntax was initially-prototyped by @dpgeorge, but received the same cool response as the original CPython await/async (during discussion, a lot of people said "we don't need it"). When every 2nd Python programmer will use await/async, we surely will implement it, but now, when it's perhaps every 100000th, why bother.

[dhylands]

i just saw coreio posted - which is based on await/async. I need to look at it closer to understand it, but it sounds like something like it would be usefiul. I need to figure out if something equivalent can be done using what we have (uasyncio).

[pfalcon]

I need to figure out if something equivalent can be done using what we have (uasyncio).

It can be, no worries - all this async/await is just syntactic sugar over yield. It wouldn't be a problem, but it was made deliberately incompatible with classical yield, and that's what I don't like for example.

[pfalcon]

0bb57bf shows some motion towards PEP 475 compliance (but I don't plan to do much of that, and the whole matter is quite depressing).

[pohmelie]

I'm not sure how «big» is new async/await syntax for micropython, but for desktop python it does a lot (finalizing and readability at most):
aio-libs/aioftp@cffbf21?diff=split#diff-95f99d6d5508820e17c812152b4ab040L729
Cause async operations with «classic» loops/context-managers can't be realized. Python is good, because it is simple as for x in seq: # do and it is very important to have same simplicity for async operations. I believe, asynchronous io — future of python (and not only).
@pfalcon

When every 2nd Python programmer will use await/async

Most of python programmers still use python 2, btw. Not because python 2 is better, but they have same arguments.

[pfalcon]

Cause async operations with «classic» loops/context-managers can't be realized.

And that's exactly the problem with implementing it - it's easy to add "await" as an alias for "yield from", it's much harder to implement async with and for. For that, someone needs to actually understand how it works and what are its limits. I don't think many people in the whole world care to do that.

Most of python programmers still use python 2, btw. Not because python 2 is better, but they have same arguments.

And they're right, and its duty of Python3 drivers to make it attractive for people. Adding new useful features is a way to do that, and churning out bunch of features of unknown usability isn't. So, everyone does its job - MicroPython makes as small as possible Python implementation to deepen Python outreach, and CPython works on providing useful features which are to be proven actually used.

Bottom line: if you're interested to provide full async/await implementation, you're certainly welcome too ;-).

[pohmelie]

@pfalcon

it's much harder to implement async with and for. For that, someone needs to actually understand how it works and what are its limits.

I think they just call/async-call magic methods.

[pohmelie]

@pfalcon

Bottom line: if you're interested to provide full async/await implementation, you're certainly welcome too ;-)

I'm afraid, I have not enough skills for that. At first, I don't familar with micropython internals.

[pfalcon]

See #1690.

[pfalcon]

#2015 proposes to bump "supported language" in README to 3.5, now that "async" keyword was implemented.

[stinos]

It is a nice feature though..
What is the general plan? Implement features from 3.5 then move on, or rather implement features of any recent version depending on demand/votes/... ?

For 3.5 the things I already wished we had multiple times are PEP 448 - additional unpacking generalizations and PEP 485 - math.isclose().
That last one isn't too hard, here's a quick implementation:stinos@4c8bb5f Do I make a PR for that or does it need a lot of adjustments?

[dpgeorge]

What is the general plan? Implement features from 3.5 then move on, or rather implement features of any recent version depending on demand/votes/... ?

I would like to get MicroPython to a point where we can claim that it implements Python 3.5, at least the syntax (so it can parse any valid Python 3.5 code). And the main blocker to that is PEP 448 (unpacking generalisations). I looked into it it depth some time ago and came to the conclusion it was not easy, required new bytecodes, and a fair amount of extra code to handle these generalisations. But probably now is a good time to revisit it and make a proof of concept patch to see exactly what is required for that feature. Then make a decision about it.

Apart from that, I don't see any reason not to include features of more recent Python if they are useful and in demand (eg like underscores in numeric literals).

That last one isn't too hard, here's a quick implementation:stinos@4c8bb5f Do I make a PR for that or does it need a lot of adjustments?

It looks ok for a PR, to further discuss it. Is it an original implementation (not copied from CPython)? Also might want to use mp_arg_parse helpers instead of looking up kw's in dicts.

[peterhinch]

If it's not too problematic PEP448 would be useful. There is evident demand: I've already had to explain to users that it's not yet supported.

I appreciate that PEP572 is Python 3.8. My interest is on the theory that it might improve performance and/or code size e.g. in comprehensions and lambdas.

[spacemanspiff2007]

I appreciate that PEP572 is Python 3.8. My interest is on the theory that it might improve performance and/or code size e.g. in comprehensions and lambdas.

Same here! It'll come anyway so it might be as well available in time with the release of 3.8

[nickovs]

Notwithstanding the fact that numpy is very unlikely to get ported to MicroPython, it might we worth implementing PEP 465 anyway since (a) it would probably be easy and small, (b) it would provide compatibility and (c) having a spare infix operator lying around is often handy.

FWIW, I've have ended up implementing a tiny linear algebra system on uPy (they are handy for optimising some dynamic systems and the ESP32 with SPIRAM has the space and horsepower to cope) and having the @ and @= operators would have been quite helpful for improving readability and compactness.

(I also note that MicroPython already supports Ellipsis and it never gets used in any of the standard libraries, so there is a precedent for implementing language features that have no built-in use cases.)

[jimmo]

Notwithstanding the fact that numpy is very unlikely to get ported to MicroPython, it might we worth implementing PEP 465 anyway since (a) it would probably be easy and small, (b) it would provide compatibility and (c) having a spare infix operator lying around is often handy.

@nickovs I have already implemented this in #4740

(I am working on porting a useful subset of numpy, based on ublas/lapack like regular numpy).

[dlech]

PEP 465 can be crossed off of the list now

[dlech]

proposed PEP 475 implementation in #5723

[dlech]

Should PEP 485 be crossed off of the list? #4894

[dpgeorge]

Should PEP 485 be crossed off of the list?

Yes, list is now updated for PEP 475 and 485.

[ghost]

hey guys
i am new in contribution! i want to contribute so how to start?

[ghost]

please help me by contacting at telegram on @Kabraji0
or email at kabraji001@gmail.com

[dlech]

Now that #5807 has been merged, we have partial support for PEP 448, namely:

Function calls are proposed to support an arbitrary number of unpackings rather than just one:

>>> print(*[1], *[2], 3)
1 2 3
>>> dict(**{'x': 1}, y=2, **{'z': 3})
{'x': 1, 'y': 2, 'z': 3}

The remaining bit that is not implemented is:

Unpacking is proposed to be allowed inside tuple, list, set, and dictionary displays:

>>> *range(4), 4
(0, 1, 2, 3, 4)
>>> [*range(4), 4]
[0, 1, 2, 3, 4]
>>> {*range(4), 4}
{0, 1, 2, 3, 4}
>>> {'x': 1, **{'y': 2}}
{'x': 1, 'y': 2}

---

I think we could leverage the partial implementation to easily implement the rest. It could work like this:

1. We need two intrinsic functions that basically look like this:

   def capture_args(*args):
       return args
   
   def capture_kwargs(**kwargs):
       return kwargs

2. If the compiler encounters * in a tuple, list or set display or ** in a dictionary display, then it "rewrites" the code like this:

   *range(4), 4
   # becomes
   tuple(capture_args(*range(4), 4))
   
   [*range(4), 4]
   # becomes
   list(capture_args(*range(4), 4))
   
   {*range(4), 4}
   # becomes
   set(capture_args(*range(4), 4))
   
   {'x': 1, **{'y': 2}}
   # becomes
   dict(capture_kwargs(x=1, **{'y': 2}))

[dpgeorge]

I think we could leverage the partial implementation to easily implement the rest.

That's interesting, and definitely simple... but I'm not sure how to implement intrinsic functions (efficiently).

I wonder how CPython does it, do they create a comprehension function to build the tuple/list/set/dict? If so that would alter the semantics.

[dlech]

I wonder how CPython does it

I looked at the patch that implements it and they just added new opcodes for each case:

+#define BUILD_LIST_UNPACK   	149
+#define BUILD_MAP_UNPACK    	150
+#define BUILD_MAP_UNPACK_WITH_CALL	151
+#define BUILD_TUPLE_UNPACK  	152
+#define BUILD_SET_UNPACK    	153

[dpgeorge]

I looked at the patch that implements it and they just added new opcodes for each case:

I guess we could do that as well.

I think this feature is rarely used (??) so if MicroPython implements it I would say it should be done as "cheaply" as possible, ie with a minimum of firmware size. Maybe adding new opcodes is the smallest way to implement it, or maybe it's some other way.

It looks like the hardest thing (biggest increase in code size) would be to change the compiler to detect these cases. So investigations would be best to start there.