New psoc-edge port, CAN on alif and mimxrt, mem_backup, manifest c_module

[dpgeorge]

Overview

This release of MicroPython adds a new "psoc-edge" port for Infineon PSOC Edge microcontrollers. This is a bare-metal port using CMSIS 6 and a set of low-level HAL submodules from Infineon's Modus Toolbox. It uses the standard ARM toolchains and builds with make. It currently supports a UART REPL, a filesystem (using QSPI flash), machine Pin, UART and RTC classes, including IRQs for Pin and UART. There is currently one supported board, the KIT_PSE84_AI.

The previous v1.28.0 release introduced a new machine.CAN API with an implementation for the stm32 port. This release implements that API on both the alif and mimxrt ports. Much effort went into making sure these three ports have the same consistent behaviour when using CAN, so that reliable applications can be built upon that API.

Also in this release is a new machine.mem_backup() interface which gives consistent access to backup memory on the alif, esp32, mimxrt, nrf, rp2, samd and stm32 ports. Calling this function returns a writable memoryview that survives (in most cases) a hard reset. The available regions can be discovered via machine.mem_backup(-1). See the documentation for more details and examples: https://docs.micropython.org/en/latest/library/machine.html .

Configuring a custom build with user C modules is now a lot simpler thanks to the addition of the c_module() function to the manifest API. This function lets a manifest file declare which user C modules should be included in the build. They are declared alongside the declaration of frozen modules, making it much easier to configure firmware with extensions. For example, a manifest.py file containing:

require("bundle-networking")
freeze("/path/to/custom/modules")
c_module("/path/to/micropython-ulab/code")

will freeze in the networking bundle from micropython-lib, freeze in custom Python code, and include the micropython-ulab C library in the build. This c_module() feature is intended to replace the existing USER_C_MODULES option (although it currently works together with that option).

Unicode support is improved in this release:

• bytes.decode() and three-argument str() now support "strict", "ignore" and "replace" for the errors argument;
• when converting between str and bytes, the encoding/decoding mode is now validated and only "utf-8", "utf8" and "ascii" are accepted;
• str.center() is fixed so it handles Unicode characters correctly;
• all printable Unicode characters are now printed verbatim (not as an escape sequence).

A USB network (Network Control Model, NCM) driver implementation has been added, providing Ethernet-over-USB backed by the lwIP stack, available to Python through the new network.USBD_NCM class. With this enabled, the host computer sees the MicroPython device as a USB Ethernet adapter, and the host is assigned an IP address via DHCP (served by the MicroPython device). Python code on the device can then use all the usual networking facilities to communicate with the host. It's disabled by default, and needs MICROPY_PY_NETWORK_USBD_NCM enabled in a custom build.

The parser now recognises const assignments with type hints, for example _X: int = const(123). There is now support for bytes.find(int) as well as int.to_bytes() with argument signed=True. The RISC-V inline assembler adds support for Zcmp opcodes and has improved register allocation. The LoongArch64 architecture is now supported and that is tested as part of CI.

Native .mpy modules built via mpy_ld.py now automatically include memset, memmove and memcpy in the linking stage so these no longer need to be manually provided. Native modules can also now be built with Clang.

The mbedTLS component is updated to v3.6.6, and support is added for TLS-PSK (pre-shared key) in the SSLContext class. Parsing of PEM certificates is also enabled for all ports using mbedTLS. LittleFS is updated to v2.11.3, TinyUSB to 0.21.0, and CMSIS v6.3.0 is now used in the mimxrt, nrf, psoc-edge, samd and stm32 ports.

Mpremote adds an alias for codeberg repositories, and fixes Unicode errors on Windows as well as some other minor bugs. It also implements incremental preparation for ROMFS deployment, allowing large partition sizes without a timeout.

The alif port implements machine.CAN, machine.WDT and the timeout argument to machine.lightsleep(). It also implements the five states of machine.reset_cause().

The esp32 port updates the IDF to v5.5.2 and adds support for ESP32-H2 chips along with two new board definitions. ESP-NOW is updated to v2.0, a new machine.wake_pins() function is added, mDNS is enabled on the Ethernet interface, the LDO is configurable from the SDCard constructor, and virtual timers are now supported via machine.Timer(-1). A new CSI (Channel State Information) API is exposed via new methods on the WLAN object, allowing access to per-packet physical layer channel data. See documentation and some examples at: https://docs.micropython.org/en/latest/library/network.WLAN.html .

The mimxrt port implements machine.CAN and configures it for all boards that break out a CAN peripheral. It improves event handling and time.sleep(0), fixes flash erase operations on Hyperflash boards, and fixes SDCard caching issues.

The nrf port updates the nrfx library to v3.14.0 (up from v2.0.0), modernises Pin.irq, adds the I2C timeout argument, and changes sys.platform to "nrf" for all boards for consistency.

The rp2 port updates pico-sdk to 2.3.0, refactors the linker scripts, adds support for DMA pacing timers via the rp2.DMATimer class, enables network.PPP on all boards that have networking enabled, and fixes watchdog interactions with machine.lightsleep().

The samd port adds adc.read_timed() and dac.write_timed() methods, and fixes handling of some UART IRQ events.

The stm32 port adds pulse_width_us() and pulse_width_ns() methods to the TimerChannel class, adds the machine.SDCard class (alias of the existing pyb.SDCard), and implements the WWDG watchdog peripheral (in addition to IWDG) with up to four watchdogs on STM32H7. There is now support for FDCAN on the STM32N6, along with significant improvements to network.LAN activation, link detection and hot-plug support. There is also a fix for SDCard caching issues.

The unix port adds the sendall() method to the socket class, and has better support for FreeBSD.

The zephyr port updates Zephyr to v4.4.0, adds support for user C modules, and the I2C driver is reworked to make continuous transactions (rather than a separate write followed by a read).

Thirteen new boards are added in this release:

• esp32 port: ESP32_GENERIC_H2, M5STACK_NANOH2, LILYGO_T3_S3, SEEED_XIAO_ESP32C3, SEEED_XIAO_ESP32C5, SEEED_XIAO_ESP32S3 and WAVESHARE_ESP32_S3_PICO;
• psoc-edge port: KIT_PSE84_AI;
• rp2 port: SOLDERED_NULA_MAX_RP2350 and ADAFRUIT_FEATHER_RP2350;
• stm32 port: STM32H747I_DISCO, WEACTSTUDIO_MINI_STM32H723 and GARATRONIC_PYMATE_CORE8ADI8DOSC.

Breaking changes

MicroPython now validates the encoding argument to str.encode() and bytes.decode() (along with the corresponding constructors), and only "utf-8", "utf8" and "ascii" are accepted. In particular, "latin-1" is not supported, and neither are uppercase variants like "UTF8". The previous behaviour for an unsupported encoding was to silently pass through the data using "utf8" encoding, but MicroPython will now raise a LookupError exception. Affected code should be changed to use a supported encoding.

The MICROPY_FORCE_32BIT flag has been removed from the unix port. Instead, 32-bit x86 is now treated as a cross-compilation target. See the unix port README for more details.

For the esp32 and stm32 ports the machine.SDCard.readblocks() and .writeblocks() methods now return 0 for success and a negative integer for failure. They previously returned a bool with True for success, so the logic is essentially reversed and affected code should be updated.

For the nrf port all boards have been consolidated to use "nrf" for their sys.platform value. Some boards previously used custom names here, such as "nrf51-DK" and "nrf52", but they are all just "nrf" now. All code using sys.platform on nrf boards needs to be updated.

The powerpc port was removed and its functionality merged into the qemu port.

For the stm32 port, on boards with Cortex-M33/M55 MCUs, peripheral register naming in the stm module has been changed such that the constants no longer end in _S or _NS. For example, previously both stm.RTC_S and stm.RTC_NS were provided but they are now removed and replaced by the single stm.RTC. The value of the constant (e.g. stm.RTC) is chosen as the _S or _NS peripheral address depending on whether the firmware is built in secure or non-secure mode, respectively.

Also on the stm32 port, the PRNG implementation of rng_get() has been removed, and this function renamed to mp_hal_get_hw_random_u32() to make it clearer what it does.

Firmware size

The change in code size since the previous release for select builds of various ports is (absolute and percentage change in the text section):

   bare-arm:    +20  +0.04%
minimal x86:    +92  +0.05%
   unix x64: +13673  +1.62%  (standard variant)
       alif:  +9696  +1.23%  (ALIF_ENSEMBLE board)
      esp32: +28328  +1.62%  (ESP32_GENERIC board)
     mimxrt:  +8120  +2.11%  (TEENSY40 board)
 renesas-ra:   +932  +0.15%  (EK_RA6M2 board)
        nrf:  +7072  +3.72%  (PCA10040 board)
        rp2:  +5136  +1.50%  (RPI_PICO board)
        rp2: +30416  +3.33%  (RPI_PICO2_W board)
       samd:  +3112  +1.13%  (ADAFRUIT_ITSYBITSY_M4_EXPRESS board)
      stm32:  +2200  +0.55%  (PYBV10 board)

These size changes provide a good summary of the changes and additions in
the firmware itself. Some of the things responsible for these changes are:

• support for int.to_bytes(signed=True);
• support for bytes.find(int);
• implementing bytes.decode() "ignore" and "replace", and validation of encoding mode;
• addition of machine.mem_backup() to most ports;
• machine.CAN support on alif and mimxrt ports;
• the requests module now supporting HTTP/1.1 on network boards;
• addition of TLS-PSK, PEM parsing and updating to mbedTLS 3.6.6;
• updating to nrfx v3 and enabling of asyncio on the nrf port;
• addition of adc.read_timed() and dac.write_timed() on the samd port;
• addition of pulse_width_us() and pulse_width_ns() on the stm32 port.

Acknowledgements

Thanks to everyone who contributed to this release: Alessandro Gatti, Amirreza Hamzavi, Andrew Leech, Andrii Anoshyn, Angus Gratton, Anson Mansfield, Antonio Galea, Calin Faja, Carl Pottle, Charalampos Stratakis, Chris Mason, Cristian Dinca, Damien George, Dan Halbert, Daniël van de Giessen, David (Pololu), Denis Dowling, Dryw Wade, Duncan Lowther, EngineerWill, Fin Maaß, Finn Glas, Francesco Pace, Frédéric Pierson, Harris, Howard Lovatt, Hugo Frisk, iabdalkader, Ihor Nehrutsa, Jacob Peck, jaenrig-ifx, Jeff Epler, Jeongseop Lim, jetpax, Jim Lipsey, Jim Mussared, Joel Stanley, Jon Nordby, Jos Verlinde, Josip Šimun Kuči, Julia Vassiliki, Keenan Johnson, Kwabena W. Agyeman, Luca Burelli, MarcelloTheArcane, massimosala, Matt Trentini, Meir Armon, Mikhail Zakharov, Mo Nazemi, Mrpli, Ned Konz, Nicko van Someren, npt-1707, o-murphy, Owen, Pavel Revak, Phil Howard, phuzzyday, Pierre Gaufillet, radiofan, Rafal, Rafal Wadowski, Rick Sorensen, rmouro_QCOM, robert-hh, Roman Zeyde, Sandor Attila Gerendi, Tomasz Cwik, Vdragon, vlad.maglasu, Weixie Cui, William Vinnicombe, Yanfeng Liu, Yuuki NAGAO.

And thanks to additional contributors to micropython-lib: Daniel Houck, Greg Darke, Hagb Green, John Moser, kolten, Mark A. Ziesemer, Max Holliday, Michael Bentley, Mike Cifelli, Mr Keuz, Naidile, Pablo Ventura, PermissionDenied7335, puuu, scivision, Stefan, tharuka-pavith, trevor.

MicroPython is a global Open Source project and contributions were made from the following timezones: -0800, -0700, -0600, -0500, -0400, -0300, +0000, +0100, +0200, +0300, +0330, +0530, +0700, +0800, +0900, +0930, +1000, +1100.

The work done in this release was funded in part through GitHub Sponsors, and in part by George Robotics, Espressif, Arduino, OpenMV, and Planet Innovation.

---

This discussion was created from the release New psoc-edge port, CAN on alif and mimxrt, mem_backup, manifest c_module.

[tetopik]

Will the esp32 machine.CAN module comes out in next v1.30 release?

[mattytrentini]

It's still in development, so we'll see. I am working on it when time permits but if you're able to help please reach out. See #19461 for more details.

[peterhinch]

Disallowing "UTF8" in str.encode() and bytes.decode() will break valid CPython code: CPython 3.10.12 allows it.

[projectgus]

Yes, indeed. This is documented as a difference to CPython on the relevant functions:

https://docs.micropython.org/en/v1.29.0/genrst/builtin_types.html#bytes-decode-only-supports-encoding-arguments-utf8-utf-8-and-ascii-other-encodings-like-latin-1-are-not-supported-other-string-forms-such-as-utf8-are-not-supported

https://docs.micropython.org/en/v1.29.0/genrst/builtin_types.html#str-encode-constructor-only-supports-encoding-arguments-utf8-utf-8-and-ascii-other-encodings-like-latin-1-are-not-supported-other-string-forms-such-as-utf8-are-not-supported

CPython encoding parsing is very liberal (things like "UtF_-_8" are also valid), so we had to draw the line somewhere. For example, if we allow "UTF8" then do we also need to allow "UTF-8"?

That said, if not supporting "UTF8" in particular causes non-trivial porting problems with actual code then we could look at adding it.

[MilhouseVH]

mem_backup: Splitting the rp2 scratch registers into 2 regions (0-3 and 5-7) just to avoid the bootrom register seems a little unnecessary, it was at least useful to be able to read that bootrom register [4] (to know in detail what caused a reset), and then there'd only need to be a single contiguous mem_backup() region (at least on RP2040).

Any chance of collapsing the 2 regions and rendering mem[4] visible with a suitable doc warning that it will always be trashed and never persisted? Or even throw an exception on write? Otherwise the mem32 hacks will still be required when mem[4] needs to be read, and then there's no reason at all to switch to mem_backup().