Skip to content

Commit

Permalink
esp32/README: Update based on new IDF v4 cmake build process.
Browse files Browse the repository at this point in the history 
Signed-off-by: Damien George <damien@micropython.org>
  • Loading branch information
@dpgeorge
dpgeorge committed Feb 15, 2021
1 parent aa3d6b6 commit e017f27
Showing 1 changed file with 76 additions and 169 deletions.
245 changes: 76 additions & 169 deletions ports/esp32/README.md
View file
@@ -1,195 +1,70 @@
MicroPython port to the ESP32
=============================

This is an experimental port of MicroPython to the Espressif ESP32
microcontroller. It uses the ESP-IDF framework and MicroPython runs as
This is a port of MicroPython to the Espressif ESP32 series of
microcontrollers. It uses the ESP-IDF framework and MicroPython runs as
a task under FreeRTOS.

Supported features include:
- REPL (Python prompt) over UART0.
- 16k stack for the MicroPython task and 96k Python heap.
- 16k stack for the MicroPython task and approximately 100k Python heap.
- Many of MicroPython's features are enabled: unicode, arbitrary-precision
integers, single-precision floats, complex numbers, frozen bytecode, as
well as many of the internal modules.
- Internal filesystem using the flash (currently 2M in size).
- The machine module with GPIO, UART, SPI, software I2C, ADC, DAC, PWM,
TouchPad, WDT and Timer.
- The network module with WLAN (WiFi) support.
- Bluetooth low-energy (BLE) support via the bluetooth module.

Development of this ESP32 port was sponsored in part by Microbric Pty Ltd.
Initial development of this ESP32 port was sponsored in part by Microbric Pty Ltd.

Setting up the toolchain and ESP-IDF
------------------------------------
Setting up ESP-IDF and the build environment
--------------------------------------------

There are two main components that are needed to build the firmware:
- the Xtensa cross-compiler that targets the CPU in the ESP32 (this is
different to the compiler used by the ESP8266)
- the Espressif IDF (IoT development framework, aka SDK)
MicroPython on ESP32 requires the Espressif IDF version 4 (IoT development
framework, aka SDK). The ESP-IDF includes the libraries and RTOS needed to
manage the ESP32 microcontroller, as well as a way to manage the required
build environment and toolchains needed to build the firmware.

The ESP-IDF changes quickly and MicroPython only supports certain versions.
Currently MicroPython supports v4.0.2, although other IDF v4 versions may also
work.

To install the ESP-IDF the full instructions can be found at the
[Espressif Getting Started guide](https://docs.espressif.com/projects/esp-idf/en/v4.0.2/get-started/index.html#installation-step-by-step).

If you are on a Windows machine then the [Windows Subsystem for
Linux](https://msdn.microsoft.com/en-au/commandline/wsl/install_guide) is the
most efficient way to install the ESP32 toolchain and build the project. If
you use WSL then follow the Linux instructions rather than the Windows
instructions.

The ESP-IDF changes quickly and MicroPython only supports certain versions.
The git hash of these versions (one for 3.x, one for 4.x) can be found by
running `make` without a configured `ESPIDF`. Then you can fetch the
required IDF using the following command:

```bash
$ cd ports/esp32
$ make ESPIDF= # This will print the supported hashes, copy the one you want.
$ export ESPIDF=$HOME/src/github.com/espressif/esp-idf # Or any path you like.
$ mkdir -p $ESPIDF
$ cd $ESPIDF
$ git clone https://github.com/espressif/esp-idf.git $ESPIDF
$ git checkout <Current supported ESP-IDF commit hash>
$ git submodule update --init --recursive
```

Note: The ESP IDF v4.x support is currently experimental. It does not
currently support PPP or wired Ethernet.

Python dependencies
===================

You will also need other dependencies from the IDF, see
`$ESPIDF/requirements.txt`, but at a minimum you need `pyserial>=3.0` and
`pyparsing>=2.0.3,<2.4.0`.

You can use Python 2 or Python 3. If you need to override the system default
add (for example) `PYTHON=python3` to any of the `make` commands below.

It is recommended to use a Python virtual environment. Even if your system
package manager already provides these libraries, the IDF v4.x is currently
incompatible with pyparsing 2.4 and higher.

For example, to set up a Python virtual environment from scratch:

```bash
$ cd ports/esp32
$ python3 -m venv build-venv
$ source build-venv/bin/activate
$ pip install --upgrade pip
$ pip install -r path/to/esp-idf/requirements.txt
```
The Espressif instructions will guide you through using the `install.sh`
(or `install.bat`) script to download the toolchain and set up your environment.
The steps to take are summarised below.

To re-enter this virtual environment in future sessions, you only need to
source the `activate` script, i.e.:
To check out a copy of the IDF use git clone:

```bash
$ cd ports/esp32
$ source build-venv/bin/activate
$ git clone -b v4.0.2 --recursive https://github.com/espressif/esp-idf.git
```

Then, to install the toolchain (which includes the GCC compiler, linker, binutils,
etc), there are two options:

1. Using the IDF scripts to install the toolchain (IDF 4.x only)
================================================================

Follow the steps at the [Espressif Getting Started guide](https://docs.espressif.com/projects/esp-idf/en/v4.0/get-started/index.html#step-3-set-up-the-tools).

This will guide you through using the `install.sh` (or `install.bat`) script
to download the toolchain and add it to your `PATH`. The steps are summarised
below:

After you've cloned and checked out the IDF to the correct version (see
above), run the `install.sh` script:

```bash
$ cd $ESPIDF
$ ./install.sh # (or install.bat on Windows)
```
(You don't need a full recursive clone; see the `ci_esp32_setup` function in
`tools/ci.sh` in this repository for more detailed set-up commands.)

Then in the `ports/esp32` directory, source the `export.sh` script to set the
`PATH`.
After you've cloned and checked out the IDF to the correct version, run the
`install.sh` script:

```bash
$ cd micropython/ports/esp32
$ source $ESPIDF/export.sh # (or path\to\esp-idf\export.bat on Windows)
$ # Run make etc, see below.
$ cd esp-idf
$ ./install.sh # (or install.bat on Windows)
$ source export.sh # (or export.bat on Windows)
```

The `install.sh` step only needs to be done once. You will need to source
`export.sh` for every new session.

Note: If you get an error about `--no-site-packages`, then modify
`$ESPIDF/tools/idf_tools.py` and make the same change as [this
commit](https://github.com/espressif/esp-idf/commit/7a18f02acd7005f7c56e62175a8d1968a1a9019d).

2. or, Downloading pre-built toolchain manually (IDF 3.x and 4.x)
=============================================================

Note: while this works with 4.x, if you're using the 4.x IDF, it's much
simpler to use the guide above, which will also get a more recent version of
the toolchain.

You can follow the 3.x guide at:

* [Linux installation](https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started/linux-setup.html)
* [MacOS installation](https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started/macos-setup.html)
* [Windows installation](https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started/windows-setup.html)

You will need to update your `PATH` environment variable to include the ESP32
toolchain. For example, you can issue the following commands on (at least)
Linux:

$ export PATH=$PATH:$HOME/esp/crosstool-NG/builds/xtensa-esp32-elf/bin

You can put this command in your `.profile` or `.bash_login`, or do it manually.

Configuring the MicroPython build
---------------------------------

You then need to set the `ESPIDF` environment/makefile variable to point to
the root of the ESP-IDF repository. The recommended way to do this is to have
a custom `makefile` in `ports/esp32` which sets any additional variables, then
includes the main `Makefile`. Note that GNU Make will preferentially run
`GNUmakefile`, then `makefile`, then `Makefile`, which is what allows this to
work. On case-insensitive filesystems, you'll need to use `GNUmakefile` rather
than `makefile`.

Create a new file in the esp32 directory called `makefile` (or `GNUmakefile`)
and add the following lines to that file:

```
ESPIDF ?= <path to root of esp-idf repository>
BOARD ?= GENERIC
#PORT ?= /dev/ttyUSB0
#FLASH_MODE ?= qio
#FLASH_SIZE ?= 4MB
#CROSS_COMPILE ?= xtensa-esp32-elf-
include Makefile
```

Be sure to enter the correct path to your local copy of the IDF repository
(and use `$(HOME)`, not tilde (`~`), to reference your home directory).

If the Xtensa cross-compiler is not in your path you can use the
`CROSS_COMPILE` variable to set its location. Other options of interest are
`PORT` for the serial port of your ESP32 module, and `FLASH_MODE` (which may
need to be `dio` for some modules) and `FLASH_SIZE`. See the Makefile for
further information.

The default ESP IDF configuration settings are provided by the `GENERIC`
board definition in the directory `boards/GENERIC`. For a custom configuration
you can define your own board directory.

Any of these variables can also be set on the make command line, e.g. to set
the `BOARD` variable, use:

```bash
$ make BOARD=TINYPICO
```

Note the use of `?=` in the `makefile` which allows them to be overridden on
the command line. There is also a `GENERIC_SPIRAM` board for for ESP32
modules that have external SPIRAM, but prefer to use a specific board target
(or define your own as necessary).

Building the firmware
---------------------

Expand All @@ -198,8 +73,7 @@ built-in scripts to bytecode. This can be done by (from the root of
this repository):

```bash
$ cd mpy-cross
$ make mpy-cross
$ make -C mpy-cross
```

Then to build MicroPython for the ESP32 run:
Expand All @@ -210,16 +84,14 @@ $ make submodules
$ make
```

This will produce binary firmware images in the `build/` subdirectory
(three of them: bootloader.bin, partitions.bin and application.bin).
This will produce a combined `firmware.bin` image in the `build-GENERIC/`
subdirectory (this firmware image is made up of: bootloader.bin, partitions.bin
and micropython.bin).

To flash the firmware you must have your ESP32 module in the bootloader
mode and connected to a serial port on your PC. Refer to the documentation
for your particular ESP32 module for how to do this. The serial port and
flash settings are set in the `Makefile`, and can be overridden in your
local `makefile`; see above for more details.

You will also need to have user permissions to access the /dev/ttyUSB0 device.
for your particular ESP32 module for how to do this.
You will also need to have user permissions to access the `/dev/ttyUSB0` device.
On Linux, you can enable this by adding your user to the `dialout` group, and
rebooting or logging out and in again. (Note: on some distributions this may
be the `uucp` group, run `ls -la /dev/ttyUSB0` to check.)
Expand All @@ -242,8 +114,23 @@ To flash the MicroPython firmware to your ESP32 use:
$ make deploy
```

This will use the `esptool.py` script (provided by ESP-IDF) to flash the
binary images to the device.
The default ESP32 board build by the above commands is the `GENERIC` one, which
should work on most ESP32 modules. You can specify a different board by passing
`BOARD=<board>` to the make commands, for example:

```bash
$ make BOARD=GENERIC_SPIRAM
```

Note: the above "make" commands are thin wrappers for the underlying `idf.py`
build tool that is part of the ESP-IDF. You can instead use `idf.py` directly,
for example:

```bash
$ idf.py build
$ idf.py -D MICROPY_BOARD=GENERIC_SPIRAM build
$ idf.py flash
```

Getting a Python prompt on the device
-------------------------------------
Expand All @@ -262,6 +149,8 @@ or
$ miniterm.py /dev/ttyUSB0 115200
```

You can also use `idf.py monitor`.

Configuring the WiFi and using the board
----------------------------------------

Expand Down Expand Up @@ -301,9 +190,27 @@ import machine
antenna = machine.Pin(16, machine.Pin.OUT, value=0)
```

Defining a custom ESP32 board
-----------------------------

The default ESP-IDF configuration settings are provided by the `GENERIC`
board definition in the directory `boards/GENERIC`. For a custom configuration
you can define your own board directory. Start a new board configuration by
copying an existing one (like `GENERIC`) and modifying it to suit your board.

MicroPython specific configuration values are defined in the board-specific
`mpconfigboard.h` file, which is included by `mpconfigport.h`. Additional
settings are put in `mpconfigboard.cmake`, including a list of `sdkconfig`
files that configure ESP-IDF settings. Some standard `sdkconfig` files are
provided in the `boards/` directory, like `boards/sdkconfig.ble`. You can
also define custom ones in your board directory.

See existing board definitions for further examples of configuration.

Configuration
Troubleshooting
---------------

* Continuous reboots after programming: Ensure FLASH_MODE is correct for your
board (e.g. ESP-WROOM-32 should be DIO). Then perform a `make clean`, rebuild,
redeploy.
* Continuous reboots after programming: Ensure `CONFIG_ESPTOOLPY_FLASHMODE` is
correct for your board (e.g. ESP-WROOM-32 should be DIO). Then perform a
`make clean`, rebuild, redeploy.

8 comments on commit e017f27

@robert-hh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some hiccups I could build the firmware, at least for GENERIC_SPIRAM. The GENERIC version does not build with some memory overflow errors. But also the SPIRAM version it too larger for the default app partition at 180000 hex size. Does anything object to increase it by default to 1f0000 hex, which is the space up the file system?

@dpgeorge
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idf.py is part of the ESP-IDF. If you source $IDF_PATH/export.sh then idf.py should be in your path, and then make should just work as it did before (I hope this is explained well enough in the new README...).

The GENERIC version does not build with some memory overflow errors.

It builds OK for me, and for CI. You need to use exactly the same toolchain which is installed by $IDF_PATH/install.sh.

But also the SPIRAM version it too larger for the default app partition at 180000 hex size

Yes indeed it is! I didn't notice that and there is no warning/error.... strange. I just checked the original build using make and IDF 4.0.1 and the images fit within 0x180000. So they must have increased in size due to IDF 4.0.2 and/or the new toolchain (in particular some SPIRAM issues may have been worked around by the new gcc and this may generate larger code). It's about a 50k increase though, which is a lot.

We will need to increase the partitions.csv so the images fit...

@robert-hh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GENERIC version builds now after removing BTREE. The fail may have been caused by including some own frozen modules. And I have increase the app partition to 1f0000 hex. But that's another story.
Another strange change is, that frozen scripts from the board specific manifest.py file are not included.

@dpgeorge
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building in release mode and disabling assertions cuts of about 200k of firmware size, so that's another option.

Another strange change is, that frozen scripts from the board specific manifest.py file are not included.

I just tried building with make BOARD=TINYPICO and it does pick up the manifest from this board.

@robert-hh
Copy link
Contributor

@robert-hh robert-hh commented on e017f27 Feb 15, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried building with make BOARD=TINYPICO and it does pick up the manifest from this board.

No luck. If I mode that section to board/manifest.py, it works. I faintly recall that I had that issue before, but I cannot remember how it was solved. No problem so far, since there is a way around.

@robert-hh
Copy link
Contributor

@robert-hh robert-hh commented on e017f27 Feb 15, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried building with make BOARD=TINYPICO and it does pick up the manifest from this board.

TINYPICO's mpconfigboard.cmake is different to the other board ones. In tinypico, the manifes line reads:
set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)
For the other boards it's
set(MICROPY_FROZEN_MANIFEST ${MICROPY_PORT_DIR}/boards/manifest.py)

@tve
Copy link
Contributor

@tve tve commented on e017f27 Feb 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I have increase the app partition to 1f0000 hex.

There's also a partitions_ota.csv to consider... I would move up more incrementally...

@robert-hh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Damien mentioned: using the Release build setting the image is smaller.

Please sign in to comment.