Serial Protocol

Angus Gratton edited this page Oct 25, 2018 · 27 revisions

Serial Protocol

This is technical documentation for the serial protocol used by the UART bootloader in the ESP8266 & ESP32 ROMs and the esptool.py software "stub loader" program.

The UART bootloader runs on chip reset if certain strapping pins are set. See Entering The Bootloader for details of this process.

The ESP8266 & ESP32 ROM loader serial protocols are similar, although ESP32 adds some additional commands and some slightly different behaviour.

By default, esptool.py uploads a stub "software loader" to the IRAM of the chip. The software loader then replaces the ROM loader for all future interactions. This standardizes much of the behaviour. Pass --no-stub to esptool.py in order to disable the software stub loader.

Packet Description

Overview

The host computer sends a SLIP encoded command request to the ESP chip. The ESP chip responds to the request with a SLIP encoded response packet, including status information and any data as a payload.

Low Level Protocol

The bootloader protocol uses SLIP packet framing for data transmissions in both directions.

Each SLIP packet begins and ends with 0xC0. Within the packet, all occurrences of 0xC0 and 0xDB are replaced with 0xDB 0xDC and 0xDB 0xDD, respectively.

Command

Each command is a SLIP packet initiated by the host and results in a response packet. Inside the packet, the packet consists of a header and a variable-length body. All multi-byte fields are little-endian.

Byte Name Comment
0 Direction Always 0x00 for requests
1 Command Command identifier (see Command Opcodes).
2-3 Size Length of Data field, in bytes.
4-7 Checksum Simple checksum of part of the data field (only used for some commands, see Checksum).
8..n Data Variable length data payload (0-65535 bytes, as indicated by Size parameter). Usage depends on specific command.

Response

Each received command will result in a response SLIP packet sent from the ESP chip to the host. Contents of the response packet is:

Byte Name Comment
0 Direction Always 0x01 for responses
1 Command Same value as Command identifier in the request packet that trigged the response
2-3 Size Size of data field. At least the length of the status bytes (2 or 4 bytes, see below).
4-7 Value Response value used by READ_REG command (see below). Zero otherwise.
8..n Data Variable length data payload. Length indicated by "Size" field.

Status bytes

The final bytes of the Data payload indicate command status:

For software loaders and ESP8266 ROM loader the final two bytes indicate status (most commands return at least a two byte Data payload):

Byte Name Comment
Size-2 Status Status flag, success (0) or failure (1)
Size-1 Error If Status is 1, this indicates the type of error.

For ESP32 ROM (only, not the software loader) the final four bytes are used, but only the first two bytes contain status information:

Byte Name Comment
Size-4 Status Status flag, success (0) or failure (1)
Size-3 Error If Status 1, this indicates the type of error.
Size-2 Reserved
Size-1 Reserved

ROM Loader Error Codes

The ROM loader sends the following error values

  • 0x05 - "Received message is invalid" (parameters or length field is invalid)
  • 0x06 - "Failed to act on received message"
  • 0x07 - "Invalid CRC in message"
  • 0x0b - "Deflate error" (ESP32 compressed uploads only)

Software Loader Status & Error

If the software loader is used:

  • The status response is always 2 bytes regardless of chip type.
  • Stub loader error codes are entirely different to the ROM loader codes. They all take the form 0xC*, or 0xFF for "unimplemented command". (Full list here).

After sending a command, the host should continue to read response packets until one is received where the Command field matches the request's Command field, or a timeout is exceeded.

Command Opcodes

Supported by software loader and ROM loaders

Byte Name Description Input Data Output Data
0x02 FLASH_BEGIN Begin Flash Download Four 32-bit words: size to erase, number of data packets, data size in one packet, flash offset.
0x03 FLASH_DATA Flash Download Data Four 32-bit words: data size, sequence number, 0, 0, then data. Uses Checksum.
0x04 FLASH_END Finish Flash Download One 32-bit word: 0 to reboot, 1 to stay in loader.
0x05 MEM_BEGIN Begin RAM Download Start total size, number of data packets, data size in one packet, memory offset
0x06 MEM_END Finish RAM Download Two 32-bit words: execute flag, entry point address
0x07 MEM_DATA RAM Download Data Four 32-bit words: data size, sequence number, 0, 0, then data. Uses Checksum.
0x08 SYNC Sync Frame 36 bytes: 0x07 0x07 0x12 0x20, followed by 32 x 0x55
0x09 WRITE_REG Write 32-bit memory address Four 32-bit words: address, value, mask and delay (in microseconds)
0x0a READ_REG Read 32-bit memory address Address as 32-bit word Read data as 32-bit word in value field.

Supported by software loader and ESP32 ROM Loader

Byte Name Description Input Data Output Data
0x0b SPI_SET_PARAMS Configure SPI flash Six 32-bit words: id, total size in bytes, block size, sector size, page size, status mask.
0x0d SPI_ATTACH Attach SPI flash 32-bit word: Zero for normal SPI flash. A second 32-bit word (should be 0) is passed to ESP32 ROM loader only.
0x0f CHANGE_BAUDRATE Change Baud rate Two 32-bit words: new baud rate, 0 if we are talking to the ROM flasher or the current/old baud rate if we are talking to the software stub flasher.
0x10 FLASH_DEFL_BEGIN Begin compressed flash download Four 32-bit words: uncompressed size, number of data packets, data packet size, flash offset.
With stub loader the uncompressed size is exact byte count to be written, whereas on ROM bootloader it is rounded up to flash erase block size.
0x11 FLASH_DEFL_DATA Compressed flash download data Four 32-bit words: data size, sequence number, 0, 0, then data. Uses Checksum. Error code 0xC1 on checksum error.
0x12 FLASH_DEFL_END End compressed flash download One 32-bit word: 0 to reboot, 1 to stay in loader.
0x13 SPI_FLASH_MD5 Calculate MD5 of flash region Four 32-bit words: address, size, 0, 0 Body contains 16 raw bytes (stub loader) or 32 hex-coded ASCII (ROM loader) of calculated MD5

Supported by software loader only (ESP8266 & ESP32)

ROM loaders will not recognise these commands.

Byte Name Descripton Input Output
0xd0 ERASE_FLASH Erase entire flash chip
0xd1 ERASE_REGION Erase flash region Two 32-bit words: flash offset to erase, erase size in bytes. Both must be multiples of flash sector size.
0xd2 READ_FLASH Read flash Four 32-bit words: flash offset, read length, flash sector size, read packet size, maximum number of un-acked packets
0xd3 RUN_USER_CODE Exits loader and runs user code

Checksum

The checksum field is ignored (can be zero) for all comands except for MEM_DATA, FLASH_DATA, and FLASH_DEFL_DATA.

Each of the _DATA command packets has the same "data payload" format:

Bytes Name Format
0-3 "Data to write" length Little endian 32-bit word.
4-7 Sequence number Little endian 32-bit word. The sequence numbers are 0 based.
8-15 0 Two words of all zeroes, unused.
16- "Data to write" Length given at beginning of payload.

The checksum is only applied to this final "data to write" section, not the first 16 bytes of data.

To calculate checksum, start with seed value 0xEF and XOR each individual byte in the "data to write". The 8-bit result is stored in the checksum field of the packet header (as a little endian 32-bit value).

Note: Because this checksum is not adequate to ensure valid data, the SPI_FLASH_MD5 command was added to validate flash contents after flashing. It is recommended that this command is always used. See Verifying Uploaded Data, below.

Functional Description

Initial Synchronisation

  • The ESP chip is reset into UART bootloader mode. The host starts by sending SYNC commands. These commands have a large data payload which is also used by the ESP chip to detect the configured baud rate. The ESP8266 will initialise at 74800bps with a 26MHz crystal and 115200bps with a 40MHz crystal, and ESP32 always initialises at 115200bps. However the sync packets can be sent at any baud rate, and the UART peripheral will detect this.

  • The host should wait until it sees a valid response to a SYNC command, indicating the ESP chip is correctly communicating.

  • esptool.py then (by default) uses the "RAM Download" sequence to upload software stub loader code to IRAM of the chip. The MEM_END command contains the entry-point address to run the software loader. The software loader then sends a custom SLIP packet of the sequence OHAI (0xC0 0x4F 0x48 0x41 0x49 0xC0), indicating that it is now running. This is the only unsolicited packet ever sent by the ESP. If the --no-stub argument is supplied to esptool.py, this entire step is skipped.

  • esptool.py then uses READ_REG commands to read various addresses on the chip, to identify chip subtype, revision, etc.

  • For commands which need to use the flash, the ESP32 ROM loader requires (and software loader on both chips support) the SPI_ATTACH and SPI_SET_PARAMS commands. See SPI Configuration Commands.

  • For software loader and/or ESP32 ROM loader, the host can send a CHANGE_BAUD command to set the baud rate to an explicit value. Compared to auto-detecting during the SYNC pulse, this can be more reliable for setting very high baud rate. esptool.py tries to sync at (maximum) 115200bps and then sends this command to go to a higher baud rate, if requested.

Writing Data

(Includes RAM Download, Flash Download, Compressed Flash Download.)

  • RAM Download (MEM_BEGIN, MEM_DATA, MEM_END) loads data into the ESP chip memory space and (optionally) executes it.
  • Flash Download (FLASH_BEGIN, FLASH_DATA, FLASH_END) flashes data into the ESP SPI flash.
  • Compressed Flash Download is the same, only the data is compressed using the gzip Deflate algorithm to reduce serial overhead. Not supported on ESP8266 ROM loader.

All three of these sequences follow a similar pattern:

  • A _BEGIN command (FLASH_BEGIN, etc) is sent which contains basic parameters for the flash erase size, start address to write to, etc. The uploader also needs to specify how many "blocks" of data (ie individual data packets) will be sent, and how big each packet is.
  • One or more _DATA commands (FLASH_DATA, etc) is sent where the data payload contains the actual data to write to flash/RAM. In the case of Compressed Flash Downloads, the data is compressed using the gzip deflate algorithm. The number of _DATA commands is specified in the _BEGIN command, as is the size of each _DATA payload. The last data block should be padded to the block size with 0xFF bytes.
  • An _END command (FLASH_END, etc) is sent to finish the download and optionally reset the chip (or jump to an address in RAM, in the case of MEM_END).

It's not necessary to send flash erase commands before sending commands to write to flash, etc. The ROM loaders erase the to-be-written region in response to the FLASH*_BEGIN command. The software loaders do just-in-time erasing as they write data, to maximise overall flashing performance (each block of data is read into RAM via serial while the previous block is simultaneously being written to flash, and 4KB and 64KB erases are done as needed before writing to flash).

The block size chosen should be small enough to fit into RAM of the device. esptool.py uses 16KB which gives good performance when used with the stub loader.

Erase Size Bug

On ESP8266 ROM loader only (not software loader), there is a bug in the interpretation of the FLASH_BEGIN "erase size" parameter. Consult the ESP8266ROM.get_erase_size() function in esptool.py for the algorithm which works around this bug and provides the correct erase size parameter to send to the ESP8266.

This workaround is not needed on ESP32, or if the ESP8266 is running the software loader.

Verifying Uploaded Data

The 8-bit checksum used in the upload protocol is not sufficient to ensure valid flash contents after upload. The uploader should send the SPI_FLASH_MD5 command (not supported on ESP8266 ROM loader) or use another method to verify flash contents.

The SPI_FLASH_MD5 command passes the start address in flash and the size of data to calculate. The MD5 value is returned in the response payload, before the status bytes.

Note that the ESP32 ROM loader returns the md5sum as 32 hex encoded ASCII bytes, whereas the software loader returns the md5sum as 16 raw data bytes.

SPI Configuration Commands

SPI Attach command

The SPI_ATTACH command enables the SPI flash interface. It takes a 32-bit data payload which is used to determine which SPI peripheral and pins should be used to connect to SPI flash.

On the ESP8266 software loader sending this command before interacting with SPI flash is optional, on ESP32 it is required. On ESP8266 ROM loader this command is not supported (SPI flash is enabled when the FLASH_BEGIN command is sent).

Value Meaning
0 Default SPI flash interface
1 HSPI interface
(other values) (ESP32 only) Pin numbers as 6-bit values, packed into a 30-bit value. Order (from MSB): HD pin, Q pin, D pin, CS pin, CLK pin.

ESP32 Only Details

(The following only apply to ESP32.)

The "Default SPI flash interface" uses pins configured via the SPI_PAD_CONFIG_xxx efuses (if unset, these efuses are all zero and the default SPI flash pins given in the datasheet are used.)

When writing the values of each pin as 6-bit numbers packed into the data word, each 6-bit value uses the following representation:

  • Pin numbers 0 through 30 are represented as themselves.
  • Pin numbers 32 & 33 are represented as values 30 & 31.
  • It is not possible to represent pins 30 & 31 or pins higher than 33. This is the same 6-bit representation used by the SPI_PAD_CONFIG_xxx efuses.

On ESP32 ROM loader only, there is an additional 4 bytes in the data payload of this command. These bytes should all be set to zero.

SPI Set Parameters

The SPI_SET_PARAMS command sets some parameters of the attached SPI flash chip (sizes, etc). This command is not supported by the ESP8266 ROM loader.

All the values which are passed except total size are hardcoded, and most are not used when writing to flash. See flash_set_parameters function in esptool.py for the values which it sends.

32-bit Read/Write

The 32-bit read/write commands (READ_REG, WRITE_REG) allow word-oriented reading and writing of memory and register data.

These commands can be used to manipulate peripherals in arbitrary ways. For example, the esptool.py "flash id" functionality is implemented by manipulating the SPI peripheral registers to send a JEDEC flash ID command to the flash chip and read the response.

Reading Flash

The software loader implements a READ_FLASH command. This command behaves differently to other commands:

  • The host sends the READ_FLASH command and the data payload contains the offset, read size, size of each individual packet of data, and the maximum number of "un-acknowledged" data packets which can be in flight at one time.
  • The software loader will send a standard response packet, with no additional data payload.
  • Now the software loader will start sending SLIP packets with raw data (of the size requested in the command). There is no metadata included with these SLIP packets.
  • After each SLIP packet is received, the host should send back a 4 byte raw SLIP acknowledgement packet with the count of packets which have been received. There is no header or other metadata included with these SLIP packets.
  • The software loader may send up to a maximum number (specified by the host in the READ_FLASH commands) of data packets before waiting for the first acknowledgement packet. No more than this "max in flight" limit can be un-acknowledged at any one time.
  • After all data packets are acknowledged received, the software loader sends a 16 byte MD5 digest of all the data which was read from flash. This is also sent as a raw SLIP packet, with no metadata.

After the read flash process is complete, the software loader goes back to normal command/response operation.

Tracing esptool.py serial communications

esptool.py v2.2 and newer have a --trace option which can be supplied in the first group of arguments (before the command). This will dump all traffic sent and received via the serial port to the console.

Here is a sample extract, showing a READ_REG command and response:

TRACE +0.000 command op=0x0a data len=4 wait_response=1 timeout=3.000 data=b'\x0c\xa0\x01`'
TRACE +0.000 Write 14 bytes: b'\xc0\x00\n\x04\x00\x00\x00\x00\x00\x0c\xa0\x01`\xc0'
TRACE +0.015 Read 1 bytes: b'\xc0'
TRACE +0.000 Read 13 bytes: b'\x01\n\x04\x00\x00\x80\x00\x00\x00\x00\x00\x00\xc0'
TRACE +0.000 Full packet: b'\x01\n\x04\x00\x00\x80\x00\x00\x00\x00\x00\x00'

The +X.XXX value is the time delta (in seconds) since the last trace line.

Values are printed as Python (byte) strings. Non-ASCII hex bytes are encoded as \x00. Printable characters are printed as their ASCII equivalents. You can use Python to convert the entire string to its hexadecimal representation by copy/pasting, ie:

$ python
Python 3.6.4 (default, Jan  5 2018, 02:35:40)
[GCC 7.2.1 20171224] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>> binascii.hexlify(b'\xc0\x00\n\x04\x00\x00\x00\x00\x00\x0c\xa0\x01`\xc0')
b'c0000a0400000000000ca00160c0'
>>>

(Note the backtick character near the end of the Python string above is printed as hexadecimal 60.)

IMPORTANT: If you don't plan to use the esptool.py software loader, pass --no-stub --trace to see interactions with the chip's built-in ROM loader only.

In addition to this trace feature, most operating systems have "system call trace" or "port trace" features which can be used to dump serial interactions.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.