WIP: ESP32 support, version 2.0 refactoring

[projectgus]

Work in progress to support ESP32 and a large amount of refactoring. Includes breaking changes compared to esptool.py v1.x (which will still be supported.)

New

• Supports ESP32.
• Compressed uploads on both ESP8266 & ESP32. Very fast!
• Automated tests, both off-device elf2image tests and on-device tests using esptool.py
• Software bootloader stub now speaks a variant of the ROM bootloader protocol. This means that the same esptool.py code can be used to talk to both ROM bootloaders and stubs. The --no-stub argument can be used to disable stub loading. Not all functionality is available without the stub (as the ROM bootloaders don't support all of esptool.py's commands.)
• elf2image no longer shells out to binutils for ELF information, ELF headers are parsed directly.
• elf2image no longer relies on special symbols like _data_start, or even names of ELF sections. Sections are identified based on their load addresses. This means linker scripts can be a lot more varied than was previously possible.
• Flash sizes are specified in megabytes (ie -fs 4MB). Old megabit style (ie -fs 16m) prints a deprecation warning.

Still TODO for version 2:

• Stub error handling needs work.
• Stub doesn't correctly reset or run the program after esptool.py runs.
• Stub code should be refactored into multiple files.
• elf2image outputs have changed due to new ELF parsing. Work for ESP32 and ESP8266 in some simple tests, but need to be more completely tested and the test cases updated.
• esptool.py --no-stub write_image -z uses zlib compression on ESP32. ESP32 & ESP8266 stubs do not yet support compressed writes.
• ESP32 flash size options are a work in progress, currently all images get default sizes.
• Class hierarchy of ESPROM/ESPxxxROM/ESPxxxStubLoader is clunky due to the patchwork set of features supported in each combination. Needs refactoring.
• Split ELF & image generation features out of esptool.py into a new espimage.py
• write_image should no longer edit the image header in flash by default. Either make a standalone command to write a new flash size, or require re-processing the binary via espimage.py.
• Test cases for flash sizes. Need clear errors if data is too big for programmed flash size, etc.
• Test cases for common error scenarios.
• README is out of date.
• Format "debug" messages in output better, better compressed ratio/performance output.
• pep8 fails.

[igrr]

#include <stddef.h>?

[projectgus]

Thank you. I had a mental blank on what this header was called!

[ghost]

Is it still possible to rely on non-vendored versions of those modules third party modules?

The vendored modules are going to make it difficult to properly package esptool for various linux distributions like Debian, Ubuntu,and Fedora.

[projectgus]

Is it still possible to rely on non-vendored versions of those modules third party modules?

Yes, at least this is intended (and I should explain this in the README, once I get around to updating it!)

The third-party modules are named in setup.py install_requires, with the idea being that if you install via pip, or setup.py install, then you can use Python "system" modules for these. If you run esptool by cloning the repo (which a lot of people do, and it's currently how we include it in esp-idf) then you use the vendored copies.

With Python 3 support coming, we may be able to some day reasonably expect that everyone has pip (hooray!) so it may be enough to have a requirements.txt and a note in the README. That would be nice.