Command to merge multiple .bin files into one file for flashing (ESPTOOL-11)

[earlt]

I am trying to create an image of the latest ESP8266_NONOS_SDK for an ESP8266-01 module using the latest esptool.py (v2.2)

Flashing with this works fine:

esptool.py --baud 115200 --port /dev/ttyUSB0 write_flash -fm dio -ff 40m -fs detect
0x00000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/boot_v1.7.bin
0x01000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/at/512+512/user1.1024.new.2.bin
0x81000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/at/512+512/user2.1024.new.2.bin
0x7E000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/blank.bin
0xFC000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/esp_init_data_default_v08.bin
0xFE000 /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/blank.bin

However, when I create an image via:

esptool.py -c esp8266 make_image
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/boot_v1.7.bin -a 0x00000
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/at/512+512/user1.1024.new.2.bin -a 0x01000
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/at/512+512/user2.1024.new.2.bin -a 0x81000
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/blank.bin -a 0x7E000
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/esp_init_data_default_v08.bin -a 0xFC000
-f /opt/arduino/esp8266/ESP8266_NONOS_SDK-master/bin/blank.bin -a 0xFE000
ESP8266-Master

and try to flash the resultant image with:

esptool.py --baud 115200 --port /dev/ttyUSB0 write_flash -fm dio -ff 40m -fs detect 0x00 ESP8266-Master0x00000.bin

the ESP8266 does not respond.

Is there something wrong with make_image?

[projectgus]

Hi,

I think there is some confusion over the different kinds of binary images.
make_image is for making a single .bin file (the .bin files themselves contain segments which are mapped or loaded to particular addresses in RAM, this RAM load address is the meaning of the -a argument).

Merging multiple binaries into a single file, which can be flashed in one go, is not currently an esptool.py feature. Although I agree it would be good to have. The Windows GUI Flasher Tool can do this.

[projectgus]

I've marked this an enhancement request for that feature.

[earlt]

Thanks for clearing up the confusion and marking it an enhancement. That will ge GREAT!

[projectgus]

No problems. Will keep this open as an enhancement request until esptool.py has this feature.

[Gueni]

Is there anything new regarding this issue ?

[AlexToulan]

This feature would be very helpful for build automation/distribution.

[sfranzyshen]

still need this feature please ...

[vtunr]

If you're interested, I rolled my own solution, https://github.com/vtunr/esp32_binary_merger, feel free to use it

[Barmaley13]

I've used @vtunr answer to append the parser. Here is the code that you can add to esptool.py

# Functions
class bin():
    def __init__(self, addr, file):
        self.addr = addr
        self.file = file
        self.file_name = os.path.basename(file.name)
        self.size = self.get_size()

    def get_size(self):
        self.file.seek(0, 2)  # seek to end
        size = self.file.tell()
        self.file.seek(0)
        return size


class multiple_bin():
    def __init__(self, name, output_folder='.'):
        self.name = name
        self.output_folder = output_folder
        try:
            os.makedirs(os.path.realpath(self.output_folder))
        except:
            pass
        self.output_path = os.path.realpath(os.path.join(self.output_folder, self.name))
        self.bin_array = []

    def add_bin(self, addr, file):
        self.bin_array.append(bin(addr, file))

    def sort_bin(self):
        swapped = True
        while swapped:
            swapped = False
            for i in range(len(self.bin_array) - 1):
                if self.bin_array[i].addr > self.bin_array[i + 1].addr:
                    self.bin_array[i], self.bin_array[i + 1] = self.bin_array[i + 1], self.bin_array[i]
                    swapped = True

    def add_bin_to_other_bin(self, previous, binary):
        with open(self.output_path, "a") as output_file:
            output_file.write('\xff' * (binary.addr - previous))
        print("Add %s from 0x%x to 0x%x (0x%x)" % (binary.file_name, binary.addr, binary.addr + binary.size, binary.size))
        with open(self.output_path, "a") as output_file:
            output_file.write(binary.file.read())
        return binary.addr + binary.size

    def create_bin(self):
        new_start = 0
        open(self.output_path, "wb").close
        for b in self.bin_array:
            new_start = self.add_bin_to_other_bin(new_start, b)

    def check_if_possible(self):
        for i in range(1, len(self.bin_array)):
            if (self.bin_array[i].addr <= (self.bin_array[i - 1].addr + self.bin_array[i - 1].size)):
                print(self.bin_array[i].addr, (self.bin_array[i - 1].addr + self.bin_array[i - 1].size))
                raise Exception("Not possible to create this bin, overlapping between %s and %s" % (
                self.bin_array[i].file_name, self.bin_array[i - 1].file_name))


def merge_image(args):
    # Check that bins are provided
    if len(args.addr_filename) == 0:
        raise FatalError('No segments specified')

    encrypt = bool(args.keyfile is not None)
    unenc_name = args.output + '.bin'
    enc_name = args.output + '.enc.bin'

    # Combine bins into unencrypted image
    mb = multiple_bin(unenc_name)
    for address, argfile in args.addr_filename:
        mb.add_bin(address, argfile)
    mb.sort_bin()
    mb.check_if_possible()
    mb.create_bin()

    # Encrypt image (if needed)
    if encrypt:
        with open(unenc_name, 'rb') as input_file:
            with open(enc_name, 'wb') as output_file:
                espsecure._flash_encryption_operation(
                    output_file, input_file, 0, args.keyfile, args.flash_crypt_conf, False)

And this needs to be inserted into main function, maybe right after parser_make_image declarations

    parser_merge_image = subparsers.add_parser(
        'merge_image',
        help='Create an application image from binary files')
    parser_merge_image.add_argument('addr_filename', metavar='<address> <filename>', help='Address followed by binary filename, separated by space',
                                    action=AddrFilenamePairAction)
    parser_merge_image.add_argument('--output', '-o', help='Output image file postfix (default: flash)', type=str, default='flash')
    parser_merge_image.add_argument('--keyfile', '-k', help="File with flash encryption key", type=argparse.FileType('rb'), default=None)
    parser_merge_image.add_argument('--flash_crypt_conf', help="Override FLASH_CRYPT_CONF efuse value (default: 0xF).",
                                   default=0xF, type=arg_auto_int)

    # Adding so we can use the same download.config for image composition
    add_spi_flash_subparsers(parser_merge_image, is_elf2image=False)

After that you can create both unencrypted and encrypted images (-k is optional)
esptool.py --port PORT merge_image -k flash_key.bin @download.config

and flash those with 0x0 offset, for instance
esptool.py --port PORT -b 460800 write_flash --flash_mode dio --flash_freq 40m --flash_size detect 0x0 flash.bin

or similar (for the encrypted flash)
esptool.py --port PORT -b 460800 write_flash --flash_mode dio --flash_freq 40m --flash_size detect 0x0 flash.enc.bin

[AlexToulan]

This looks great! Thank you, I'll give it a shot. *Alex Toulan* Software Engineer

…

On Fri, Aug 28, 2020 at 2:02 PM Barmaley13 ***@***.***> wrote: I've used @vtunr <https://github.com/vtunr> answer to append the parser. Here is the code that you can add to esptool.py # Functions class bin(): def __init__(self, addr, file): self.addr = addr self.file = file self.file_name = os.path.basename(file.name) self.size = self.get_size() def get_size(self): self.file.seek(0, 2) # seek to end size = self.file.tell() self.file.seek(0) return size class multiple_bin(): def __init__(self, name, output_folder='.'): self.name = name self.output_folder = output_folder try: os.makedirs(os.path.realpath(self.output_folder)) except: pass self.output_path = os.path.realpath(os.path.join(self.output_folder, self.name)) self.bin_array = [] def add_bin(self, addr, file): self.bin_array.append(bin(addr, file)) def sort_bin(self): swapped = True while swapped: swapped = False for i in range(len(self.bin_array) - 1): if self.bin_array[i].addr > self.bin_array[i + 1].addr: self.bin_array[i], self.bin_array[i + 1] = self.bin_array[i + 1], self.bin_array[i] swapped = True def add_bin_to_other_bin(self, previous, binary): with open(self.output_path, "a") as output_file: output_file.write('\xff' * (binary.addr - previous)) print("Add %s from 0x%x to 0x%x (0x%x)" % (binary.file_name, binary.addr, binary.addr + binary.size, binary.size)) with open(self.output_path, "a") as output_file: output_file.write(binary.file.read()) return binary.addr + binary.size def create_bin(self): new_start = 0 open(self.output_path, "wb").close for b in self.bin_array: new_start = self.add_bin_to_other_bin(new_start, b) def check_if_possible(self): for i in range(1, len(self.bin_array)): if (self.bin_array[i].addr <= (self.bin_array[i - 1].addr + self.bin_array[i - 1].size)): print(self.bin_array[i].addr, (self.bin_array[i - 1].addr + self.bin_array[i - 1].size)) raise Exception("Not possible to create this bin, overlapping between %s and %s" % ( self.bin_array[i].file_name, self.bin_array[i - 1].file_name)) def merge_image(args): # Check that bins are provided if len(args.addr_filename) == 0: raise FatalError('No segments specified') encrypt = bool(args.keyfile is not None) unenc_name = args.output + '.bin' enc_name = args.output + '.enc.bin' # Combine bins into unencrypted image mb = multiple_bin(unenc_name) for address, argfile in args.addr_filename: mb.add_bin(address, argfile) mb.sort_bin() mb.check_if_possible() mb.create_bin() # Encrypt image (if needed) if encrypt: with open(unenc_name, 'rb') as input_file: with open(enc_name, 'wb') as output_file: espsecure._flash_encryption_operation( output_file, input_file, 0, args.keyfile, args.flash_crypt_conf, False) And this needs to be inserted into main function, maybe right after parser_make_image declarations parser_merge_image = subparsers.add_parser( 'merge_image', help='Create an application image from binary files') parser_merge_image.add_argument('addr_filename', metavar='<address> <filename>', help='Address followed by binary filename, separated by space', action=AddrFilenamePairAction) parser_merge_image.add_argument('--output', '-o', help='Output image file postfix (default: flash)', type=str, default='flash') parser_merge_image.add_argument('--keyfile', '-k', help="File with flash encryption key", type=argparse.FileType('rb'), default=None) parser_merge_image.add_argument('--flash_crypt_conf', help="Override FLASH_CRYPT_CONF efuse value (default: 0xF).", default=0xF, type=arg_auto_int) # Adding so we can use the same download.config for image composition add_spi_flash_subparsers(parser_merge_image, is_elf2image=False) After that you can create both unencrypted and encrypted images (-k is optional) esptool.py --port PORT merge_image -k flash_key.bin @download.config and flash those with 0x0 offset, for instance esptool.py --port PORT -b 460800 write_flash --flash_mode dio --flash_freq 40m --flash_size detect 0x0 flash.bin or similar (for the encrypted flash) esptool.py --port PORT -b 460800 write_flash --flash_mode dio --flash_freq 40m --flash_size detect 0x0 flash.enc.bin — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#254 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASVK4WDSS6Z7EMYVIVQVXLSC7WMZANCNFSM4EJTBV5Q> .

[fabianoriccardi]

Hi, will this feature be implemented in the next versions?

Also the term "make_image" may be misleading for "newcomers". If I had well understand, actually it is like "merging binary segments in a single binary image", right?

[Barmaley13]

I've added pull request. I guess it is up to maintainers to add it or deny it. I know that make_image and merge_image are confusing. But they serve different purpose. Adding merge_image would make esptool backwards compatible..

[fabianoriccardi]

Thanks for your pull request.

But they serve different purpose. Adding merge_image would make esptool backwards compatible.

Yes, but since the v3.0.0 is under development, a deprecation warning I admissible, so that 2 clearly distinct commands can be added. At least if maintainers wish it.

What is the meaning of '@' symbol in the following command? Output file?

esptool.py --port PORT merge_image -k flash_key.bin @download.config

[Barmaley13]

@ is used to append arguments from a text file