esp8266: Support up to 16 MB of flash #2335

Open
Ratfink opened this Issue Aug 21, 2016 · 11 comments

Projects

None yet

6 participants

@Ratfink
Ratfink commented Aug 21, 2016

The ESP8266 supports up to 16 megabytes of SPI flash (source, page 12). NodeMCU has supported this much flash for about two months now, and I notice that MicroPython does not. I don't know all the nuances of the ESP8266, but from what I understand there are three things that should all encode the same size: the flash chip's ID, the SpiFlashChip structure from the SDK, and the bootloader size ID. They might not all agree about the flash chip's size, so we try to make them all agree in flashbdev.py. It looks like in the similar routine from NodeMCU, they use size IDs 8 and 9 for 8 MB and 16 MB, respectively. Setting these IDs in MicroPython seems to make mpy think that it has only 512 kB of flash, and I'm not sure why. Ignoring the bootloader size ID and only calculating flash size from the flash chip's ID seems to work, but somehow that seems like the wrong thing to do.

Long story short, I want to add support for 8 and 16 MB of flash to MicroPython, but I don't know what the right way to do it is. Any advice would be greatly appreciated.

@dpgeorge
Contributor
@Ratfink
Ratfink commented Aug 22, 2016

I tried making the suggested change:

diff --git a/esp8266/modules/flashbdev.py b/esp8266/modules/flashbdev.py
index 98b0208..284fa1a 100644
--- a/esp8266/modules/flashbdev.py
+++ b/esp8266/modules/flashbdev.py
@@ -37,6 +37,10 @@ def set_bl_flash_size(real_size):
         code = 3
     elif real_size == 4096*1024:
         code = 4
+    elif real_size == 8192*1024:
+        code = 5
+    elif real_size == 16384*1024:
+        code = 6
     else:
         code = 2
     buf = bytearray(4096)
@@ -50,7 +54,7 @@ def set_bl_flash_size(real_size):
 size = esp.flash_id() >> 16
 # Check that it looks like realistic power of 2 for flash sizes
 # commonly used with esp8266
-if 22 >= size >= 18:
+if 24 >= size >= 18:
     size = 1 << size
     if size != esp.flash_size():
         import machine

It results in a boot loop, printing this repeatedly:

l��r��  yr�'b����x�p�lp�������p�lll��r��        yr�'b��l����x��p�������p��cl`䈌������rn������x���b���b����ߌ��b��bl`�>r�8�l`#��r�����8����r�����8�����lrr��9l�Ğ����br�b�������b�#�n��1�n��`ll������l��l�����;�n��Β��b���b�>��ܒ��b�|lrrr�n�����|~����       6��������b����xbl`��l�#4 ets_task(40100390, 3, 3fff6300, 4)
Bootloader Flash size appear to have been set incorrectly, trying to fix
bcn 0
del if1
usl

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30860, room 16 
tail 12
chksum 0x17
ho 0 tail 12 room 4
load 0x3ffe8000, len 1060, room 12 
tail 8
chksum 0xd7
load 0x3ffe8430, len 3000, room 0 
tail 8
chksum 0xb1
csum 0xb1

If I set it to use code 8 for 8 MB and 9 for 16 MB, like NodeMCU does, I get a similar boot loop, but with more stuff being printed each time it boots:

�����������r������r�l�������������8������
���l����l���l����l��������l�����������܌���rl��r��       yr�'b�����x�p�lp�������p�lll��r��       yr�'b��l
����8��p�������p��cl`��������rn������8���b���b�����ߌ��b��bl`�>r�x��l`#��r��#3 ets_task(40100390, 3, 3fff
6300, 4)
Fatal exception 9(LoadStoreAlignmentCause):
epc1=0x40250a35, epc2=0x00000000, epc3=0x00000000, excvaddr=0x0000001b, depc=0x00000000

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30860, room 16
tail 12
chksum 0x17
ho 0 tail 12 room 4
load 0x3ffe8000, len 1060, room 12
tail 8
chksum 0xd7
load 0x3ffe8430, len 3000, room 0
tail 8
chksum 0xb1
csum 0xb1
system param error
47 00 00 00 08 00 00 00 0c d0 27 40 46 06 00 00 5e 0e 00 00 76 12 00 00 94 d0 27 40 fe 0f 00 00
88 d0 27 40 6e 0f 00 00 7c d0 27 40 7e 0f 00 00 70 d0 27 40 be 13 00 00 64 d0 27 40 ce 13 00 00
58 d0 27 40 96 08 00 00 4c d0 27 40 04 26 27 40 02 00 01 00 68 1a 27 40 04 26 27 40 04 00 02 00 
d4 1a 27 40 04 26 27 40 04 00 02 00 18 1b 27 40 04 26 27 40 02 00 01 00 4c 1b 27 40 04 26 27 40 
02 00 01 00 90 1b 27 40 04 26 27 40 02 00 01 00 b4 1b 27 40 04 26 27 40 02 00 01 00 d8 1b 27 40 
23 25 64 20 65 74 73 5f 74 61 73 6b 28 25 70 2c 20 25 64 2c 20 25 70 2c 20 25 64 29 0a 00 74 61 
73 6b 20 70 72 69 6f 20 6f 75 74 20 6f 66 20 72 61 6e 67 65 3a 20 25 64 0a 00 65 74 73 5f 70 6f 
73 74 3a 20 74 61 73 6b 20 25 64 20 71 75 65 75 65 20 66 75 6c 6c 0a 00 65 74 73 5f 61 6c 74 5f 
74 61 73 6b 3a 20 65 74 73 5f 72 75 6e 00 23 20 41 50 41 31 30 32 20 64 72 69 76 65 72 20 66 6f 
72 20 4d 69 63 72 6f 50 79 74 68 6f 6e 20 6f 6e 20 45 53 50 38 32 36 36 0a 23 20 4d 49 54 20 6c 
69 63 65 6e 73 65 3b 20 43 6f 70 79 72 69 67 68 74 20 28 63 29 20 32 30 31 36 20 52 6f 62 65 72 
74 20 46 6f 73 73 2c 20 44 61 6e 69 65 6c 20 42 75 73 63 68 0a 0a 66 72 6f 6d 20 65 73 70 20 69 
6d 70 6f 72 74 20 61 70 61 31 30 32 5f 77 72 69 74 65 0a 0a 63 6c 61 73 73 20 41 50 41 31 30 32 
3a 0a 20 20 20 20 64 65 66 20 5f 5f 69 6e 69 74 5f 5f 28 73 65 6c 66 2c 20 63 6c 6f 63 6b 5f 70 
69 6e 2c 20 64 61 74 61 5f 70 69 6e 2c 20 6e 29 3a 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 63 
6c 6f 63 6b 5f 70 69 6e 20 3d 20 63 6c 6f 63 6b 5f 70 69 6e 0a 20 20 20 20 20 20 20 20 73 65 6c 
66 2e 64 61 74 61 5f 70 69 6e 20 3d 20 64 61 74 61 5f 70 69 6e 0a 20 20 20 20 20 20 20 20 73 65 
6c 66 2e 6e 20 3d 20 6e 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 62 75 66 20 3d 20 62 79 74 65 
61 72 72 61 79 28 6e 20 2a 20 34 29 0a 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 63 6c 6f 63 6b 
5f 70 69 6e 2e 69 6e 69 74 28 63 6c 6f 63 6b 5f 70 69 6e 2e 4f 55 54 29 0a 20 20 20 20 20 20 20 
20 73 65 6c 66 2e 64 61 74 61 5f 70 69 6e 2e 69 6e 69 74 28 64 61 74 61 5f 70 69 6e 2e 4f 55 54 
29 0a 0a 20 20 20 20 64 65 66 20 5f 5f 73 65 74 69 74 65 6d 5f 5f 28 73 65 6c 66 2c 20 69 6e 64 
65 78 2c 20 76 61 6c 29 3a 0a 20 20 20 20 20 20 20 20 72 2c 20 67 2c 20 62 2c 20 62 72 69 67 68 
74 6e 65 73 73 20 3d 20 76 61 6c 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 62 75 66 5b 69 6e 64 
65 78 20 2a 20 34 5d 20 3d 20 72 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 62 75 66 5b 69 6e 64 
65 78 20 2a 20 34 20 2b 20 31 5d 20 3d 20 67 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 62 75 66 
5b 69 6e 64 65 78 20 2a 20 34 20 2b 20 32 5d 20 3d 20 62 0a 20 20 20 20 20 20 20 20 73 65 6c 66 
2e 62 75 66 5b 69 6e 64 65 78 20 2a 20 34 20 2b 20 33 5d 20 3d 20 62 72 69 67 68 74 6e 65 73 73 
0a 0a 20 20 20 20 64 65 66 20 5f 5f 67 65 74 69 74 65 6d 5f 5f 28 73 65 6c 66 2c 20 69 6e 64 65 
78 29 3a 0a 20 20 20 20 20 20 20 20 69 20 3d 20 69 6e 64 65 78 20 2a 20 34 0a 20 20 20 20 20 20 
20 20 72 65 74 75 72 6e 20 73 65 6c 66 2e 62 75 66 5b 69 5d 2c 20 73 65 6c 66 2e 62 75 66 5b 69 
20 2b 20 31 5d 2c 20 73 65 6c 66 2e 62 75 66 5b 69 20 2b 20 32 5d 2c 20 73 65 6c 66 2e 62 75 66 
5b 69 20 2b 20 33 5d 0a 0a 20 20 20 20 64 65 66 20 77 72 69 74 65 28 73 65 6c 66 29 3a 0a 20 20 
20 20 20 20 20 20 61 70 61 31 30 32 5f 77 72 69 74 65 28 73 65 6c 66 2e 63 6c 6f 63 6b 5f 70 69 
6e 2c 20 73 65 6c 66 2e 64 61 74 61 5f 70 69 6e 2c 20 73 65 6c 66 2e 62 75 66 29 0a 00 23 20 4e 
65 6f 50 69 78 65 6c 20 64 72 69 76 65 72 20 66 6f 72 20 4d 69 63 72 6f 50 79 74 68 6f 6e 20 6f 
6e 20 45 53 
70 69 6e 29 0a 0a 20 20 20 20 64 65 66 20 72 65 61 64 62 79 74 65 28 73 65 6c 66 29 3a 0a 20 20 
20 20 20 20 20 20 72 65 74 75 72 6e 20 5f 6f 77 2e 72 65 61 64 62 79 74 65 28 73 65 6c 66 2e 70 
69 6e 29 0a 0a 20 20 20 20 64 65 66 20 72 65 61 64 28 73 65 6c 66 2c 20 63 6f 75 6e 74 29 3a 0a 
20 20 20 20 20 20 20 20 62 75 66 20 3d 20 62 79 74 65 61 72 72 61 79 28 63 6f 75 6e 74 29 0a 20 
20 20 20 20 20 20 20 66 6f 72 20 69 20 69 6e 20 72 61 6e 67 65 28 63 6f 75 6e 74 29 3a 0a 20 20 
20 20 20 20 20 20 20 20 20 20 62 75 66 5b 69 5d 20 3d 20 5f 6f 77 2e 72 65 61 64 62 79 74 65 28 
73 65 6c 66 2e 70 69 6e 29 0a 20 20 20 20 20 20 20 20 72 65 74 75 72 6e 20 62 75 66 0a 0a 20 20 
20 20 64 65 66 20 77 72 69 74 65 62 69 74 28 73 65 6c 66 2c 20 76 61 6c 75 65 29 3a 0a 20 20 20 
20 20 20 20 20 72 65 74 75 72 6e 20 5f 6f 77 2e 77 72 69 74 65 62 69 74 28 73 65 6c 66 2e 70 69 
6e 2c 20 76 61 6c 75 65 29 0a 0a 20 20 20 20 64 65 66 20 77 72 69 74 65 62 79 74 65 28 73 65 6c 
66 2c 20 76 61 6c 75 65 29 3a 0a 20 20 20 20 20 20 20 20 72 65 74 75 72 6e 20 5f 6f 77 2e 77 72 
69 74 65 62 79 74 65 28 73 65 6c 66 2e 70 69 6e 2c 20 76 61 6c 75 65 29 0a 0a 20 20 20 20 64 65 
66 20 77 72 69 74 65 28 73 65 6c 66 2c 20 62 75 66 29 3a 0a 20 20 20 20 20 20 20 20 66 6f 72 20 
62 20 69 6e 20 62 75 66 3a 0a 20 20 20 20 20 20 20 20 20 20 20 20 5f 6f 77 2e 77 72 69 74 65 62 
79 74 65 28 73 65 6c 66 2e 70 69 6e 2c 20 62 29 0a 0a 20 20 20 20 64 65 66 20 73 65 6c 65 63 74 
5f 72 6f 6d 28 73 65 6c 66 2c 20 72 6f 6d 29 3a 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 72 65 
73 65 74 28 29 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 77 72 69 74 65 62 79 74 65 28 4d 41 54 
43 48 5f 52 4f 4d 29 0a 20 20 20 20 20 20 20 20 73 65 6c 66 2e 77 72 69 74 65 28 72 6f 6d 29 0a 
0a 20 20 20 20 64 65 66 20 73 63 61 6e 28 73 65 6c 66 29 3a 0a 20 20 20 20 20 20 20 20 64 65 76 
69 63 65 73 20 3d 20 5b 5d 0a 20 20 20 20 20 20 20 20 64 69 66 66 20 3d 20 36 35 0a 20 20 20 20 
20 20 20 20 72 6f 6d 20 3d 20 46 61 6c 73 65 0a 20 20 20 20 20 20 20 20 66 6f 72 20 69 20 69 6e 
20 72 61 6e 67 65 28 30 78 66 66 29 3a 0a 20 20 20 20 20 20 20 20 20 20 20 20 72 6f 6d 2c 20 64 
69 66 66 20 3d 20 73 65 6c 66 2e 5f 73 65 61 72 63 68 5f 72 6f 6d 28 72 6f 6d 2c 20 64 69 66 66 
29 0a 20 20 20 20 20 20 20 20 20 20 20 20 69 66 20 72 6f 6d 3a 0a 20 20 20 20 20 20 20 20 20 20 
20 20 20 20 20 20 64 65 76 69 63 65 73 20 2b 3d 20 5b 72 6f 6d 5d 0a 20 20 20 20 20 20 20 20 20 
20 20 20 69 66 20 64 69 66 66 20 3d 3d 20 30 3a 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
20 62 72 65 61 6b 0a 20 20 20 20 20 20 20 20 72 65 74 75 72 6e 20 64 65 76 69 63 65 73 0a 0a 20 
20 20 20 64 65 66 20 5f 73 65 61 72 63 68 5f 72 6f 6d 28 73 65 6c 66 2c 20 6c 5f 72 6f 6d 2c 20 
64 69 66 66 29 3a 0a 20 20 20 20 20 20 20 20 69 66 20 6e 6f 74 20 73 65 6c 66 2e 72 65 73 65 74 
28 29 3a 0a 20 20 20 20 20 20 20 20 20 20 20 20 72 65 74 75 72 6e 20 4e 6f 6e 65 2c 20 30 0a 20 
20 20 20 20 20 20 20 73 65 6c 66 2e 77 72 69 74 65 62 79 74 65 28 53 45 41 52 43 48 5f 52 4f 4d
29 0a 20 20 20 20 20 20 20 20 69 66 20 6e 6f 74 20 6c 5f 72 6f 6d 3a 0a 20 20 20 20 20 20 20 20
20 20 20 20 6c 5f 72 6f 6d 20 3d 20 62 79 74 65 61 72 72 61 79 28 38 29 0a 20 20 20 20 20 20 20
20 72 6f 6d 20 3d 20 62 79 74 65 61 72 72 61 79 28 38 29 0a 20 20 20 20 20 20 20 20 6e 65 78 74
5f 64 69 66 66 20 3d 20 30 0a 20 20 20 20 20 20 20 20 69 20 3d 20 36 34 0a 20 20 20 20 20 20 20
20 66 6f 72 20 62 79 74 65 20 69 6e 20 72 61 6e 67 65 28 38 29 3a

All that hex decodes to the following, which is mainly portions of esp8266/scripts/apa102.py, esp8266/scripts/neopixel.py, and esp8266/scripts/onewire.py:

G
Ð'@F�^v��Ð'@þÐ'@n|Ð'@~pÐ'@¾�dÐ'@Î�XÐ'LÐ'@�&'@��h�'@�&'@��Ô�'@�&'@����'@�&'@��L�'@�&'@����'@�&'@��´�'@�&'@��Ø�'@#%d ets_task(%p, %d, %p, %d)
task prio out of range: %d
ets_post: task %d queue full
ets_alt_task: ets_run# APA102 driver for MicroPython on ESP8266
# MIT license; Copyright (c) 2016 Robert Foss, Daniel Busch

from esp import apa102_write

class APA102:
    def __init__(self, clock_pin, data_pin, n):
        self.clock_pin = clock_pin
        self.data_pin = data_pin
        self.n = n
        self.buf = bytearray(n * 4)

        self.clock_pin.init(clock_pin.OUT)
        self.data_pin.init(data_pin.OUT)

    def __setitem__(self, index, val):
        r, g, b, brightness = val
        self.buf[index * 4] = r
        self.buf[index * 4 + 1] = g
        self.buf[index * 4 + 2] = b
        self.buf[index * 4 + 3] = brightness

    def __getitem__(self, index):
        i = index * 4
        return self.buf[i], self.buf[i + 1], self.buf[i + 2], self.buf[i + 3]

    def write(self):
        apa102_write(self.clock_pin, self.data_pin, self.buf)
# NeoPixel driver for MicroPython on ESpin)

    def readbyte(self):
        return _ow.readbyte(self.pin)

    def read(self, count):
        buf = bytearray(count)
        for i in range(count):
            buf[i] = _ow.readbyte(self.pin)
        return buf

    def writebit(self, value):
        return _ow.writebit(self.pin, value)

    def writebyte(self, value):
        return _ow.writebyte(self.pin, value)

    def write(self, buf):
        for b in buf:
            _ow.writebyte(self.pin, b)

    def select_rom(self, rom):
        self.reset()
        self.writebyte(MATCH_ROM)
        self.write(rom)

    def scan(self):
        devices = []
        diff = 65
        rom = False
        for i in range(0xff):
            rom, diff = self._search_rom(rom, diff)
            if rom:
                devices += [rom]
            if diff == 0:
                break
        return devices

    def _search_rom(self, l_rom, diff):
        if not self.reset():
            return None, 0
        self.writebyte(SEARCH_ROM)
        if not l_rom:
            l_rom = bytearray(8)
        rom = bytearray(8)
        next_diff = 0
        i = 64
        for byte in range(8):```
@dpgeorge
Contributor

Try erasing the flash entirely before deploying the image. It looks like the ESP SDK is finding the wifi/system parameters in the wrong location.

@Ratfink
Ratfink commented Aug 25, 2016

I erased the flash before deploying the image every time above. It does seem like it's reading the system parameters from the wrong location. From this pull request from NodeMCU, it looks like the ESP SDK doesn't really support more than 4 MB of flash even though the chip itself does. Maybe it would be better to not worry about supporting this much flash until Espressif themselves do.

@dpgeorge
Contributor

it looks like the ESP SDK doesn't really support more than 4 MB of flash even though the chip itself does. Maybe it would be better to not worry about supporting this much flash until Espressif themselves do.

Yes. We can revisit this issue when upgrading to the latest ESP SDK, which has a new way of locating and storing the persistent wifi/config parameters.

@peterhinch
Contributor

Is that kosher? It doesn't appear on http://www.wemos.cc/

@mcauser
Contributor
mcauser commented Aug 31, 2016

Yeah it's legit. Was only released a few hrs ago. It'll be on their website soon, I guess.

@deshipu
Contributor
deshipu commented Oct 19, 2016 edited

So now that we use the new sdk, I tried this again, but I'm still getting a boot loop.

l��|��rrnb��l�b�lb쌜���lb�lrlll��|��rrnb��ll��b�b쌜��b��lrl�l��|��rrnb��l��b�b쌜��lb�lbl����n�r��n|�llll`��r�l�l�l`��r�l�l�l`��r�l���ll`rl��rl���b��b�bbr�rb��n�nn�l��l�l��ll������l�n����bll�rp���bl�brlrlr�n�����n����brb��b쎜�l쌌�pbl`��l�#4 ets_task(40100164, 3, 3fff6388, 4)
Bootloader Flash size appear to have been set incorrectly, trying to fix
bcn 0
del if1
usl

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 32020, room 16 
tail 4
chksum 0x18
load 0x3ffe8000, len 1084, room 4 
tail 8
chksum 0xd6
load 0x3ffe8440, len 3000, room 0 
tail 8
chksum 0xac
csum 0xac
@robert-hh
Contributor

I looked a little bit around on this topic and found this link: http://www.packom.org/esp8266/16mb/flash/eeprom/2016/10/14/esp8266-16mbyte-flash_handling.html
It also point to the ESP SDK documentation and uses a patch for the flashchip data structure, which embedded in ROM and has the flash size hard coded. espressif seems to have built a patch mechanism into it's code for these purposes. I did not dig further, but in addition to the changes to the python code, that seems to be required.

@deshipu
Contributor
deshipu commented Feb 24, 2017 edited

There is some hope, the next version of (non OS) SDK should support it: espressif/esptool#123 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment