New Flash API, and nice surprise for Pico owners!

[gfwilliams]

Posted at 2015-07-17 by @gfwilliams

Hi,

I just did a new build, with a more universal API for accessing flash memory, which will work on both the Original board and the Pico. It's here and will be in 1v80 when it's released.

Rough outline:

>var f = require("Flash");
>var freeFlash = process.memory().flash_code_start;
>f.getPage(freeFlash)
={ "addr": 134234112, "length": 16384 }
>f.read(12, freeFlash)
=new Uint8Array([255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255])
>f.write("Hello World!", 134234112)
=undefined
>f.read(12, freeFlash)
=new Uint8Array([72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33])
>f.erasePage(freeFlash)
=undefined
>f.read(12, freeFlash)
=new Uint8Array([255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255])

It'll all go in the reference when the full release is made.

Be careful, because you can easily blow away your bootloader with it if you're not careful!

And the surprise is...

>var freeFlash = process.memory().flash_start + 511*1024;
=134740992
>f.read(12, freeFlash)
=new Uint8Array([255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255])
>f.write("Hello World!", freeFlash)
=undefined
>f.read(12, freeFlash)
=new Uint8Array([72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33])

So there's a whole extra page of flash memory, 128kB long at address 0x8060000 that we shouldn't have. Much like on the original boards, the chip is a STM32F401CE even though it's marked as a STM32F401CD.

While there's no guarantee that newer boards will be like this, I reckon all the current ones are!

... also this firmware has some new changes that use simple RLE compression when saving program code to flash. It means I've been able to raise the amount of variables from 3000 up to 5100. While you won't be able to save a program that uses all of them, realistically you need a few free variables to run your program, and those will compress down.

[espruino-discuss]

Posted at 2015-07-17 by DrAzzy

WOOT WOOT WOOT! MOAR RAM!
Just in time too - I might have scraped by under 3kJSV, but it might have been tight. But 5.2kJSV will be plenty!

It's a shame we can't rely on the bonus flash, that'd be the perfect place to save the code to!

So I should discontinue the picoram builds now?

[gfwilliams]

Posted at 2015-07-17 by @gfwilliams

So I should discontinue the picoram builds now?

Totally up to you. You could I guess tweak PICO_1V3.py such that it sticks the saved data in that final page of flash, then you won't hit problems if the RLE compression doesn't manage to compress the data enough.

[espruino-discuss]

Posted at 2015-07-20 by DrAzzy

Well, this build seems to work for me? I tried it on the room controller (the one I've posted pics of here), and save()'ed code seems to work correctly, though I wasn't pushing the size limits.

[gfwilliams]

Posted at 2015-07-21 by @gfwilliams

Great, thanks!

We should be able to do a lot better on the compression front, I'll have to look at it a bit more - but for now RLE seems to do a relatively good job.

[espruino-discuss]

Posted at 2015-07-21 by DrAzzy

!!!

I just realized something. This new flash capability could be used to overwrite the bootloader, you say... Doesn't that also mean it could be a way to update the bootloader without having to enter DFU mode? Entering DFU mode is a bit difficult if you've soldered the pico onto something, so the pads you have to short can't be accessed without unsoldering 26 castellated pads.

[gfwilliams]

Posted at 2015-07-22 by @gfwilliams

Yes, you can do that with it too...

I was just avoiding doing it because it's a great way to brick your Pico (you'd then have to enter DFU to fix it) if something goes wrong :)

But if you created a Uint8Array with the first 16kB of the Espruino binary, then erased the page at 0x08000000 and then wrote the array into that address, you should be sorted.

but if you want to enter DFU with a Pico soldered down, there's another method too. If you look on the top of the board there's a small gold teardrop. That's BOOT0, so if you just short that to 3.3v while power is applied the board will enter DFU mode :)

[espruino-discuss]

Posted at 2015-08-01 by DrAzzy

Uhh...

Bad news. I've been having hard failures that require reflashing firmware to fix.

No idea what I do, but sometimes after save(), next time I unplug and replug, the serial isn't recognized, and appears as Unknown USB Device (Device descriptor request failed).

:-(

Very strange. The issue may be linked to the webserver it was trying to talk to not being online.

I've gone back to release version for now.

Attachments:

• VoiceController.js

[gfwilliams]

Posted at 2015-08-03 by @gfwilliams

:( I actually did a full release on Fri 24th, then had to back it all out because I realised there was a problem.

I found something bad happened repeatably when I did:

• save();
• Maybe power off/on
• do not reset()
• Change something
• save();

Does that sound like your problem?

[espruino-discuss]

Posted at 2015-08-03 by DrAzzy

Yup

[gfwilliams]

Posted at 2015-08-03 by @gfwilliams

Please could you try this build when it's compiled - it's just the latest on GitHub.

I think I may have found the problem now - at least I haven't managed to get it to die yet.

[espruino-discuss]

Posted at 2015-08-03 by DrAzzy

Cool, I'll flash it as soon as I get home :-D

[gfwilliams]

Posted at 2015-08-05 by @gfwilliams

Anyone had any luck with this yet?

[espruino-discuss]

Posted at 2015-11-08 by Sacha

Hi Gordon

I'm able to access the extra page on the pico. Great.
Have a question regarding Flash.erasePage. Will this allways erase the complete page, in this case the 128KB flashmemory ? Is there a way to only erase a part and rewrite that part ?

Thanks

Sacha

[espruino-discuss]

Posted at 2015-11-09 by Dennis

From my experience with other NAND flash memories, I would strongly assume that you can only erase a whole page, because that's the way they work. There are non-volative RAMs out there as well, without that limitation, but they are much more expensive.

I'm not sure about this particular chip though.

[gfwilliams]

Posted at 2015-11-09 by @gfwilliams

Yes, I'm afraid it's only the complete page. The STM32F4 does have different size pages (there are 16k and 64k ones too, earlier in memory) but those are used for Espruino.

[espruino-discuss]

Posted at 2015-11-09 by Sacha

Hi Gordon,

Running this code:

// Flashtest

function gen_256() {
    var i;
    var str='';
    for(i=0;i<256;i++) {
        str+='X';
    }
    return(str);
}

var bytes256=gen_256();
var f = require("Flash");
var freeFlash = process.memory().flash_start + 511*1024;
console.log(f.getPage(freeFlash));
console.log('LEN:'+bytes256.length);
f.erasePage(freeFlash);
console.log('Nr.1, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.2, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.3, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.4, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.5, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.6, at '+freeFlash);
f.write(bytes256, freeFlash);
freeFlash+=256;
console.log('Nr.7, at '+freeFlash);
f.write(bytes256, freeFlash);

[espruino-discuss]

Posted at 2015-11-09 by Sacha

Sorry have troubles formatting it as code.

The output running above code is:

 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v81.149 Copyright 2015 G.Williams
>echo(0);
{ "addr": 134610944, "length": 131072 }
LEN:256
Nr.1, at 134740992
Nr.2, at 134741248
Nr.3, at 134741504
Nr.4, at 134741760

Freeze. Have to unplug/plug it back to recover.

Sacha

[espruino-discuss]

Posted at 2015-11-09 by Sacha

Doublepost

[gfwilliams]

Posted at 2015-11-10 by @gfwilliams

Hi Sacha,

To format code, just stick 3 backticks before and after it - check out this page. It seems like the forum guys have broken the code button now!

The issue is just that you're going past the end of available memory...

• You have have 512*1024 bytes total in the chip
• You start at 511*1024
• The first 4*256 = 1024 bytes write great
• But now you're at the end of memory and any further writes cause a hardware error

If you change the start address to 510*1024 then it should work fine - in fact you've got the whole page from 384*1024 (which you can set with freeFlash = f.getPage(freeFlash).addr)

[espruino-discuss]

Posted at 2015-11-10 by Sacha

Wow, that easy. I did not see where the staring point of the free memory is. Will try it this evening.
Thanks

Sacha

[espruino-discuss]

Posted at 2015-11-11 by Sacha

Hi Gordon
Works perfect. I'm able to use the extra mem.
Sacha

[espruino-discuss]

Posted at 2015-11-25 by Sacha

Hi Gorden
Sorry, something goes wrong with 1.82. This code worked with 1.81. Now the Pico crashes with the following on the console:

>ff = process.memory().flash_start + 384*1024;
=134610944
>f=require("Flash");
=function () { [native code] }
>f.erasePage(ff);
=ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

After reinsert the pico into USB, the red LED is on permanently. Have to reflash to access the pico.

Thanks

Sacha

[gfwilliams]

Posted at 2015-11-25 by @gfwilliams

Wow, that's not good. Thanks - I'll look into it.

[gfwilliams]

Posted at 2015-11-25 by @gfwilliams

Ok, that's bad news. Just checked and the firmware is actually too big and is encroaching on that last page.

Annoyingly the build-time check I have didn't work with the Pico's flash memory layout. I'll try and save a bit of space in the build and will re-upload with a 1v83.

[espruino-discuss]

Posted at 2015-11-25 by Sacha

Ok. I will use 1v81 in the meantime.

[gfwilliams]

Posted at 2015-11-27 by @gfwilliams

1v83 released, and there's now enough memory free - so you can go back to using the flash again.

[espruino-discuss]

Posted at 2015-11-27 by @allObjects

...for example to store all this read/only TSL stuff...

[gfwilliams]

Posted at 2015-11-27 by @gfwilliams

... if only! Sadly I don't think there's a way to do that with the firmware as it is now, because the API requires that all 3 variables are in the same object at once.

However I might be able to make some minor changes to make that happen. I have just filed an issue for it: espruino/Espruino#738

[espruino-discuss]

Posted at 2015-11-28 by Sacha

Hi Gordon
Thank you very much! Works again. Compared to 1v81, in my current project, there is around 220 more free memory.
Sacha

[espruino-discuss]

Posted at 2015-11-28 by @allObjects

@gordon, I see your 'disgust' having to break negative news... but before finishing, your 'back-burner' already infused a solution idea... Thanks.

PS: running from flash without copying into RAM was my first thought after taking a first look at Espruino's JS interpreter.