Number of bytes increases while removing lines of code - Too big to save to flash

[espruino-discuss]

Posted at 2017-02-23 by Robin

Wed 2017.02.22

I felt this more of a topic for 'General' but also specific to the 'ESP8266' Will understand if needed to be moved to the other category.

As I attempt to clean up lines of code that will no longer fit in the available memory space, I notice that the number of bytes attempted to be saved increases, even after deleting both lines of code and comments.

The error 'Too big to save to flash' occurs and from a quick observation, it can be seen that the intended number of bytes 13139 is larger than the 12284 allotted, despite 1/3 of memory remaining.

Why, when there is 1/3 of the available space remaining, the inability to save a few additional lines of code?

1v89 Copyright 2016 G.Williams
Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:512/512, manuf 0xe0 chip 0x4016
>
{ "free": 415, "usage": 1285, "total": 1700, "history": 1 }
>save()
=undefined
Erasing Flash.....
Writing...............
ERROR: Too big to save to flash (13139 vs 12284 bytes)
Deleting command history and trying again...
Erasing Flash.....
Writing...............
ERROR: Too Running onInit()...
[L914] inside onInit()

I removed at least thirty lines of code and twenty comment lines, yet the total bytes only decreased by 48 and the memory usage block doesn't even budge. I expected around 1000+ bytes to decrease.

Writing...............
ERROR: Too big to save to flash (13091 vs 12284 bytes)
Deleting command history and trying again...

>process.memory()
={ "free": 415, "usage": 1285, "total": 1700, "history": 1 }
>

So I became more aggressive and whacked another fifty lines and to my surprise the number of bytes jumped up to 17568, despite aggressive removal of code lines. With the usage now lower and free space now larger, I now expected to be able to easily save. But as one can see, the save() routine wont allow this smaller block of source code to be archived.

Writing....................
ERROR: Too big to save to flash (17568 vs 12284 bytes)
Deleting command history and trying again...

>process.memory()
={ "free": 532, "usage": 1168, "total": 1700, "history": 1 }
>

Is this the memory leaking that was described here?:
http://forum.espruino.com/comments/13469254/

I read over:
http://www.espruino.com/ESP8266_Flashing
http://www.espruino.com/Internals
http://www.espruino.com/Performance
but didn't locate suitable info other than 'Variables are usually stored in fixed-size 16 byte blocks'

The offsets from an ESP-12 allow for 0x3FFF or 3 * 16 * 256 = 12288 bytes which I can see is used for the 12284 value
0x3FC000
0x3FE000
0x400000

But the other values are more puzzling.

Q: How much in bytes is the actual space indicated by 'free'

Q: How large are these blocks re: total 1700

Q: What is the relationship between the total blocks 1700 and used bytes 12284

Q: Are comments stripped from the source before 'Send to Espruino'? e.g. are they counted in used byte count?

Q: Why couldn't I save this code block despite having ample free space?

[espruino-discuss]

Posted at 2017-02-23 by Wilberforce

try this on the left side:
reset();

then upload via ide, see if starting from a clean interpreter state makes a difference (This is usually the default in the IDE settings to reset first.

Q: How much in bytes is the actual space indicated by 'free'

it's not bytes but jsvars....

Q: How large are these blocks re: total 1700

So you have 1700 jsvars free

Q: What is the relationship between the total blocks 1700 and used bytes 12284

12284 Bytes is the number of chars in you javascript source.

Q: Are comments stripped from the source before 'Send to Espruino'? e.g. are they counted in used byte count?

It depends ;-) The position of the comments is a factor if they 'take up space' - I can't recall the details
This is why modules get minified before upload to strip comments and shorten any externally referenced variables.

Q: Why couldn't I save this code block despite having ample free space?

The ESP8266 is pretty strapped for space. It really is quite limiting.

---

Posted at 2017-02-23 by @MaBecker

Hi Robin,

Is this the memory leaking that was described here?:

No, as you are running on 1.89

I removed at least thirty lines of code and twenty comment lines, yet the total bytes only decreased by 48 and the memory usage block doesn't even budge. I expected around 1000+ bytes to decrease.

• check section MINIFICATION in WebIDE SETTINGS

• use dump() to list code

• shorten long names eg var sFCC = String.fromCharCode; if used more than one time to shrink size of code.

---

Posted at 2017-02-23 by navas

global["\xFF"].history = [];

Add this on top-off the line of our source code and enable minification for code from IDE settings.

http://take.ms/CVBrq

---

Posted at 2017-02-23 by @gfwilliams

This might be handy? http://www.espruino.com/Performance

Using Save on send in the IDE will write the raw JS code to the device, so your changes to the number of characters will have a direct effect.

Using the normal save() saves (and compresses) an image of memory to flash. It means that if you define things like arrays, they'll take up significant amounts of memory. Comments outside functions won't use space, but comments inside will.

But because it's compressed the amount of memory taken up is virtually impossible to know beforehand - if something happens and data is re-arranged in memory then even that can effect the compression

---

Posted at 2017-02-23 by DrAzzy

The ESP8266 in particular doesn't have as much space in the flash to save code to as it does RAM to store the code you have uploaded. So you can get into situations like yours, where your code fits in memory, but you can't save it because even compressed, there's not enough space in the flash for it.

---

Posted at 2017-02-24 by Robin

Thr 2017.02.23

Thank you for your responses to each of my questions:

What is the relationship between the total blocks 1700 and used bytes 12284 - 12284 Bytes is the number of chars in your javascript source

What I was after was total used space of 1700 blocks. @wilberforce your comment 'So you have 1700 jsvars free' allowed me to do a Google search on the needed concept of 'jsvar' which turned up additional detail.

http://www.espruino.com/Internals

Variables are usually stored in fixed-size 16 byte blocks. In small devices this can get down to 12 bytes (10 bit addresses) and on PCs it's 32 bytes (32 bit addresses)

Which then led to:

http://forum.espruino.com/conversations/276825/

As a rule of thumb, expect to be able to store 12 characters for each 'JsVar' reported by process.memory, so if you have 5000 variables then that's 60,000 characters.

----snip from above #1

The offsets from an ESP-12 allow for 0x3FFF or 3 * 16 * 256 = 12288 bytes which I can see is used for the 12284 value
0x3FC000
0x3FE000
0x400000

and performing the calculation: 1700 * 12 = 20400 which exceeds the physical capacity of 12284 actual bytes alloted. On to 12288 / 12 = 1024 a rough approximation on the number of 'jsvars' that will fit in the available memory.

Author note: Still would like to know why 1700 is the magic limit

Writing............
Compressed 27200 bytes to 9390
Checking...
>process.memory()
={ "free": 761, "usage": 939, "total": 1700, "history": 1 }
>

So, my take on this is, before minification, the environment can temporarily work with more code as it is unknown what the minification process can compress down towards. Looking at the process.memory() 'usage' argument can only approximate whether the current source code will fit. Since our base 10 is close to the 12 multiplier, one could approximate the save success probability by multiplying the 'usage' value by 10 and if that is below 12284, be reasonably certain of a successful save(). As a precaution, this implies one needs to be really observant as the 'usage' argument approaches 1000.

Note that individual mileage may vary. ;-)

---

Posted at 2017-02-24 by Robin

Thr 2017.02.23

Thank you @MaBecker for the reminders

shorten long names eg var sFCC = String.fromCharCode;

I'll use this technique in my nearly identical html header block responses. Will take some time to refine, but this will shrink the number of lines of code significantly.

---

Posted at 2017-02-24 by Robin

Thr 2017.02.23

global["\xFF"].history = [];

For clarification, is this just clearing the memory space of pre-existing console history entries?

I did find a separate reference on hidden global variables:

http://forum.espruino.com/conversations/274778/

trace(global["\xFF"])

Thank you for your input @navas

---

Posted at 2017-02-24 by Robin

Thr 2017.02.23

@gfwilliams

Comments outside functions won't use space, but comments inside will.

That is a key piece I was after, and answers what would have been my next question. Thank you.

Curious why function comments are preserved during save(). Is the limiting factor the amount of space the interpreter and minification engine take up and/or the number of additional pass(es) penalty needed to pre-parse and remove embedded comments?

I'm starting to believe the reason in my save code example, the number of bytes kept increasing despite removing lines of code was that the existing state machine for the embedded comments was derailed when using a block comment around existing code containing a line comment, and/or when mixed languages contained embedded escape chars. Not finding the true end-of-comment kept piling up chars as code, eventually exceeding the available space required for the save.

Just realized, it's more likely the complexity of the state engine in attempting to retain actual code. Hadn't meant that to sound nit-picky as we realize what an undertaking it is to create a software machine. You should be quite proud of what you have accomplished as much as we are mesmorized over what we wish we could do. It is this quality that has allowed the Espruino brand to cultivate this growing community.

Gordon, Thank you for all your effort.

---

Posted at 2017-02-24 by @gfwilliams

as it is unknown what the minification process can compress down towards

It's not minification - there is actually proper file compression inside Espruino that takes the contents of RAM and squeezes it (reversibly) into Flash.

To be honest on most ESP8266 parts you have bags of free flash. There's absolutely no need for the 12k restriction but I think it's been left in to be compatible with the 512k ESP01 parts.

Curious why function comments are preserved during save()

Espruino keeps the code you uploaded so that you can edit it on/recover it from the device if you need to. Otherwise it'd make more sense to just pre-compile everything.

If you want comments removed, turn on minification in the Web IDE and they'll get ripped out before even being sent to the device.

Not finding the true end-of-comment kept piling up chars as code, eventually exceeding the available space required for the save.

I would be a bit dubious about this - Espruino's been in use a lot and I'd be surprised if a bug like that still existed. If you get some code that exhibits the behaviour post it up and I'll check it out.

---

Posted at 2017-02-24 by @MaBecker

To be honest on most ESP8266 parts you have bags of free flash. There's absolutely no need for the 12k restriction but I think it's been left in to be compatible with the 512k ESP01 parts.

Yes - Compatible setting that work for all ESP models:

• three pages to save code (0x78000 - 0x7A000)
• one page to save wifi (0x7B000)
• firmware can be up to 468KB

---

Posted at 2017-02-26 by Robin

If you get some code that exhibits the behaviour post it up and I'll check it out.

@gfwilliams,

Although I wasn't able to duplicate the 500+ byte save() increases that occurred when deleting 4K chunks of source, I was able to capture a 248 byte increase example.

In order not to muddy this thread with massive source code image blocks, I'm providing it here in five files.

Increase occurs between file PASS1 and PASS2 after the removal of both internal comments and wrapper functions.

Although this is of low priority, one might find some devious gotcha's lurking about. Would love to hear your findings, even if by email.

Robin

---

Attachments:

• NotesGordon20170224.txt
• code20170206ESP8266FileServer.jsSTART.js
• code20170206ESP8266FileServer.jsPASS1.js
• code20170206ESP8266FileServer.jsPASS2.js
• code20170206ESP8266FileServer.jsPASS3.js

---

Posted at 2017-03-13 by @gfwilliams

So in this case, are you running with minification turned on?

You're saying you upload code20170206ESP8266FileServer.jsSTART.js exactly as it is, and then remove:

//this is the problem section
    
//   for (var keysv in objGip) {
////    console.log("[563] key: "+objGip[ 'ip' ]);
//    console.log("[563] key: "+objGip[ ip ]);
   
//   }    
    
////    rettext = sGip.toString();
////    rettext = "ip: " + objGip[ 'ip' ].toString();
//    rettext = "ip: " + objGip[ ip ].toString();
//       console.log( "[571] *** rettext: [ " +  rettext + " ]" );

//    rettext = "ip: " + objGip.ip.toString();
//       console.log( "[571a] *** rettext: [ " +  rettext + " ]" );

from onPageRequest, and the code gets bigger?

Because when I try this on a normal Espruino board with 1v91, I get:

// before
>process.memory().usage
=1341
Compressed 81600 bytes to 31405

// after
>process.memory().usage
=1311
Compressed 81600 bytes to 30856

or PASS1: Compressed 81600 bytes to 25920
and PASS2: Compressed 81600 bytes to 25058

or after minification:

PASS1: Compressed 81600 bytes to 16304
and PASS2: Compressed 81600 bytes to 15895

which is exactly what you'd expect? Sadly I can't find my working NodeMCU board at the moment to test this on.

If you're looking to save memory, it seems like you have multiple joinWifi functions, and also both onPageRequest and pageHandler which I guess do similar things?

---

Posted at 2017-03-13 by Robin

Mon 2017.03.13

Thank you for running tests on your boards. It's unfortunate that duplicating my experience didn't occur.

are you running with minification turned on

I haven't made any changes to the WebIDE settings for any of the examples. (for minification - left factory defaults as is - yes?)

ref:

//this is the problem section

I left this comment intact to avoid adding more confusion when removing lines of code. The comment actually refers to the commented out block that follows, there was an unresolved issue decoding the unknown JSON object that is returned from wifi.scan()

"You're saying you upload code20170206ESP8266FileServer.jsSTART.js­ exactly as it is, and then remove:"

Yes, with clarification.

I removed around 3K-4K of bytes each time, around ten times more than indicated above, both code and external comments. I was also jumping around seeking sections to remove, so this might be a contributing factor. e.g. whitespace

The only way to tell exactly what was removed would be to compare side by side with software that could detect the changes.

When I was able to duplicate this increase, I created the file 'NotesGordon20170224.txt' and appended the steps in chronological order.

I indicated (the best I could as a text file isn't the best here) which file, brief list of areas I removed code, the WebIDE process.memory()
output, followed by the saved bytes resolution, renamed and saved the file, then repeated the process. I did all the editing in the WebIDE

Could it be that ver 1v89 is problematic here?
I never was able to resolve why I couldn't flash 1v91 on this ESP8266-12 so I'm stuck at 1v89 ref: 'save() on espruino_1v91.122_esp8266 results in immediate Disconnected prompt'

Note that I never did test on my Pico as I was working with wifi and EspruinoWifi was not available from suppliers here in the USA.

I'm not seeking a solution to the limited space issue nor am I seeking help in resolving coding issues. It's unfortunate that what happens on
my PC isn't being duplicated. I note that we are working with a different board and different software version.

Still wondering if not being able to flash is related to the save() disconnect issues others have had with the puck and this weird anomaly when using large file sizes. If this has become too much a time sink, then maybe we should just shelve this for now and see if others ever have similar issues. I felt it important to document for the reasons above and the fact it is repeatable.

EDIT: After several hours of mulling this over I realize the following:

Point 1
In each test case above, a complete new file was loaded into the WebIDE before a transfer via 'Send to Espruino' button press.

In my case, I used one file and reduced that, never reloading a fresh data set. Although the original file is gone, I started with a file that executed as expected, taking up one half the destination space of around 800 jsvar blocks. I inserted a 4K char chunk of function code towards the end of the file. This caused the error

Uncaught SyntaxError: Got UNFINISHED STRING expected EOF at line 1 col 8640

I started to remove unneeded comments and functions totaling around 4K chars each attempt. Then saved that result to a file, those that are included above.
Note that I'm always using the same file in the WebIDE memory space.

Point 2
The available size on the 1v91 test above is much larger and compresses to a much larger result ~30000 vs ~10000
The difference using an Espruino board vs an ESP8266-12

>process.memory().usage
=1311
Compressed 81600 bytes to 30856

vs

>process.memory()
={ "free": 482, "usage": 1218, "total": 1700, "history": 431 }
ERROR: Too big to save to flash (16445 vs 12284 bytes)
Compressed 27200 bytes to 11884

---

Posted at 2017-03-14 by @gfwilliams

I haven't made any changes to the WebIDE settings for any of the examples. (for minification - left factory defaults as is - yes?)
This caused the error: Uncaught SyntaxError: Got UNFINISHED STRING expected EOF at line 1 col 8640

I'd be almost certain you've definitely changed the Web IDE settings, since the file you gave me doesn't have any 8600 character long lines. 'minification' in settings (the first item) is supposed to say 'No Minification' and I bet it doesn't.

Just to follow up with what you sent in the PM, as this is probably useful for everyone:

I said that this was because minification was turned on, and the Web IDE had rammed everything into one giant line of text, which overflowed the available memory and got cropped, not finishing the second string.

I'm not in agreement that there isn't any managing to finish the second part

It's not the second string, the line that's printed in the error actually starts thousands of characters from the start of the file . If you take a look under settings->console you should see the code that's actually uploaded.

It should look a lot like this - I won't post the whole thing in the forum because it's huge:

"\u0010reset();\n\u0010\u001b[1dfunction pageHandler(b,a){u=url.parse(b.url,!0),console.log('[074] inside MaBe pageHandler u [ '+b.url.toString()+' ]'),console.log('[075] inside MaBe pageHandler u.method [ '+u.method+' ]'),console.log('[076] inside MaBe pageHandler u.host [ '+u.host+' ]'),console.log('[077] inside MaBe pageHandler u.path [ '+u.path+' ]'),console.log('[078] inside MaBe pageHandler u.pathname [ '+u.pathname+' ]'),console.log('[079] inside MaBe pageHandler u.search [ '+u.search+' ]'),console.log('[080] inside MaBe pageHandler u.port [ '+u.port+' ]'),console.log('[081] inside MaBe pageHandler u.query [ '+u.query+' ]'),b.method=='POST'&&
...
'+a.ssid),console.log('[L905] details.mac '+a.mac),console.log('[L906] details.channel '+a.channel);}),setTimeout(onInit,1500);\n\n"

But DOCTYPE is buried in there towards the end. So when you get this:

>ERROR: Out of Memory!
Uncaught SyntaxError: Got UNFINISHED STRING expected EOF
 at line 1 col 8640
...Wifi');var C={SSID:'',PAGE:"<!DOCTYPE html><html lang=\"en\"...
                              ^

You'll notice that the pointer is pointing to the second string on the line, but actually that string starts at nearer column 7700!

So the fact that the error is at column 8640 means it's at character 940 of your string, when you say your string is about 1k long - so it's cropped.

Despite the displayed error, the code executes as expected.

I believe it will have executed everything up until that point. Everything after the declaration of C will have been lost, but presumably that's not used in your code.

If you were getting the out of memory error all along, that probably explains everything

When you reduce the code size enough that you get no error, everything executes fine and all functions get added. When you get the error uploading because the code is too big, some functions don't get added and so the final code looks smaller.

and performing the calculation: 1700 * 12 = 20400 which exceeds the physical capacity of 12284 actual bytes allotted. On to 12288 / 12 = 1024 a rough approximation on the number of 'jsvars' that will fit in the available memory.

12288 is the amount of bytes in flash - after compression. But yes, 1024 is a reasonable minimum.

However depending on what code you upload, Espruino can't get 100% usage - see http://www.espruino.com/Performance

Q: Author note: Still would like to know why 1700 is the magic limit and not around 1024 a calculated value

It's because 12288 is the amount of Flash, but you have more RAM available. At >1024 vars it's 16 bytes a var, so 1700 * 16 = 27200 bytes.

Q: Despite indicated remaining memory, has this 8K file reached a perceived capacity limit for this ESP to handle?

It's because you're trying to upload 8K bytes in one giant line. When Espruino has a string that it has to append to a few characters at a time, it's not as efficient at storage, which explains why it runs out more quickly.

Q: Is there a suggested limit. e.g. has a known reasonable physical limit been determined?

It depends on the board, and even on the firmware version. I don't know for sure.

---

But just to sum up:

It looks like there's a 'bug' in minification which causes everything to be put on a single line, but only when 'Minification' is set to 'Esprima (offline)'. It's not the default. Set minification to Closure(online) - Whitespace Only and I bet it'll work perfectly, but the default is actually 'off'.

I filed a bug for this here: espruino/EspruinoTools#61

But yeah, if you get errors, always look at the very first error, and don't always take subsequent ones at face value :)

---

Posted at 2017-03-20 by Robin

Sun 2017.03.19

'minification' in settings (the first item) is supposed to say 'No Minification' and I bet it doesn't.

in reference to the brash 'I bet it doesn't' verbage along with verbose specific recommendations in forum post 'Blackjack Secret Card Counter'

Oh nooo! @gfwilliams you are not giving up on Espruino to explore your new passion of gambling, are you?? ;-)

Kidding aside, it is good to know you were able to duplicate and identify what was going on, and also presume what the cause was, filing the bug report indicating such. Knowledge and experience does go along way, doesn't it.

Although I never intentionally set options under the minification settings, however I did check, but have no explanation for why the option 'Esprima (Offline)' was enabled and not the default as you indicated.

Sun spots, aliens, intruders, the neighbor, late nights coding, mornings without coffee?

Insert sponsor advertisement and marketing hype here

A morning without coffee? Try Espruino brand coffee. Ooooh, I smell additional revenue stream. . . .

Yes you heard it here first, a sample packet of the Espruino blend and mug with every official board purchased!

[Off Soapbox]

It's possible I fiddled with this on the day of installation, and never really put back how I remembered the defaults to be. More likely a window covering the WebIDE was being clicked on, and as Windows has that nasty habit of delaying click recognition, passed the click through to the WebIDE, even though the covering window hadn't been dismissed yet. Can't tell you how many times I've been burned by that anomaly. Note to self: Put Cray XT5 on X-Mas list

Still befuddled with: 1700 * 16 = 27200 bytes I get the simple math.

It's because 12288 is the amount of Flash, but you have more RAM available. At >1024 vars it's 16 bytes a var, so 1700 * 16 = 27200 bytes.

Maybe a better question would have been, why isn't the remainder more in line with a common 2n multiple? When sending code, the internal methods use the 1024 boundary which is 4 0xFF a nice computer number but 27200 seems so clumsy, it doesn't seem to have as nice an explanation. 256128=32768 is too large and 25664=16384 too small. 27200 hummmmmmm?

Thank you Gordon, for staying with this until completion. I know this has been a bit drawn out when there is so much else that requires your expertise.

I know others will appreciate your explanation as it gives a good analysis of what is going on under the hood.
Robin

---

Posted at 2017-03-20 by @gfwilliams

Yes, I'm not sure. I think at some point we did suggest that you use minification to get the code size down, and that could have been it?

It's fixed now, so when I next update the Web IDE you should see the improvements.

why isn't the remainder more in line with a common 2n multiple?

That's an easy one. The chip itself may well have 32/64/128/etc bytes of RAM, but other things use that RAM as well (execution stack, wifi, etc). Espruino uses up what's left, which is pretty much guaranteed not to be a nice round number.