New features for improved memory usage

[gfwilliams]

Posted at 2016-11-24 by @gfwilliams

Just a quick note to say that the most recent builds of Espruino (so not 1v89, but version 1v90 when it's released) contain an improvement that'll really help with the amount of memory available.

• Flash the new firmware
• In the Web IDE, go to Settings -> communications and set Modules uploaded as functions and Save on send
• Upload your code

So what happens? Well, the Web IDE used to upload your modules as strings, which would then be decoded and executed by Espruino. It now uploads them as a function.

It means:

• Upload is faster since the modules don't have to be decoded
• Since modules aren't decoded, you don't need as much free memory to upload modules (you used to need 3x the size of the module available)
• And crucially: If you're saving all your code directly to flash (with save on send), the text of any functions that you define will actually stay stored in flash and won't use your RAM

[espruino-discuss]

Posted at 2016-11-24 by Spocki

Small change, big impact. Thanks ;-)

[espruino-discuss]

Posted at 2017-05-14 by AntiCat

Great feature, I think you should be promoting it more.

[espruino-discuss]

Posted at 2017-05-14 by AntiCat

Does this feature disable minify of editor code?

[espruino-discuss]

Posted at 2017-05-14 by @allObjects

Nope, uploading modules and uploading your code in the editor are independent things.

The upload of the code goes through the code and finds all your require("moduleName") occurrences an uploads the modules separately and does the same with each module it uploads, which guarantees that nested required modules will be found in the Modules cache at runtime when your code will run and 'ask' for them (...when the require(...) is finally actually executed...)

[espruino-discuss]

Posted at 2017-05-14 by AntiCat

Thank you for your explanation. I'm not sure if I expressed my question correctly.

If I enable Modules uploaded as functions and Save on send and upload code.
Then dump() returns the code like I wrote it in the editor, including every new line and comment. Is this an intended behaviour?

Attachments:

• demo.png

[espruino-discuss]

Posted at 2017-05-15 by @allObjects

In the IDE's Settings - Minification - with the first option Minification you specify how your code should be uploaded: as is in the editor - with all comments and white and other spaces - or minified. Minified has multiple options from little minification and little space saving but still very readable code 'down' to maximum minification with maximum space saving and little resemblance with original code. Check it out.

The next minification option is for how modules should be handled.

Again, your code and (your) modules can have their own, independent special treatment... ;-)

[espruino-discuss]

Posted at 2017-05-16 by hungryforcodes

Weirdly I tried this on an ESP8266 -- and first the uploaded code was the same size as the IDE code (suggesting it hadn't been minified -- as its usually much smaller by almost 1/3), and then at one point during the upload the board just freaked out with an out of memory error and then a continuous stream of "Execution Interrupted", until the upload failed due to the "Prompt not detected error...".

Under communications I had selected upload modules as functions, and then after "Save On Send" was set to "Yes". And under Minification I have both modules and regular code set to simple optimizations using Closure.

The board was flashed with a standard v92, using the last official release.

[espruino-discuss]

Posted at 2017-05-16 by @allObjects

Prompt not detected may come from a different issue: When you have code that is executing your application on upload vs. having an onInit() (or a-like), Espruino gets too busy to actually complete with upload and then upload timeout hits because prompt is not detected.

Some related background - like 2 cents - to upload:

Upload by the IDE is nothing else than a) disable echo in console, b) auto/programmatic entering of code in prompt 'line by line' and 'completed JS expression. If prompt is not 'coming back' / detected by IDE within reasonable time, IDE complains as experienced.

Therefore, it is best to not start any application code in 'Level 0 (zero)', and put the start into onInit() {...} function. I does not only prevent upload issues but also makes sure that all your initialization take place in a reliable and repeatable fashion... (such as setting up timers and watches, even though many - simple - examples do not follow this and save active timers and watches, which on a re-power may either not get started properly or may be duplicated and create nice debug challenges...)

For additional details see this conversation simple explanation how to save code that espruino run on start?.

[espruino-discuss]

Posted at 2017-05-16 by hungryforcodes

Yes, I use onInit. However my point was more that the code loads and works fine though if I disable Save On Send. If I enable it it runs out of memory on upload. To me it looks like the code is not getting minified when Save On Send is set.

Anyways no big deal I'll just disable it as I always have. I was just curious to try this feature.

[espruino-discuss]

Posted at 2017-05-16 by AntiCat

I verified all options and tried once with save and send on and once with save and send off.
There is definitely a bug:

---

Attachments:

• savendsend.png

[espruino-discuss]

Posted at 2017-05-17 by hungryforcodes

I guess another way to put this is -- have I and @AnticaT misunderstood this feature -- or is there really a bug? :)

[gfwilliams]

Posted at 2017-05-22 by @gfwilliams

It looks like there is an issue, yes. The minification happened after the 'save on send' transformation (by which time the code was all a string). Thanks for spotting it.

I've now fixed it, so if you're running the IDE straight from GitHub you'll see a difference, otherwise you'll have to wait for the next update I'm afraid.

[espruino-discuss]

Posted at 2017-05-22 by Wilberforce

Is there any version numbers in the ide - can you tell what version you have installed?

How do you run from github?

[gfwilliams]

Posted at 2017-05-22 by @gfwilliams

There's no obvious way of finding the version number I'm afraid.

Instructions for installing are on GitHub - it's pretty straightforward: https://github.com/espruino/EspruinoWebIDE#installing-from-github-latest-version

[espruino-discuss]

Posted at 2017-05-22 by hungryforcodes

So I built the IDE successfully -- and now it minimizes! Great!

However in the end it seems to complain about not having a connection to the internet?

 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v92.876 Copyright 2016 G.Williams
Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:512/512, manuf 0xe0 chip 0x4016
>Erasing Flash..................
Writing..........
Compressed 24800 bytes to 7447
Checking...
DonLoading 7447 bytes from flash...
Uncaught InternalError: Not connected to the internet
Uncaught InternalError: Not connected to the internet
Uncaught InternalError: Not connected to the internet
Uncaught InternalError: Not connected to the intern

I did this a few times and always the same result. Tried both options in Save On Send (including the execute after reset()). Not sure if anyone else has had this problem.

Disabling "Save On Send" acts as normal and my code runs. I had to reflash my board after, btw. Not sure why, resetting with rst cause 4, after a normal upload. Then it would boot normally with my code, but was weird, so I just reflashed it. These flash chips are notoriously unreliable, so I'll look into that.

Also, is the font smaller on the new version?

[gfwilliams]

Posted at 2017-05-23 by @gfwilliams

After disabling 'save on send' you probably want to run E.setBootCode("") to clear out any saved code - the flashing might have done that for you but it's a bit slower.

There errors you're having will be coming from your code - so you'd need to post up an example of what you're doing on the ESP8266 forums. My guess is you might be executing some code in the global scope (eg. not in onInit or a timeout) that expects an internet connection.

[espruino-discuss]

Posted at 2017-05-23 by hungryforcodes

Well the font is smaller for sure :P

OK thanks for that tip about E.setBootCode(""), I'll add it to my Espruino cook book :P

I'll have a look, its an IR controlled WS2812 light (using the "IRReceiver" module), but that code doesn't do much internet wise except run Wifi.stopAP() at the beginning, and elsewhere Wifi.startAP(), so I can telnet in and modify it.

But why would the code run fine if "Save On Send" is disabled? Maybe I'm not understanding some subtleties about the feature.

[gfwilliams]

Posted at 2017-05-23 by @gfwilliams

Without Save on send, your code is sent over and executed straight away. var a = 1+2 will create a variable called a with value 3.

With it on, your code is sent and stored as text. var a = 1+2 is only executed at boot/restart.

So if you accessed the net, chances are at upload time you'd have an internet connection, but when the board restarted right away you wouldn't.

All I can think is there was some issue loading the modules and the ESP8266 build has some code added that searches online for modules? Could you look in the Web IDE's console and post up the exact code that gets uploaded (ideally for a minimal bit of code that exhibits the problem).

[espruino-discuss]

Posted at 2017-05-25 by hungryforcodes

Thanks for this explanation -- it was VERY helpful :)

So I basically managed to reduce my entire code to:

console.log("hello");

and it was still the same error. Your suggestion of looking at the logs was good and I may have found the problem. Basically it appears at one point to look for an ESP8266_4MB.json at the http://www.espruino.com/json/ ,but there isn't one there, as it would appear that its not been updated (ie: there's a board in GitHub for this board, but not in the /json directory online).

Could this be it? I can't get my code much smaller than one console log :P And yes I did reflash the board each time before I did this, to make sure I started from virgin state on the board.

GET chrome.storage.sync = {"AUTO_SAVE_CODE":true,"
BAUD_RATE":"115200","BLOCKLY_EXTENSIONS":"|bluetooth|robot|","BLOCKLY_LANGUAGE":"en","BLOCKLY_TO_JS":false,"BOARD_JSON_URL":"http://www.espruino.com/json","CODE":"console.log(\"hello\");\n\n","COMPILATION":true,"COMPILATION_URL":"http://www.espruino.com:32766","ENABLE_Testing":false,"ENV_ON_CONNECT":true,"FONT_SIZE":12,"MINIFICATION_DeadCode":true,"MINIFICATION_LEVEL":"SIMPLE_OPTIMIZATIONS","MINIFICATION_Literal":true,"MINIFICATION_Mangle":true,"MINIFICATION_Unreachable":true,"MINIFICATION_Unused":true,"MODULE_AS_FUNCTION":true,"MODULE_EXTENSIONS":".min.js|.js","MODULE_MINIFICATION_LEVEL":"SIMPLE_OPTIMIZATIONS","MODULE_PROXY_ENABLED":false,"MODULE_PROXY_PORT":"","MODULE_PROXY_URL":"","MODULE_URL":"https://www.espruino.com/modules","NPM_MODULES":false,"OFFLINE_DATA":"","OFFLINE_DATA_DOWNLOAD":"","OFFLINE_DATA_UPLOAD":"","OFFLINE_ENABLED":false,"RESET_BEFORE_SEND":true,"SAVE_ON_SEND":"1","SERIAL_AUDIO":"0","SERIAL_TCPIP":"","SERIAL_THROTTLE_SEND":false,"SET_TIME_ON_WRITE":false,"SHOW_CODE_LINK_ICON":false,"SHOW_WEBCAM_ICON":0,"STORE_LINE_NUMBERS":true,"Sound_Error":"","Sound_Info":"","Sound_Success":"","Sound_Warning":"","Speak_Error":false,"Speak_Info":false,"Speak_Success":false,"Speak_Warning":false,"UI_MODE":"Normal","WEB_BLUETOOTH":true}
Initialising SettingsConsole
Initialising Utils
Initialising Config
Initialising Notifications
Initialising Status
Initialising App
Initialising File
Initialising Code
Initialising Serial
  - Initialising Serial Chrome Serial
  - Initialising Serial Chrome Socket
  - Initialising Serial Audio
  - Initialising Serial Web Bluetooth
Initialising Terminal
Initialising CodeWriter
Initialising Modules
Initialising Env
Initialising Flasher
Initialising EditorBlockly
Initialising EditorJavaScript
Initialising Send
Initialising MenuPortSelector
Initialising MenuSettings
Initialising MenuFlasher
Initialising SettingsAbout
Initialising SettingsFlasher
Initialising BoardJSON
Initialising VersionChecker
Initialising Compiler
Initialising Assembler
Initialising GetGitHub
Initialising NPMModules
Initialising SetTime
Initialising Unicode
Initialising Minify
Initialising SaveOnSend
Initialising Tutorial
Initialising Webcam
Initialising FontSize
Initialising UiMode
Initialising URLHandler
Initialising CodeLink
Initialising Project
Initialising Testing
Initialising Notification_Sound
Initialising Tern
Initialising Debugger
Initialising Tour
Initialising SettingsProfile
Initialising HelpLinks
Initialising Offline
Loaded code from storage.
GET chrome.storage.local.OFFLINE_DATA = 0 bytes
>>> Connecting...
Set Slow Write = true
[object Object]
Connected [object Object]
Found a prompt... great!
>>> Sending...
---> "\u0010console.log(\"<\",\"<<\",JSON.stringify(process.env),\">>\",\">\")\n"
>>> Sent
Got "< << {\"VERSION\":\"1v92.902\",\"BUILD_DATE\":\"May 25 2017\",\"BUILD_TIME\":\"14:26:02\",\"GIT_COMMIT\":\"c234c0083c12697aef6cb5b48ad8199e8bdadfb2\",\"BOARD\":\"ESP8266_4MB\",\"CHIP\":\"ESP8266\",\"CHIP_FAMILY\":\"ESP8266\",\"FLASH\":0,\"RAM\":81920,\"SERIAL\":\"60019400-6edd\",\"CONSOLE\":\"Serial1\",\"EXPORTS\":{\"jsvLock\":1075850808,\"jsvLockAgainSafe\":1075850968,\"jsvUnLock\":1075852352,\"jsvSkipName\":1075862264,\"jsvMathsOp\":1075868128,\"jsvMathsOpSkipNames\":1075869824,\"jsvNewFromFloat\":1075859684,\"jsvNewFromInteger\":1075859584,\"jsvNewFromString\":1075859284,\"jsvNewFromBool\":1075859652,\"jsvGetFloat\":1075865904,\"jsvGetInteger\":1075861588,\"jsvGetBool\":1075866264,\"jspeiFindInScopes\":1075882632,\"jspReplaceWith\":1075882332,\"jspeFunctionCall\":1075893188,\"jspGetNamedVariable\":1075884476,\"jspGetNamedField\":1075884920,\"jspGetVarNamedField\":1075885052,\"jsvNewWithFlags\":1075859016}} >> >\r\n>"
Loading http://www.espruino.com/json/ESP8266_4MB.json
ERROR: Not Found
Board JSON loaded
Set Slow Write = true
Device found (connectionId=1)
[success] Connected to port /dev/ttyUSB0
>>> Connected to port /dev/ttyUSB0
[notify_info] No error. Minifying 23 bytes to 22 bytes.
Save on send transformed code to: E.setBootCode("console.log(\"hello\");\n");load(); 
Found a prompt... great!
>>> Sending...
---> "\u0010reset();\n\u0010E.setBootCode(\"console.log(\\\"hello\\\");\\n\");load();\n\n"
Splitting at "reset();\n", delay 250
>>> Sent
Disconnect callback...
WARNING: [notify_warn] Disconnected
>>> Disconnected
Splitting at "\u0003", delay 250
>>> Connecting...
Set Slow Write = true
[object Object]
Connected [object Object]
Received a prompt after sending newline... good!
WARNING: No result found - just got "\bUncaught InternalError: Not connected to the internet\r\n>\bUncaught InternalError: Not connected to the internet\r\n>\bUncaught InternalError: Not connected to the internet\r\n>\bUncaught InternalError: Not connected to the internet\r\n>\bUncaught InternalError: Not connected to the internet\r\n>\bUncaught InternalError: Not connected to the internet\

If this is true though it raises another issue. The IDE will freak out with ANY custom board that you make (I have a couple with different JSVars and stuff). Can I put these in a local directory, and where would be the best place?

Thanks again :)
=hfc

[espruino-discuss]

Posted at 2017-05-25 by hungryforcodes

So I guess thats not it, as I made a copy of an ESP8266_4MB.json (based on ESP8266_BOARD.json in espruino.com/json) and put it on one of my servers, and the IDE found it fine and same error occured.

I also reflashed my chips to stock V92 (non 4MB version), in the hopes that would change something, but this didn't work either.

So a bit mysterious. Another ideas of what I could test?

Has anyone else tried this feature on an ESP8266, and what has been the result?

Thanks again :)
=hfc

[gfwilliams]

Posted at 2017-05-26 by @gfwilliams

I have no idea what's up, but it'd appear not to be a Web IDE problem - can you ask on the ESP8266 section of the forum and see if there are any ideas there?

[espruino-discuss]

Posted at 2017-05-27 by hungryforcodes

OK, I'll bug the gang on that section when I have a few more cycles available :)

[espruino-discuss]

Posted at 2017-06-08 by LeoWang

I had same error message "Not connected to the internet" while trying this feature on my esp8266.

[gfwilliams]

Posted at 2017-06-08 by @gfwilliams

Please ask on the ESP8266 section of the forum. That's what it's for.

[espruino-discuss]

Posted at 2017-07-26 by Polypod

Very nice feature, many thanks, the memory saving is huge. Now, how much memory per bytes of code do we need ? My esp8622 says out of memory above 10000 bytes of code.

The discussion on the problem running this feature on ESP8266 is solved here : http://forum.espruino.com/conversations/307523/

[gfwilliams]

Posted at 2017-07-27 by @gfwilliams

It's worth noting that since this post there's also 1v93, with E.setFlags({pretokenise:1}).

While it can't be used with code stored in flash, for the functions stored in RAM it'll make a really big difference

http://forum.espruino.com/conversations/307447

[espruino-discuss]

Posted at 2017-08-20 by Polypod

Is there a way to use this feature and send modules "as functions" to flash one by one instead of all at once ?

The reason I ask is because I get an "out of memory" when sending more than 10k of cumulated modules this way, which is appreciably more than before this feature. When I set "upload as function" and "save on send", in the console I see all modules are sent at once in one line of "setBootCode(code)". Is it possible to call setBootCode multiple times without erasing previous code saved in flash?

I already found how to do something similar on esp822 using Flash module and setMemoryArea() in the other available flash areas (see this post :http://forum.espruino.com/conversations/290975/), but not in the area used by setBootCode. That would be useful for the other boards without additional flash memory.

[gfwilliams]

Posted at 2017-08-21 by @gfwilliams

So this is for ESP8266? On ESP8266 512k builds you only have 12KB flash memory anyway, so it wouldn't gain you a lot! You can always just write a bunch of data somewhere else in flash and then just save some bootloader code like eval(E.memoryArea( addr, len ))): http://www.espruino.com/Reference#l_E_memoryArea

At the moment there isn't anything, but you could potentially write to the saved memory area yourself. It's only a two uint32s with 2 lengths in, then the data, then a 'magic byte' at the end:

https://raw.githubusercontent.com/espruino/Espruino/09cdeafc4447b16c6f1758e0e2c7119a6e60543a/targets/esp32/docs/SaveAndLoad.md

Potentially E.setBootCode could be modified and the IDE could split data up and write it separately though.

[espruino-discuss]

Posted at 2017-08-22 by Polypod

Thanks gordon for the clarification. Yes, I use an ESP8266, I didn't realize it was so little available there. And thanks for the pointer, i will look into reading/writing myself in these areas.