Store lua libs as compiled lua bytecode

[trentgill]

using some combination of luac and xxd (aka Hex Dump) (or see this non-xxd version: https://codeplea.com/embedding-files-in-c-programs). This would replace l2h.fnl which is our minimal solution that stores scripts as raw lua wrapped in c strings.

i don't think this will effect RAM utilization, but perhaps luac performs optimizations that loadstring() in the runtime doesn't? both runtime RAM usage & flash storage should be benchmarked before switching.

goals:

• reduce binary filesize
• reduce boot time
• compile-time warnings for core lua code (for free)

requirements:

• should not complicate the build (make) process significantly
• maintain current include syntax for luaenv (may require changes to lib/lualink and lua/bootstrap)

**this is an alternative to #40 **

[csboling]

Investigated this some here, hit a bit of a wall. Lua doesn't bootstrap and running ^^First gives:

> ^^F
lib: size_t size mismatch in precompiled chunk
First.lua: size_t size mismatch in precompiled chunk
Using default script.
Running: First.lua
attempt to call a nil value
stack traceback:

It seems that Lua bytecode is not cross-platform (?!) and this size_t size mismatch error arises from trying to execute in a platform where, presumably, size_t is 32 bits not 64. Others have dealt with this by recompiling luac on the host, changing some defines to match the target's datatypes or so. This seems to me to constitute complicating the build process significantly. Seems kinda surprising that something like this isn't a runtime option for luac.

[trentgill]

from @simonvanderveldt:
"

ed note: This whole process could likely be deprecated if some smart person figures out how we can upload lua bytecode to crow directly, rather than manually convert everything into C strings & load them at runtime. The key benefit here is that lua syntax errors will be caught by that process (using luac i guess), and perhaps smaller RAM footprint (because the C strings aren't loaded into memory at all).

Would that be something like this? https://github.com/nodemcu/nodemcu-firmware/blob/8d091c476edf6ae2977a5f2a74bf5824d07d6183/docs/compiling.md#compiling-lua-on-your-pc-for-uploading
"

from @trentgill:
"
Yes, and specifically the LFS concept (https://github.com/nodemcu/nodemcu-firmware/blob/8d091c476edf6ae2977a5f2a74bf5824d07d6183/docs/lfs.md) is what we really need I think. The issue being that even if we can precompile the lua libraries, they will currently still be copied from flash into RAM, which saves us nothing but startup time.

After a quick read about LFS, it seems like this is a big endeavour, and likely something that requires another developer to make a reality.

The big benefit is that we would free up a huge amount of RAM (presently Lua takes up over 100kB at boot time, and we only have 256kB total, a good chunk of which is devoted to hardware buffers & runtime processing). In practice we only get access to about 50kB of RAM in the VM before crashes or memory errors occur. If someone takes this project on, it would more than double the free memory available to scripts. This would have the added benefit of enabling support for >8kB userscripts (currently limited by the RAM allocation that happens while uploading & compiling on chip).
"

[trentgill]

One option to explore (which doesn't get around the Flash / RAM issue) is to use bin2c. This would allow running luac on the desktop (with a compiler setting for arm32?) and included lua files as lua bytecode wrapped in C.

This might actually increase the size of binary which is the opposite of our goal, but perhaps changes the RAM footprint? Unlikely. But it's only a 1hour adventure to try it out and see the implications.

[trentgill]

Closing this issue because I believe it is not a real limitation we will hit. As I've been extending the lua libs, it seems that we are going to run out of RAM (taken up by the compiled lua) before we run out of flash.

I did some investigation of the dofile approach we use to load C-strings as lua libs and it appears that there is minimal RAM overhead versus just entering code at the command-line. Sure it creates a bunch of garbage to be collected, but after all libs are loaded (and immediately before init()) we collect all garbage.

In essence, the limiting factor is RAM, and it is not affected significantly by the method of loading lua code. Instead I'm opening #390 as it is the realworld bottleneck.