improve 'l2h' lua to c-header converter script

[trentgill]

ed note: This whole process could likely be deprecated if some smart person figures out how we can upload lua bytecode to crow directly, rather than manually convert everything into C strings & load them at runtime. The key benefit here is that lua syntax errors will be caught by that process (using luac i guess), and perhaps smaller RAM footprint (because the C strings aren't loaded into memory at all).

I'm unclear on the memory implications of the current approach - Some help profiling the memory usage would likely be a good use of time.

/////////////////////////////////////////////////////

nb: l2h is written using 'fennel', which is a lisp syntax for the lua language. if you want to rewrite it in plain lua, that's ok, but you'll need to update the Makefile to stop it from overwriting your work!

no attempt is made to validate the crow standard libs before 'make' creates binaries. while it's not the role of l2h to do full testing of these libraries, it is useful to have l2h validate the code as at least able to be parsed. all that should be necessary is to use loadfile() on the .lua file and return an error if it fails to parse.

• return with error if lua code has syntax errors

currently the lua files are simply copied, line-by-line, into a c-header file. in particular this means that comments & whitespace are taking up space in flash, and must be parsed out by the lua interpreter at runtime. importantly, once the lua code is flashed onto the chip there is no way to retrieve it, so it doesn't need to stay human readable.

we can reduce flash usage, and RAM overhead at load time the following ways. top of the list is easiest & biggest impact, while end of list is either difficult or has only a small benefit:

• remove comment lines
• remove empty lines
• strip in-line comments
• strip leading whitespace per line
• don't add a newline character at end of each line unless necessary (is it ever?)
• remove all other non-functional whitespace
• rename local arguments with known scope to single-character names
• precalculate known constants
  note! most of the above is done by the lua interpreter / compiler (ie luac), so perhaps a more lucrative use of time is to find a way to upload precompiled lua bytecode, rather than raw lua strings. if taking this approach, we should make sure to compare binary sizes as bytecode can apparently be larger than textual representation.

currently the crow standard library enforces using the single-quote ' character for indicating strings or chars, or the double-brak [[ and ]] form (though this is rarely used). when these libraries are wrapped into c-headers with l2h each line is wrapped in a " so the c-compiler can parse them as strings. this means that if a user writes lua code that uses the " to denote a string, the resulting c-header file will not be correctly formed. a big step in the right direction would simply be to parse for the " character in the lua source, and add a leading \ to escape it in C.

• escape doublequote " characters in lua source

[simonvanderveldt]

ed note: This whole process could likely be deprecated if some smart person figures out how we can upload lua bytecode to crow directly, rather than manually convert everything into C strings & load them at runtime. The key benefit here is that lua syntax errors will be caught by that process (using luac i guess), and perhaps smaller RAM footprint (because the C strings aren't loaded into memory at all).

Would that be something like this? https://github.com/nodemcu/nodemcu-firmware/blob/8d091c476edf6ae2977a5f2a74bf5824d07d6183/docs/compiling.md#compiling-lua-on-your-pc-for-uploading

[trentgill]

Yes, and specifically the LFS concept (https://github.com/nodemcu/nodemcu-firmware/blob/8d091c476edf6ae2977a5f2a74bf5824d07d6183/docs/lfs.md) is what we really need I think. The issue being that even if we can precompile the lua libraries, they will currently still be copied from flash into RAM, which saves us nothing but startup time.

After a quick read about LFS, it seems like this is a big endeavour, and likely something that requires another developer to make a reality.

The big benefit is that we would free up a huge amount of RAM (presently Lua takes up over 100kB at boot time, and we only have 256kB total, a good chunk of which is devoted to hardware buffers & runtime processing). In practice we only get access to about 50kB of RAM in the VM before crashes or memory errors occur. If someone takes this project on, it would more than double the free memory available to scripts. This would have the added benefit of enabling support for >8kB userscripts (currently limited by the RAM allocation that happens while uploading & compiling on chip).

[trentgill]

This conversation seems to belong at #122 rather than here, though even that issue is not quite clear about what needs to occur or what the motivation is. New issue altogether?

[simonvanderveldt]

Yeah, might be better in #122 or as a new issue, although I guess the question remains if anyone would have time to work on it. If we know upfront that won't happen then maybe it's not worth the effort of creating an issue for it.

The items remaining in this issue seem minor, especially since comments are already stripped.

[trentgill]

Closed in favour of #122 which i'd like to keep open as it will become necessary at some point if the libraries continue to grow in size.