Limited WinX support for luac.cross and spiffsimg

[TerryE]

Missing feature

This is really a corollary to #2292. We need a process for supplying native WinX executables for luac.cross and spiffsimg.

Justification

With the availability of Marcel's Cloud build service, very few ESP Lua developers need to have a NodeMCU firmware build environment. Also whilst both luac.cross and spiffsimgare considered essential for heavyweight Lua developers, most developers seem happy to do without them.

LFS changes all this because you must use an LFS-enabled version of luac.cross to generate your images. You also need spiffsimg if you prepare your own FS images for downloading.

Luckily the Makefiles for both app/lua/lua_cross and tools don't callback to parent makes, and so these can be easily built standalone from the relevant directory. So what we need to do is to document how a WinX user can build these, preferably with free software. Whatever this process is, these makefiles should contain the necessary conditional statement to work in both environments. Perhaps the easiest approach might be to use MinGW or Cygwin, both of which use the GNU toolchain.

Workarounds

I can't do this work because I don't have any Windows platforms (purely Linux (and Android) so I have no need of workarounds. Perhaps another alternative would be for one of the core developers to do a private build and we follow the PHP practice of making a zip available with the current working release of the WinX binaries for these components.

[cwrseck]

I think there's a significant need for luac.cross, which isn't well enough known. I've been working on a fairly small application for temperature monitoring, and despite all my tinkering I rapidly ran out of space to compile it (to bytecode) on the ESP8266. luac.cross, now I've gotten around to using it, has allowed code size to increase considerably and made debugging a lot easier.

A windows version of luac.cross is definitely needed.

Will

[TerryE]

now I've gotten around to using it, has allowed code size to increase considerably and made debugging a lot easier.

@cwrseck, this is partly a result of the history of the use of the Lua stack on NodeMCU. The early versions (before 1.1 IIRC) didn't have a working luac.cross and used the node.compile() function to generate all compiled lua files. I've been working steadily to improved the core over the last few years. I got the traceback working properly, then did he LCD patch with allowed you to strip out the debug info that traceback didn't need, plus used compression to reduced the line info by ~ 10×, so that retaining line info became a sensible option. I also added luac.cross.

Unfortunately the node.compile() function doesn't use the stripdebug options, so using this loses all traceback. What you can do if you don't have luac.cross is to use the string.dump(loadfile(name)) approach is a separate "compile only" boot.

But as far as I am concerned life is too short. My provisioning system handles all of this and does the compiles on the host. I just tweak any files on that modules shadow directory on the host and command the app to reprovision itself. 5 seconds later I am running the new version on the ESP module.

As I said, I only run Linix, but the big problem here that most ESP developers still stick to WinX. I think that they are crazy, but I respect their right to choose crazy. 😄

[karrots]

Windows 10 has this solved ;)

https://docs.microsoft.com/en-us/windows/wsl/install-win10

[TerryE]

@karrots Jooathan, I think that you would be doing your fellow WinX developers a great service if you wrote a simple short How To that took a lua developer step by step through the whole process from installing an Ubuntu subsystem, fetching the repo, and either

• installing build-essential and git, building standalone Xtensa toochain using pfalcon's toochain, pulling or fetching the firmware, configuring the app/include/user*.h files and them making the firmware (including luac.cross and spiffimg, or
• installing build-essential and git, getting and upacking a latest release then making the app/lua_cross and tools directories. Or
• Doing a Windows docker install as per Marcel's NodeMCU Build on Docker

Yes, I could sketch this out, but whoever does this needs to do a full dress rehearsal on a WinX machine to validate the script before issuing this as a PR to link into our doc tree. So I can't do this.

There may be other simpler options that I haven't though of, but given that the only Win$ that I touch nowadays is cleaning and upgrading the PCs of many of all my IT-illiterate friends and family. There are other issues like: how much space does this take up on the system drive, how do you run lua.cross from the Windows command shell, etc.

If we do use docker then AFAIK, the docker container is set up do a fresh build on each execution, and essentially only the bin directory is preserved. This means that the process takes a few mins on a reasonably powerful PC. This is essential for its primary Travis use, but this is a total pain here. It should be able to set a switch so that the entire NodeMCU tree is cashed persistently in the container, so that makes are truly incremental and only take a couple of seconds.

So I suggest that this should belong to you and Marcel .

[cwrseck]

Reference luac.cross:

Unfortunately loadfile() throws the pervasive E:M error (I wish I knew what
that meant) on this file; what's more, it doesn't throw it reliably, so it's
never quite clear whether the code will run or not. I'd guess that to be an
effect of the GC, except that a reset doesn't improve things. Hence my
enthusiasm for luac.cross; so far, code that compiles on Linux will run on
the ESP8266.

dofile() and node.compile() are certainly quick and easy ways to compile small
applications, but before your improvements building large chunks of ESP8266
code could be pretty frustrating, and it might be worthwhile (once the Windows
binaries are available) to recommend luac.cross in the FAQ, with a note that
if it is locally built, it must be with LUA_NUMBER_INTEGRAL undefined.

Will

[TerryE]

The E:M error is thrown by the compiler when it runs out of RAM during the compilation. There is a fairly low compilation unit size, but nothing (other (than being a PITA) to stop you breaking your app into lots of small compilation units. But using luac.cross certainly eases this problem.

With the LFS patch, once deployed, luac.crossis just part of the standard make and doesn't matter if LUA_NUMBER_INTEGRAL is defined or not -- so long as the same flavour is used for both the firmware and the cross compiler.

[karrots]

I don't have a windows machine at home either. I might be able to cook up some instructions using the work machine.

Re Docker for Windows, unfortunately, docker isn't an emulation layer. So you can't move a docker image made for Linux and run it on Windows. That said the Windows Linux subsystem based on the link posted earlier looks like more or less a full install of Ubuntu or another distro of your choice. I haven't tried it yet.

[TerryE]

I don't have a windows machine at home either.

Bugger!. Another real PC user. 😄 When I retired early, I returned all of my work PC kit to HP. The next couple of laptops I bought were dual boot, but the only time I ever seemed to boot into Windows was to do the occasional security upgrade (which always took ages) and so it became just easier to use single boot Ubuntu with headless VMs for playpens.

We could really use a true Windows-only developer to do this learning curve.

[joysfera]

If there are no Windows developers left then why bother with supporting dev tools on Windows? ;-)

[TerryE]

It's like iPad/iPhone vs Android: both platforms have their strength and weaknesses, and in general people end up choosing one or the other depending on what fits them. This isn't a matter of right and wrong, but rather one of personal preference. The same argument applies for OSX vs Linux vs WinX: what an individual developer uses is up to her or him. The underlying issue here is that we have two communities of developers who use or are involved with NodMCU:

1. The application developers who are building IoT projects and products using the NodeMCU platform. I suspect that a large majority of theses use Marcel's Cloud Service to generate their firmware, and a significant majority of the ~200K builds are a straightforward master build using a very limited set of core modules. And most of these are Windows users.
2. The very small number of these developers that do C development and are active Github contributors to the project. IMO, there are no more than 20 such developers at any one time, and these people do their own builds and use Linux-based development platforms. (As I said earlier, I don't even own a Windows platform.)

But if developers in this first category want to use LFS then they either need to have windows-executable NodeMCU luac.cross and spriffsimg binaries, or access to a cloud service where they can upload a zip file of their application and receive a SPIFFS image in returm.

[joysfera]

OK. As @karrots wrote earlier I believe the Ubuntu on Win10 (WSL) is the best way.
The first steps are documented here:
https://tutorials.ubuntu.com/tutorial/tutorial-ubuntu-on-windows

This is documentation from the other vendor:
https://docs.microsoft.com/en-us/windows/wsl/install-win10

Unfortunately, Docker does not run on WSL directly but luckily there are workarounds:
https://stackoverflow.com/questions/48008675/run-docker-on-wsl-windows-subsystem-for-linux-ubuntu
https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
https://medium.com/@sebagomez/installing-the-docker-client-on-ubuntus-windows-subsystem-for-linux-612b392a44c4
or an official and more complicated way with a relay:
https://blogs.msdn.microsoft.com/commandline/2017/12/08/cross-post-wsl-interoperability-with-docker/

The last step would be Marcel's https://hub.docker.com/r/marcelstoer/nodemcu-build/

I would put the documentation together from these tested and proven bits of information instead of writing something from scratch. The closer it gets to the workflow of a Linux user the better it will work in the long run.

A cloud service building SPIFFS images would be far better, though. I bet even Linux/Mac users would be using it (just like they currently use the cloud service for building firmware images).

[joysfera]

I got sidetracked completely, sorry. WSL way will not build native Windows binaries.
Either back to Cygwin/MingW or resort to running Linux binaries on Windows (either using WSL or emulator like Virtual box).
This github issue asks for native binaries so the WSL/virtualization is out of question, I'm afraid.

[TerryE]

Petr, you like me are probably very comfortable with virtualization. (In fact I used to be contributor to Virtualbox and was a moderator on their user forums back when Sun owned them.) but again needing to use this sort of technology is a step too far for many IoT developers. I think that the best compromise wouldbe to get the app/lua/lua_cross and the tools build processes Mingw compatible and to make these binaries available to Windows users.

[joysfera]

OK, until somebody cross compiles the cross compiler: I have just created a Debian package of luac.cross and spiffsimg binaries. Installing it on Windows 10 with WSL is a piece of cake: simply go to Windows Store, buy Ubuntu (for free), it installs automagically within few seconds and since then you can run your trusted Linux binaries in MS Windows. Then you're just two steps away from the Lua cross compiler for NodeMCU in MS-Windows (run these commands in your new Ubuntu shell, of course):

wget https://pstehlik.cz/data/lua-nodemcu.deb
sudo dpkg -i lua-nodemcu.deb

That's all. And if you didn't know that, your Windows C:\ drive is fully accessible in /mnt/c/ path so if you wrote your Hello world Lua program in Notepad.exe and stored it to C:\Temp\hello.lua you can now call (still in the Ubuntu window)

luac.cross /mnt/c/Temp/test.lua

to get your bytecode for NodeMCU. You'll find it in C:\Temp\luac.out file.

In the Debian package I also included manual pages, documentation of spiffsimg and last but not least, step-by-step instructions how to rebuild the binaries (somebody familiar with docker should review it eventually).

Enjoy!

[TerryE]

@joysfera Petr, in fact there are three separate build variants of luac.cross Integer, FP (16byte Tvalues), FP (12byte Tvalues); though I doubt anyone will use the 16byte TValue version soon as it just wastes 25% of your RAM for no gains.

An issue with WSL is that it is only supported on the later Win10 versions, so anyone on Win 8.x or Win 7 can't use it.