RTOS-SDK, ESP32 and the way forward

[jmattsson]

Edit: the below progress update refers to the dev-rtos branch which targeted the RTOS-SDK and the ESP31B. With the final release of the ESP32, Espressif abandoned the RTOS SDK in favour of their new IoT Development Framework (IDF). While the IDF is vastly superior to the previous SDKs, it does set our porting effort back a fair bit. Progress updates on the IDF/ESP32 port of NodeMCU can be found further down in this discussion.

With the ESP32 release coming up in a few months time, it's time to seriously start thinking about the way forward. I think it's a given that we'd all like to see NodeMCU run on the ESP32 as well. With the ESP32 there is only the RTOS SDK however, which means we really need to consider how to get ourselves switched from the non-OS SDK over to RTOS.

Since $work is rather interested in shifting some of our products over to the ESP32 I've had a bit of time to investigate the effort that will be required in terms of NodeMCU. I've been "spiking" over on the DiUS dev-rtos branch to see what I can get going. Here's the overview so far:

• Make NodeMCU compile with RTOS-SDK headers rather than non-OS SDK headers. There's a bunch of glue in the sdk-overrides/directory which would need cleaning up, but overall this step wasn't too bad - the SDK functions are largely the same.
• Make NodeMCU link with RTOS SDK. This took a bunch of changes, and a couple of functions needed to be stubbed for now.
• Reimplement NodeMCU task interface on top of RTOS tasks. Thanks to Terry's earlier work, this seems to be relatively straight forward. Once it gets run-tested some issues may surface though (cue Jaws music...)
• Complete reimplementation of our exception handler to allow constants in flash. Turns out the RTOS-SDK doesn't use any of the ROM functions for hooking exceptions (probably performance reasons), and the documented method of installing user hooks simply does not exist. Took a fair chunk of work to find a good way to hi-jack the UserExceptionVector, but on the upside it's now also a whole lot faster than the previous one.
• Remove NodeMCU's partial libc implementation. This was conflicting with the SDK's libc and causing complete hangs. On the upside, there now is a real libc available. Almost all the various c_ prefixed functions (and a bunch of os_ prefixed ones) have been consolidated back to standard C library names.
• Fixed SPI flash reading functions. The RTOS-SDK changed the flashchip variable from a pointer to a struct, so our use of it bombed completely...
• Understand why printf() now doesn't work, but ets_printf() does. printf now working, without bounce buffers.
• Get to the Lua prompt being printed. So far, so good.
• Make UART driver RTOS compatible. The UART driver wasn't to blame, my buggy task.c implementation was. Whoopsie.
• Get to a (mostly) working Lua prompt. This would be a major milestone, and hopefully be the starting point for others to join in the effort.
• Deal with timer callbacks executing from a different task context. Considering that pretty much all NodeMCU code is written expecting run-to-completion semantics, switching to a preempting OS framework has huge potential for random lockups and crashes. My current approach is to dedicate a single RTOS task to running all of NodeMCU in, which should hopefully mitigate most issues. To make this happen I think I'll probably need to wrap the timer API to have the actual callbacks posted back to the NodeMCU task for execution. Having done the transition for the tmr module, the whole thing appears to be easier to just change in each place where needed than attempt to wrap everything. Besides, having high-priority timer callbacks might be useful for some drivers.
• Understand which tasks the SDK callbacks execute from, and develop a strategy for dealing with that. Similar to the timers, but a bit more challenging, possibly. Our earlier work in not allowing Lua callbacks to run from SDK callbacks should help here and limit the amount of rework needed. I hope. As expected callbacks seem to be called from various RTOS tasks directly, such as the rtT high priority timer task and the tiT TCP/IP task, not to mention the uiT task which is what user_init() runs in. It will be up to everyone who is taking an SDK callback to either deal with it fully within that callback without referencing data used by other tasks, or copy the necessary information from said callback and relay it back to the main nodemcu RTOS task. Appropriate locking must be done though. I've updated the sntp module as an exercise, and while it grew a little bit it was pretty straight forward. [cue everyone pointing out things now wrong with it...]
• Make output redirection work. Needs a putc handler installed which can queue characters across into the nodemcu task.
• Make silencing of SDK output work. Espressif didn't provide a system_set_os_print() function in the RTOS-SDK and wants you to install a putc handler to suppress everything instead (which is useless, since then we'd have to put a mutex around each printf call if want system_set_os_printf() like functionality). I fixed this by placing all the SDK functions first in irom, and then in the wrapped printf() checking the return address - if it's in the SDK part of irom and we've flagged off SDK prints, then we suppress it. Rather sneaky, but works well and with almost no cost.
• Revisit printf override. The internal print() function takes peculiar arguments, but we now match those to the letter I believe.
• Find out what's using so much stack space. Currently I'm running the NodeMCU task with an unsupported stack size just to prevent everything from falling over due to the stack being smashed. If anyone has any stack analysis tools that could work for the ESP platform, I'd love to hear about them, since -fstack-usage is not available.
• ??? (no profit guaranteed)
• Reimplement the net module (and others) on top of lwIP API, since espconn is only partially supported on the ESP8266 RTOS, and not at all on the ESP32 RTOS. Probably look at including mbedTLS for TLS support.
• Fix whatever other issues and races we encounter. There will be races, I'm sure. There will also be regressions
• Look at upgrading various components and drivers to take better advantage of the RTOS aspects.

If we can get our current NodeMCU to run stable on the ESP8266 with the RTOS SDK, it should be quite easy to get ESP32 support in I believe. If/when I get my hands on ESP32 hardware, I'll have an even better idea.

Oh, and the dev-rtos branch is subject to force-pushing and other unpleasant things, and it is most assuredly not ready for public consumption, but if you want to track my progress you'll see it there.

[TerryE]

@jmattsson I will up the priority on the net over LwIP. I really wanted to go that way anyway, but this was the push that I needed.

About to hop on a Ferry and travelling for the rest of the day, but will post back at the weekend. 😄

[devsaurus]

Took a fair chunk of work to find a good way to hi-jack the UserExceptionVector, but on the upside it's now also a whole lot faster than the previous one.

Is this improvement also applicable to our current firmware?

[jmattsson]

It could probably be moved over to the regular SDK with some care, but I haven't looked at that.

[jmattsson]

Okay, I'm at a Lua prompt now!

NodeMCU 1.5.1 build unspecified powered by Lua 5.1.4 on RTOS-SDK 1.4.0(c599790)
lua: cannot open init.lua
> 
=node.heap()
39240

The printf doesn't seem to support tabs(?!), because everything seems to end up on its own line, but that's for later... I seem to be running into the auto-bauder leaving things at 74880 until I bang characters at it, even when I've initialized the baudrate to 115200. I don't know what's up with that; @pjsg maybe you have some idea?

Is this the point where I suggest we get this branch into the official repo and let everyone loose on it to try to bang it into shape? After we make sure it's hidden in the cloud builder, of course @marcelstoer .

I'm now eagerly looking forward to getting my hands on the ESP32 dev board from @nodemcu! :)

[marcelstoer]

Hurray Johny, sounds exciting!

Is this the point where I suggest we get this branch into the official repo and let everyone loose on it to try to bang it into shape?

Definitely! However, I suggest to track of the challenges you encounter in a separate issue on GitHub. Otherwise, this one will soon become confusing and hard to follow. It would help if we could define new labels ourselves @nodemcu so we could create RTOS SDK, non-OS SDK or ESP32 to distinguish between the issues.

After we make sure it's hidden in the cloud builder, of course @marcelstoer .

I used to pull active branches from GiHub API (really cool API) and then blacklisted some. However, a few months ago I switched to statically define master and dev only. So, no more maintenance overhead and no more fear of things potentially falling over with every new branch.

[pjsg]

On 31/05/2016 00:12, Johny Mattsson wrote:

The |printf| doesn't seem to support tabs(?!), because everything
seems to end up on its own line, but that's for later... I seem to be
running into the auto-bauder leaving things at 74880 until I bang
characters at it, even when I've initialized the baudrate to 115200. I
don't know what's up with that; @pjsg https://github.com/pjsg maybe
you have some idea?

That sounds very odd. I'll take a look if you tell me your branch name.
The autobaud code does not (AFAIK) touch the uart configuration until it
detects the baud rate in use....

[marcelstoer]

Philip it's https://github.com/DiUS/nodemcu-firmware/tree/dev-rtos

[devsaurus]

Almost all the various c_ prefixed functions have been consolidated back to standard C library names.

Changing c_puts() to puts() brought along two regressions.
Functionality-wise, output redirection is not supported any more with node.output(). c_puts() routed the string through output_redirect() in node.c which triggers an optional callback.
On the formatting side it generates additional newlines - libc's puts() unconditionally appends a newline char to the string which c_puts() / output_redirect() didn't do.

This is currently a show stopper for my dev environment with ESPlorer (and potentially other similar tools).

[jmattsson]

Thanks for diagnosing that!

puts() adding a newline is standards compliant, so if we're assuming it doesn't then that's our bug(s). Looking at the output_redirect() I see that it never captured characters printed via putc(), so that redirect was never complete in the first place :(

The old-style redirect we were using won't be possible with the pre-empting RTOS, as we'd at "best" end up running Lua code reentrantly from whichever RTOS tasks are printing. I suspect what we'll need is to install a proper handler via os_install_putc() and queue characters over into the Lua task. Shouldn't be too bad to do - we need to install one of those anyway to be able to implement the system_set_os_print() function anyway.

[jmattsson]

@marcelstoer Thanks, I hadn't realised you switched from black-list to white-list!

I've pushed a dev-rtos branch into the main repo now, and will continue work against that. I consider this a free-for-all branch for the time being - there is so much to test and fix and polish that I'd prefer to risk stepping on toes (and having toes stepped on) than using the PR mutex. I'm contemplating starting a wiki page to document what I learn as a I go - would that be sufficiently useful to be worth the effort?

@devsaurus I believe I've taken care of the newline madness now. I've yet to address the output redirect.

[devsaurus]

Thanks a bunch, Johny! That'll allow me to have some test drive with the new rtos flavour. Output redirection is not that important atm, as long as there's serial comm 😃

As a side note: I found that you need to manually download and extract ESP8266_RTOS_SDK_v1.4.0_16_02_28 (presumably) to rtos-sdk. Also update init data from rtos-sdk/bin/esp_init_data_default.bin - just to be sure.

[jmattsson]

The RTOS SDK is a git submodule so a simple git submodule update --init should do the trick.

[devsaurus]

node.compile() crashes 100% on an integer build with u8g enabled on top of defaults (doesn't happen for float):

> node.compile("telnet.lua")
Fatal exception (28): 
epc1=0x401008a2
epc2=0x00000000
epc3=0x40105adb
epcvaddr=0x4024e717
depc=0x00000000
rtn_add=0x4010089b
<repeating dump>

Also dofile() is broken.
I even got a meaningful error for a small script saying:

"�M�?�"(stack_size = 0,task handle = 3fff3bf0) overflow the heap_size.
"�M�?�"(stack_size = 0,task handle = 3fff3bf0) overflow the heap_size.
 ets Jan  8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x40100000, len 26856, room 16 
tail 8
chksum 0xfe
load 0x3ffe8000, len 2392, room 0 
tail 8
chksum 0xd4
load 0x3ffe8958, len 8, room 0 
tail 8
chksum 0x5c
csum 0x5c

[jmattsson]

Both of those look like they're due to stack overflow (again). This seems like it's going to be our biggest pain point in this transition - we have stuff that uses way more than the 2k heap RTOS wants to let us use. If we say each function call uses ~100 bytes of stack on average, the 2k stack limit should still allow us to go roughly 20 functions deep. I suspect something is placing too large arrays or objects on the stack, in a frequently used code path. Help tracking this down would be much appreciated.

Edit: you can increase the nodemcu RTOS task stack size in user_main.c, but that obviously directly impacts free heap, and I don't yet know why the SDK docs state that 2k is the upper limit since we're already running past that.