IDF web server module #3502

jmattsson · 2022-02-06T03:57:14Z

This PR is for the dev branch rather than for the release branch.
This PR is compliant with the other contributing guidelines as well (if not, please describe why).
I have thoroughly tested my contribution.
The code changes are reflected in the documentation at docs/*.

This PR adds two new modules:

httpd which provides an interface to Espressif's web server that is included in the IDF.
eromfs (Embedded ReadOnly Mountable File Sets) which makes it easy to include files within the firmware itself, allowing for easy OTA upgrades. This is predominantly aimed at including the static web contents when using the httpd module, though other uses are of course also possible.

The httpd module is able to offload all static file serving to the IDF's web server task, for minimum impact to the LVM. Dynamic route handlers are bounced into the LVM task. Chunked body receiving and sending is supported. For details, refer to the documentation.

...since the IDF was telling me the name had changed.

Lua-interface to the standard esp_http_server component.

Also changed arguments to dynamic uri handlers to be more in line with traditional interfaces.

pjsg

Does the espressif server support websockets? I actually use a LUA based httpserver with support for websockets as it enabled some highly interactive use cases....

pjsg · 2022-02-07T00:26:01Z

components/modules/eromfs.c

+
+#pragma pack(push, 1)
+typedef struct {
+  uint8_t  rec_len;


If you moved the rec_len to be just before the name here and below, then you wouldn't need to do that pragma pack, and the unit16/unit32 would be aligned.

Yes, but then I'd still have filler at the end, and I didn't want to waste space needlessly. If you have a volume with a large pack of small icon files the padding adds up.

pjsg · 2022-02-07T00:27:35Z

components/modules/eromfs.c

+         entry_ = (const record_t *)(((char *)entry_) + entry_->rec_len)) \
+    { \
+      uint8_t name_len = entry_->rec_len - sizeof(record_t); \
+      if (strncmp(xname, entry_->name, name_len) == 0) \


This doesn't quite work -- if xname is "foobar" and name is foo (with namelen 3), you will get a match.

Good call-out! There's a length comparison missing there. Will fix.

pjsg · 2022-02-07T00:33:33Z

docs/modules/httpd.md

+```
+
+#### Parameters
+- `route` The route prefix. Typically in the form of \*.ext to serve all files


I don't think that you want the backslash here

Don't I? Without that it'll start an italic section, will it not?

The joy and wonders of loosely defined Markdown 😄 Using a backslash means being on the safe side here as we have to cater for at least two interpreters: GitHub (GFM) and MkDocs. Did you consider putting *.ext in "" in both places?

Dare I ask what having it double-quoted would do to the markdown parsers?

I didn't mention this wrt the parser, sorry. It's just that once it'll read 'the ".ext" extension' and once 'the form of *.ext to serve' (w/o quotes here).

Ah, right. So *.ext is the wildcard form needed to register a handler for all files with the extension .ext. Maybe I should switch to code format and drop the quotes?

pjsg · 2022-02-07T00:37:17Z

docs/modules/httpd.md

+  the body has been reached. May raise an error if reading the body fails
+  for some reason (e.g. timeout, network error).
+
+Note that the provided `req` object is _only valid_ within the scope of this


How does this work with getbody()? Do I have to finish consuming the body in this callback? This might imply that the entire request has been bffered by the espressif server before the callback is invoked.

What actually happens if you do store this object? I hope that it doesn't crash the runtime.....

Yes, you have to finish processing the body within the call-back. This is a restriction of the esp_http_server. Afaik what it does underneath the hood is to close the receive window to the sender to avoid having to buffer excessively.

If you were to call getbody() (or access the headers object) outside the callback, you'll get a Lua error raised telling you off for doing so. Again, this is due to the limitations of the esp_http_server component itself.

jmattsson · 2022-02-15T07:50:44Z

Does the espressif server support websockets? I actually use a LUA based httpserver with support for websockets as it enabled some highly interactive use cases....

There is some websocket support, yes. I haven't tried wiring it up yet as I haven't had a use-case for it. When I looked at it it wasn't entirely straight forward how one would best surface it into Lua.

pjsg · 2022-02-26T16:20:50Z

It appears that the Espressif web server just calls the handler function for each websocket message received and then allows you to transmit a response later. In this way it is very similar to the standard processing except that (I think) the callback can send multiple messages back through the websocket.

The differences are that you have to use httpd_ws_recv_frame and httpd_ws_send_frame when interacting with the websocket.

I think that the LUA api should look like:

The handler function is called with two arguments:

the received frame
A context object (table) with a send method on it that can be used for sending response frames. The user program can also attach state variable to this object.

The received frame will be nil if this is the start of a session.

The context object can be saved until later.

The frame (in each direction) is represented by a table with two keys: type and data. The type is either text or binary. The data is a string. Three additional types are used for control frames -- close, ping, pong.

Receiving a close message means that this is the last message that will be received.

Sending a close message will close the connection.

jmattsson · 2022-03-04T01:15:33Z

@pjsg That sounds pretty reasonable. Did you find any actual API documentation for the Espressif WebSocket API, or just the example?

Did you want to work on that, or did you want me to? I could possibly fit it in, but I couldn't give you a definite time frame. And are you okay with me merging in this PR in the meantime, and treating websocket support as a future enhancement?

pjsg · 2022-03-04T13:18:33Z

Yes. Merge it and file a ticket for the enhancement.

…

On Thu, Mar 3, 2022, 20:15 Johny Mattsson ***@***.***> wrote: @pjsg <https://github.com/pjsg> That sounds pretty reasonable. Did you find any actual API documentation for the Espressif WebSocket API, or just the example? Did you want to work on that, or did you want me to? I could possibly fit it in, but I couldn't give you a definite time frame. And are you okay with me merging in this PR in the meantime, and treating websocket support as a future enhancement? — Reply to this email directly, view it on GitHub <#3502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALQLTLCQ57VZ42X44DNDL3U6FP4DANCNFSM5NUZJXBA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

jmattsson added 7 commits November 13, 2021 13:13

Updated sdkconfig option name.

219afd4

...since the IDF was telling me the name had changed.

Implementation of httpd module.

d39ec3e

Lua-interface to the standard esp_http_server component.

Added chunked body sending to httpd module.

d54359d

Also changed arguments to dynamic uri handlers to be more in line with traditional interfaces.

Completed auto-indexer support in httpd module.

577bdbf

Documented the httpd module.

38f7f71

Added eromfs module.

445281f

Added eromfs suggestion to httpd docs.

4bc2cbe

jmattsson added new module ESP32 labels Feb 6, 2022

jmattsson requested review from marcelstoer and pjsg February 6, 2022 03:57

pjsg reviewed Feb 7, 2022

View reviewed changes

Fix sloppy name comparisons in eromfs lookups.

4d330dd

jmattsson merged commit cb43481 into nodemcu:dev-esp32-idf4 Mar 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IDF web server module #3502

IDF web server module #3502

jmattsson commented Feb 6, 2022 •

edited by marcelstoer

Loading

pjsg left a comment

pjsg Feb 7, 2022

jmattsson Feb 15, 2022

pjsg Feb 7, 2022

jmattsson Feb 15, 2022

pjsg Feb 7, 2022

jmattsson Feb 15, 2022

marcelstoer Feb 15, 2022

jmattsson Feb 15, 2022

marcelstoer Feb 15, 2022

jmattsson Feb 15, 2022

pjsg Feb 7, 2022

jmattsson Feb 15, 2022

jmattsson commented Feb 15, 2022

pjsg commented Feb 26, 2022

jmattsson commented Mar 4, 2022

pjsg commented Mar 4, 2022 via email

IDF web server module #3502

IDF web server module #3502

Conversation

jmattsson commented Feb 6, 2022 • edited by marcelstoer Loading

pjsg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmattsson commented Feb 15, 2022

pjsg commented Feb 26, 2022

jmattsson commented Mar 4, 2022

pjsg commented Mar 4, 2022 via email

jmattsson commented Feb 6, 2022 •

edited by marcelstoer

Loading