/rfid endpoint - Serve with chunked response #280

tueddy · 2023-12-12T10:51:23Z

/rfid endpoint:

A static buffer of 8KB is currently used to list the RFID tags. If this buffer is full, the returned JSON is truncated and not all entries are displayed in the web interface. . This happens with me from approx. 60 entries (depends on path length). It could also lead to a low memory situation and, in the worst case, to a crash.

With this PR, a list is first created holding the keys only and later the details such as path and game mode are sent as a chunked response with a smaller buffer.
No changes in the delivered JSON except that it is always complete

SZenglein · 2023-12-12T19:31:14Z

src/Web.cpp

+}
+
+static String tagIdToJsonStr(const char *key) {
+	StaticJsonDocument<512> doc;


Isn't 512 bytes a bit large for a single tag id?

A single tag id looks like:

{ "id": "003108198106", "fileOrUrl": "/Die Schönsten Lieder Zum Einschlafen", "playMode": 3, "lastPlayPos": 1842837, "trackLastPlayed": 0 }

maxLen of "fileOrUrl" is MAX_FILEPATH_LENTGH = 256, size calculation with https://arduinojson.org/v6/assistant gives about 350 Bytes needed. Recommended is power of 2, so 512 Bytes is best size.

SZenglein · 2023-12-12T19:38:14Z

src/Web.cpp

 	}
+	static std::vector<String> nvsKeys {};


This vector will never shrink, just something to be aware of. Though it's good to avoid frequent allocations...

nvsKeys is declared as local variable and i assume it gets out of scope after function handleGetRFIDRequest() finished.
Am i wrong here?

It's declared as static, meaning it lives as long as the entire program.
If you didn't add nvsKeys.clear() directly afterwards, I it should continue to grow with each call.

The upside is that push_back does not need to reserve more memory every time and instead can re-use the same heap space. The downside is that the heap space that is the capacity of the vector is never reduced or freed.

SZenglein · 2023-12-12T19:50:53Z

src/Web.cpp

+			if (nvsIndex == 0) {
+				// start, write first tag
+				json = tagIdToJsonStr(nvsKeys[nvsIndex].c_str());
+				len += sprintf(((char *) buffer), "[%s", json.c_str());


What if maxlen < json len? Can we handle that case? We should make sure to never write more bytes to buffer than allowed.

Note: put the first tag also in the loop, then you won't forget the size check too

Good catch!
I never saw a buffer smaller than 1500 Bytes so JSON length never exceeds.
But i will do an additional check here for code safety.

SZenglein · 2023-12-12T19:54:27Z

src/Web.cpp

+	nvsKeys.clear();
+	// Dumps all RFID-keys from NVS into key array
+	listNVSKeys("rfidTags", &nvsKeys, DumpNvsToArrayCallback);
+	if (nvsKeys.size() == 0) {


Would not need this special handling for size == 0 if in the chunked response there was no special handling for the first tag.

Writing the first tag there is no comma in JSON, it is starting with the second entry.
That's is the reason to handle the special case size==0 here and not even start a chunked response.

Not starting the chunked response in the first place is probably a good idea.

SZenglein · 2023-12-12T20:01:23Z

src/Web.cpp

+			}
+			if (nvsIndex == nvsKeys.size()) {
+				// finish
+				len += sprintf(((char *) buffer + len), "]");


Edge case: that single byte for ']' or ',' could be one byte more than maxLen.

Why not write '[' and ']' by themselves without sprintf?

/rfid returns an array of tag-id keys /rfid/details returns an array of tag-ids and details. Optional GET param "id" to list only a single assignment.

SZenglein · 2023-12-13T21:56:38Z

src/Web.cpp

@@ -457,6 +457,7 @@ void webserverStart(void) {

 		// RFID
 		wServer.on("/rfid", HTTP_GET, handleGetRFIDRequest);
+		wServer.addRewrite(new OneParamRewrite("/rfid/details", "/rfid?details=true"));


OneParamRewrite is usually for path parameters in the form of /path/{some_param} -> /path?some_param={some_param} because AsyncWebServer cannot handle path params natively.

Does this even work and do something? Even if it does, using the "normal" AsyncWebRewrite makes more sense: https://github.com/me-no-dev/ESPAsyncWebServer#param-rewrite-with-matching

Yes it works fine, do you have a concrete improved code here?

Didn't test it, but just replacing 'OneParamRewrite' with 'AsyncWebRewrite' should do the same thing.

SZenglein · 2023-12-13T22:05:54Z

src/Web.cpp

-				json = tagIdToJsonStr(nvsKeys[nvsIndex].c_str());
-				len += sprintf(((char *) buffer), "[%s", json.c_str());
+				json = tagIdToJsonStr(nvsKeys[nvsIndex].c_str(), withDetails);
+				if (json.length() > maxLen) {


You check that, but 4 lines later you're writing json.length() + 1 bytes, so you should also check for json.length() + 1 available space

SZenglein · 2023-12-13T22:08:42Z

src/Web.cpp

 				nvsIndex++;
 			}
 			while (nvsIndex < nvsKeys.size()) {
 				// write tags as long we have enough room
-				json = tagIdToJsonStr(nvsKeys[nvsIndex].c_str());
+				json = tagIdToJsonStr(nvsKeys[nvsIndex].c_str(), withDetails);
 				if ((len + json.length()) > maxLen) {


Same issue, check is for 1 byte too few because of the comma

SZenglein · 2023-12-13T22:11:42Z

src/Web.cpp

 				nvsIndex++;
 			}
 			if (nvsIndex == nvsKeys.size()) {
 				// finish
-				len += sprintf(((char *) buffer + len), "]");
+				len += snprintf(((char *) buffer + len), maxLen, "]");


That byte is written without a length check at all.

SZenglein · 2023-12-13T22:16:39Z

Regarding the buffer copying, it's still calling for memory issues IMHO.
Most importantly, every code like this:

snprintf(((char *) buffer + len), maxLen, "...", ....);

Needs to be converted to

snprintf(((char *) buffer + len), maxLen-len, "...", ...);

that would already prevent any invalid memory access when writing the buffer. If the string doesn't fit, the json would still be invalid though.

endpoint /rfid/ids returns only tag ids buffer check

…mbda return type

rfid chunked response

3e2b0f6

SZenglein reviewed Dec 12, 2023

View reviewed changes

tueddy added 2 commits December 13, 2023 21:25

fixes

788d4ff

/rfid returns an array of tag-id keys /rfid/details returns an array of tag-ids and details. Optional GET param "id" to list only a single assignment.

make clang check happy

0d9c7b3

SZenglein reviewed Dec 13, 2023

View reviewed changes

tueddy added 5 commits December 14, 2023 15:52

endpoint /rfid returns tag details (as before)

b080121

endpoint /rfid/ids returns only tag ids buffer check

clang

86fa4f4

fix error: inconsistent types 'int' and 'unsigned int' deduced for la…

871df02

…mbda return type

nvsIndex is size_t

403d2e5

ids->ids-only

9c3b512

tueddy merged commit a1a35c7 into biologist79:dev Dec 29, 2023
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

/rfid endpoint - Serve with chunked response #280

/rfid endpoint - Serve with chunked response #280

tueddy commented Dec 12, 2023

SZenglein Dec 12, 2023

tueddy Dec 12, 2023

SZenglein Dec 12, 2023

tueddy Dec 12, 2023

SZenglein Dec 12, 2023

SZenglein Dec 12, 2023

SZenglein Dec 12, 2023

tueddy Dec 12, 2023

SZenglein Dec 12, 2023

tueddy Dec 12, 2023

SZenglein Dec 12, 2023

SZenglein Dec 12, 2023

SZenglein Dec 13, 2023

tueddy Dec 13, 2023

SZenglein Dec 13, 2023

SZenglein Dec 13, 2023

SZenglein Dec 13, 2023

SZenglein Dec 13, 2023

SZenglein commented Dec 13, 2023 •

edited

/rfid endpoint - Serve with chunked response #280

/rfid endpoint - Serve with chunked response #280

Conversation

tueddy commented Dec 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SZenglein commented Dec 13, 2023 • edited

SZenglein commented Dec 13, 2023 •

edited