server.send() fails to send STRINGS/HTML greater than 6600~ byte long

[horendus]

Basic Infos

Hardware

Hardware: ESP-12MOD
Core Version: 2.3.0

Description

Problem description

Settings in IDE

Module: NODEMCU
Flash Size: 4MB
CPU Frequency: 120Mhz
Flash Mode: GIO
Flash Frequency: 40Mhz
Upload Using: SERIAL
Reset Method: nodemcu

I'm finding that the ESP8266 is unable to send STRINGS (making up a HTML web page) greater than
around 6600 Bytes.

This is the command I am using to send the STRING. server.send ( 200, "text/html", WebPage_HTML())
(WebPage_HTML() Returns the STRING)

I know the function is generating a perfect string because I have checked over the return value in the serial console windows.

Once I trim the STRING/HTML down to less than 6600byte it is received by the web browser however going any higher I have very random results in the form of strange characters showing up in the top left of my browser window and MISSING HTML code (checking webpage SOURCE)

Is this a known limit to how big the string can be for sending to the client machine?**

What is the suggested strategy to get around this and is this a documented problem?

I have tried changing the HTTP_DOWNLOAD_UNIT_SIZE to 8000. This makes the page load VERY FAST but doesn't effect the 6600~byte limit, a nice speed boost though (maybe putting it out of spec with norms/conventions...)

I assumed the the only limit would be the RAM needed for the string manipulations before sending the STRING to the server.send () command however this doesn't appear to be the case as I have plenty of free RAM and im not getting a MCU crash.

Is this a known limitation?

[horendus]

After doing more testing I have found this issue is indeed related to the amount of FREE RAM.

If your ESP drops bellow 7,000KB - 8,000KB~ free ram then you start getting missing/corrupt characters sent randomly to the WIFI.CLIENT.

If you push it to drop down to 0KB you get a system reboot/crash (expected)

Anyone have a theory on why you would get such problems long before actually running out on RAM?
Perhaps when 7,000KB is remaining on the stack you start dipping into the ESP8266s underlying processes pool?

In order to work around the RAM limitations you can do the following

Break up HTML generating functions into smaller sections of around 5,000KB (depending on remaining RAM) and then send using multiple server.sends. There is however a trick to it, related to the content-length part of the header. If you dont override the content length the client will only read the FIRST server.send command and cut of the rest of the data out. This is because the first send command includes the header and only injects THAT STRINGS length into the header, because it doest yet know about the next server.sends you will be doing.

String HTML_PART_1(){
  html = "<SOME HTML>";
  html += "much much more HTML";
  return html;
}

String HTML_PART_2(){
  html = "<SOME HTML>";
  html += "much much more HTML";
  return html;
}

Send the webpage using the following (loose) method

server.setContentLength(HTML_PART_1().length() + HTML_PART_2().length());   (or hardcode the length)
server.send(200,"text/html",HTML_PART_1());       <---Initial send includes the header
server.sendContent((HTML_PART_2());                 <--- Any subsequent sends use this function 

[igrr]

If the string you are sending is 6600 bytes long, TCP stack will need another 6600 bytes to store a copy of the string while the transmission is in progress. So if you go below 7kB of free heap, there will be ~400 bytes available in heap at some point, which may not be enough for some TCP or WiFi stack purposes. Also, if the remaining 7kB are fragmented (i.e. not a contiguous chunk of RAM but a bunch of smaller blocks), then the TCP stack will not be able to allocate the temporary storage used for transmitting. My suggestion would be to

• use PROGMEM and server.send_P to send data from PROGMEM — in this case it doesn't need to be copied to RAM twice, you avoid allocating Strings and all the associated issues.

• if the data is not constant (e.g. you need to perform some replacements in the HTML), then implement a class which has a Stream interface and pass an instance of that class to server.streamFile. Then you can handle readBytes method and read data in chunks when the server library requests it.

[siara-cc]

As of now, ESP8266WebServer library supports chunked encoding and it can be used as follows. This allows ESP8266 to send data size beyond the amount of RAM available, without even implementing Streams:

      server.setContentLength(CONTENT_LENGTH_UNKNOWN);
      server.send ( 200, "text/html", first_part.c_str());
      while (...) {
          resp = "...";
          resp += "...";
          server.sendContent(resp);
      }