Client disconnects from server when replying with large HTTP body

[kaiiorg]

I'm working on a project that requires that I serve an Angular web application so the user can use the Arduino Nano 33 IoT based device in a stand alone mode. Anyone who's done anything with Angular knows that its files quickly get larger than I could ever hope to fit in the Nano's 256KB of flash:

My solution to this is to serve the files from the SD card I'm already using for configuration and general purpose storage via greiman/SdFat. Since the Nano is also rather limited in RAM, I can't read it all into memory at once. I also can't just read one byte at a time, because it'd take forever to transfer a file. So, I have to buffer chunks of data at a time. Fortunately, both the SdFat library and the WiFiNINA library make this fairly easy to do, so it actually wasn't too hard to implement.

My problem is that the client occasionally gets disconnected (client.connected() returns false) before the entire file is finished transferring. I haven't seen this happen on smaller files, but it appears that it happens at random and is more likely to occur on larger files. Even those even as small as the 266KB picture of my cat occasionally do it:

I've confirmed that this occurs even after updating the NINA module to 1.4.1, moving to different networks with different access points, and when it is put in AP mode and directly connected to. I'm not sure if I'm doing something wrong or if there's something funny going on with the library.

I've managed to reproduce the problem by modifying the existing WiFiWebServer example to read from SD cards: kaiiorg/WiFiNINA-SD-Webserver. The hardware I'm using uses a different SD select pin, so you'll probably need to change SD_SELECT_PIN. cat.jpg should go in the SD's root folder. It might take a few tries for it to occur.

[kaiiorg]

Took a capture in wireshark and it looks like the nano is setting the FIN flag before it is finished.

capture.zip

[lasselukkari]

Did you figure out a solution for this. We seem to be facing the same problem.

[lasselukkari]

This behaviour can be reproduced without the SD card.

Here is a sketch that has been modified from the WiFiNINA WiFiWebServer example.

#include <SPI.h>
#include <WiFiNINA.h>

#define WIFI_SSID ""
#define WIFI_PASSWORD ""

WiFiServer server(80);

void setup() {
  Serial.begin(9600);

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println(WiFi.localIP());

  server.begin();
}

void loop() {
  // listen for incoming clients
  WiFiClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/plain");
          client.println("Content-Length: 100000");
          client.println("Connection: close");  // the connection will be closed after completion of the response
          client.println();
          for (int i = 0; i < 100; i++) {
            client.print(
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
              "****************************************************************************************************"
            );
          }

          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }


    if (client.getWriteError()) {
      Serial.println("Client write error detected");
    }
    
    // give the web browser time to receive the data
    delay(1);
    client.stop();
  }
}

Reload the page a few times with a browser and it will fail randomly.

The error state is detected by the library. When the writing fails the client.getWriteError() returns an error.

[ocrdu]

@lasselukkari: Just for trouble shooting: What happens when you remove the Content-Length line from the sent header? Most browsers will accept that when Connection: close is set.

[lasselukkari]

There is no big difference really. When there is no content length header the browser does not know it has not received the whole content and just displays the partial response.

The HTTP protocol is irrelevant detail here. In a nutshell the problem is that the client.write function becomes unreliable when called frequently with a 1kB data. When I increased the chunk to be 10kB the data received by the browser becomes complete garbage. I have not tested does this only affect the clients returned from a server instance.

The write error is detected by the library. After the write has failed the getWriteError() function return an error code. I would start by investigating where this error is coming from.

This problem actually popped up in another repo when a user of my own library reported problems with the MKR WiFi 1010 board. I bought one these boards to debug the problem and found this.

[lasselukkari]

I think the problem is that the version of WifiClient used in the nina-fm has a bug. It is not handling partial sends at all in write function here like they do it here . The code in the nina-fm expects that the lwip_send_r function always writes everything when in reality it may just write it partially. Unless the the functionality of the lwip_send_r has not changed between the versions I do not see how this could work. Most likely the issue should be moved to the nina-fm repo.

[kaiiorg]

Did you figure out a solution for this. We seem to be facing the same problem.

I never did; I ended up switching to an esp32 since I wasn't far into development. Sorry for not responding in a reasonable time, I had notifications turned off for this issue for some reason and was busy due to family problems.