[DO NOT MERGE] Rework socket buffer layer

[sandeepmistry]

This change reworks the socket buffer layer such that the receive buffers are dynamically allocated inside per socket instead of relying on WiFiClient, so the buffer does not get reassigned when the copy constructor or assignment is done.

I've also changed the callbacks to be static to allow for some member variables to be private. As well added a per socket parent field to allow for more than one pending connection per server socket.

I've tested the example sketches on a WiFI101 Shield + Uno and MKR1000, things seem good so far but we need more testing.

It think it will resolve #36, #70, #74 and a piece of #72.

cc/ @agdl @facchinm

[RamonRibes]

@sandeepmistry
Please give me instructions about how could i test this patch.
Thanks!

[sandeepmistry]

@RamonRibes, here are some instructions:

1. Download and extract this zip file: https://github.com/sandeepmistry/WiFi101/archive/socket-buffer.zip
2. In <sketchbook>\libraries\WiFi101 you'll have a version of the WiFi101 library from the library manager. On Windows the default sketchbook folder is ~\Documents\Arduino. Move this folder somewhere else for now.
3. Replace the contents of the folder of the previous step with the contents of the extracted zip file. You should still have a <sketchbook>\libraries\WiFi101\WiFi101.h file but a version with my patches.
4. Restart your Arduino IDE
5. Load some sketches and test away :)

[RamonRibes]

Hi, @sandeepmistry,
Test procedure:

1. I followed your instructions, so my wifi101 library contains your patched WiFi101.h.
2. Kept selected MKR1000 board from pull request Add mDNS server to Provisioning example #152.
3. Loaded&compiled SimpleWebServerWifi, with serial monitor opened.
4. Accessing webpage with no issues.
5. Closed serial monitor.
6. On the next access, page delays response. Led delays 5 seconds to change state, and webpage delays 20 seconds to refresh.
7. If i re-open the serial monitor, the webpage response is fast and as expected.
8. If, instead, i go on accessing the webpage with serial monitor closed (each time waiting the webpage to refresh), the response delay increases on each access, until a value of 20 seconds to change led state, and 50 seconds to refresh the webpage.

...so it's a partial success (the board doesn't hangs).

regards.

[sandeepmistry]

@RamonRibes thank you for trying this.

Can you also please run the following tests?

• Comment out all the Serial.println(...); from the sketch, and run again
• Put the Serial.println(...); back in the sketch and run a longer test with the wall charger.

I'll take a look at running the same tests you ran with my Windows 10 VM in the next day or so.

[sandeepmistry]

@RamonRibes

1. I followed your instructions, so my wifi101 library contains your patched WiFi101.h.

Did you copy all the files from the zip over to the WiFi101 folder? It seems like you did, but I would like to confirm.

[RamonRibes]

@sandeepmistry,
-Yes, i replaced the whole Wifi101 directory contents from yours.
-Tested SimpleWebServerWifi with no Serial.begin(..), Serial.println(..), Serial.print(..), Serial.write(..). No issues found in a short test (opening and closing the serial monitor has no results: there's nothing to print).
-Re-loaded SimpleWebServerWifi with all 'Serial...', and tested without opening serial monitor (i think that if you do so, this is the same as feeding the board alone with a mobile charger, isn't it?).
No issues found in the first minutes. I'm going to test along next day.

[sandeepmistry]

@RamonRibes thanks for trying this out.

I'm looking at another change to remove the long delays when the serial monitor is open and closed when the sketch contains Serial.println(...) statements.

(i think that if you do so, this is the same as feeding the board alone with a mobile charger, isn't it?).
Very similar, but the USB stack on the MKR1000 will be activated when connected to a PC.

Btw, how much current does your mobile charger provide? I think it will be either 500mA or 1A, you'll find this labeled on the charger.

Any updates on the long running test?

[RamonRibes]

@sandeepmistry,
My mobile charger's provide from 300 to 700 mA (no differences found powering the board), but got a tablet charger who provides up to 2.1 A. Will test later.
In the other hand, let me inform you that my first long running test was to be stopped due to router problems & reboot. After that, the board stops responding.
My second attempt failed this morning (after the past evening&night), when i found the board non-responding, nor being wifi connected.
At this point, i wonder if it's possible that the root of our problems would be wifi unconnection.
So i decided modify the SimpleWebServerWifi sketch in those ways:

-Eliminate this part (unnecessary in a MKR1000):

// check for the presence of the shield:
 if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    while (true);       // don't continue
 }

-Move the connection part to a new function 'verifyWifiConnection()', who will check in every loop for propper connection.
-Put a counter for the page access (to have any idea about how many accesses the web page served so far).

Here you have the sketch:

/*
  WiFi Web Server LED Blink

  A simple web server that lets you blink an LED via the web.
  This sketch will print the IP address of your WiFi Shield (once connected)
  to the Serial monitor. From there, you can open that address in a web browser
  to turn on and off the LED on pin 6.

  If the IP address of your shield is yourAddress:
  http://yourAddress/H turns the LED on
  http://yourAddress/L turns it off

  This example is written for a network using WPA encryption. For
  WEP or WPA, change the Wifi.begin() call accordingly.

  Circuit:
   WiFi shield attached
   LED attached to pin 6

  created 25 Nov 2012
  by Tom Igoe
*/
#include <SPI.h>
#include <WiFi101.h>

char ssid[] = "abc";      //  your network SSID (name)
char pass[] = "def";   // your network password
int keyIndex = 0;                 // your network key Index number (needed only for WEP)

int accessCount = 0;                      // will count page access.
int status = WL_IDLE_STATUS;
WiFiServer server(80);

void setup() {
  Serial.begin(9600);      // initialize serial communication
  pinMode(6, OUTPUT);      // set the LED pin mode
}

void loop() {
  WiFiClient client;

  verifyWifiConnection();
  client = server.available();   // listen for incoming clients
  if (client) {                             // if you get a client,
    Serial.println("new client");           // print a message out the serial port
    String currentLine = "";                // make a String to hold incoming data from the client
    while (client.connected()) {            // loop while the client's connected
      if (client.available()) {             // if there's bytes to read from the client,
        char c = client.read();             // read a byte, then
        Serial.write(c);                    // print it out the serial monitor
        if (c == '\n') {                    // if the byte is a newline character

          // if the current line is blank, you got two newline characters in a row.
          // that's the end of the client HTTP request, so send a response:
          if (currentLine.length() == 0) {
            // HTTP headers always start with a response code (e.g. HTTP/1.1 200 OK)
            // and a content-type so the client knows what's coming, then a blank line:
            client.println("HTTP/1.1 200 OK");
            client.println("Content-type:text/html");
            client.println();

            // the content of the HTTP response follows the header:
            client.print("Number of accesses so far: ");
            client.print(++accessCount);
            client.print("<br>");
            client.print("Click <a href=\"/H\">here</a> turn the LED on pin 6 on<br>");
            client.print("Click <a href=\"/L\">here</a> turn the LED on pin 6 off<br>");

            // The HTTP response ends with another blank line:
            client.println();
            // break out of the while loop:
            break;
          }
          else {      // if you got a newline, then clear currentLine:
            currentLine = "";
          }
        }
        else if (c != '\r') {    // if you got anything else but a carriage return character,
          currentLine += c;      // add it to the end of the currentLine
        }

        // Check to see if the client request was "GET /H" or "GET /L":
        if (currentLine.endsWith("GET /H")) {
          digitalWrite(6, HIGH);               // GET /H turns the LED on
        }
        if (currentLine.endsWith("GET /L")) {
          digitalWrite(6, LOW);                // GET /L turns the LED off
        }
      }
    }
    // close the connection:
    client.stop();
    Serial.println("client disonnected.");
  }
}

void printWifiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
  // print where to go in a browser:
  Serial.print("To see this page in action, open a browser to http://");
  Serial.println(ip);
}

void verifyWifiConnection() {
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);

    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
    // wait 10 seconds for connection:
    delay(10000);
    printWifiStatus();                        // you're connected now, so print out the status
    server.begin();                           // start the web server on port 80
  }
}

Will let home for weekend. Hope the monday i'll find the board still working :-)
I'll keep you informed...
Regards.

[RamonRibes]

Hi, @sandeepmistry.
After the weekend, yesterday evening i found the MKR board non responding, and also cold (meaning it's not working at all).
I restarted and it worked again. This morning, the firs time i accessed it, the webpage was displayed fast, but when selected to turn the led on, the response delayed about 10 seconds (maybe the board enters itself in a low power mode?). The following accesses&responses were fast.
The page access counter is > 300 so far (each page access really increases the counter 2 times...).
Regards.

[sandeepmistry]

Hi @RamonRibes,

Good idea on the re-connect. One note, I would suggest removing the call to server.begin(); in verifyWifiConnection(), this will actually fail and put the original server socket in a bad state.

Maybe the first delay was due to the board not being connected to the access point at that point in time?

Another note, the board I setup 2 weeks ago with the WiFi101 v0.9.1 is still running the web server fine. I'm using an Apple AirPort Express as the AP.

[RamonRibes]

@sandeepmistry,
It's a problem if you couldn't test in a W10 system by yourself, but you'll be receiving all the help i could bring.
The long test board started yesterday (about 24 hours) is still working fine. We got about 460 accesses and about 24 hours of connection.
I've added #74, and started new test, on a new board, on the following sketch (is as the previous one, but removing server.begin(); in verifyWifiConnection(), as you suggested, and adding a wifi re-connection counter):

/*
  WiFi Web Server LED Blink

  A simple web server that lets you blink an LED via the web.
  This sketch will print the IP address of your WiFi Shield (once connected)
  to the Serial monitor. From there, you can open that address in a web browser
  to turn on and off the LED on pin 6.

  If the IP address of your shield is yourAddress:
  http://yourAddress/H turns the LED on
  http://yourAddress/L turns it off

  This example is written for a network using WPA encryption. For
  WEP or WPA, change the Wifi.begin() call accordingly.

  Circuit:
   WiFi shield attached
   LED attached to pin 6

  created 25 Nov 2012
  by Tom Igoe
*/
#include <SPI.h>
#include <WiFi101.h>

char ssid[] = "WLAN_7ED1";      //  your network SSID (name)
char pass[] = "arUtIjXdz8Wqc08byhV8";   // your network password
int keyIndex = 0;                 // your network key Index number (needed only for WEP)

int accessCount = 0;                      // will count page access.
int connectionCount = 0;                  // will count wifi re-connections.
int status = WL_IDLE_STATUS;
WiFiServer server(80);

void setup() {
  Serial.begin(9600);      // initialize serial communication
  pinMode(6, OUTPUT);      // set the LED pin mode
  verifyWifiConnection();
  server.begin();                           // start the web server on port 80
}

void loop() {
  WiFiClient client;

  verifyWifiConnection();
  client = server.available();   // listen for incoming clients
  if (client) {                             // if you get a client,
    Serial.println("new client");           // print a message out the serial port
    String currentLine = "";                // make a String to hold incoming data from the client
    while (client.connected()) {            // loop while the client's connected
      if (client.available()) {             // if there's bytes to read from the client,
        char c = client.read();             // read a byte, then
        Serial.write(c);                    // print it out the serial monitor
        if (c == '\n') {                    // if the byte is a newline character

          // if the current line is blank, you got two newline characters in a row.
          // that's the end of the client HTTP request, so send a response:
          if (currentLine.length() == 0) {
            // HTTP headers always start with a response code (e.g. HTTP/1.1 200 OK)
            // and a content-type so the client knows what's coming, then a blank line:
            client.println("HTTP/1.1 200 OK");
            client.println("Content-type:text/html");
            client.println();

            // the content of the HTTP response follows the header:
            client.print("Number of accesses so far: ");
            client.print(++accessCount);
            client.print("<br>");
            client.print("Number of wifi connections so far: ");
            client.print(connectionCount);
            client.print("<br>");
            client.print("Click <a href=\"/H\">here</a> turn the LED on pin 6 on<br>");
            client.print("Click <a href=\"/L\">here</a> turn the LED on pin 6 off<br>");

            // The HTTP response ends with another blank line:
            client.println();
            // break out of the while loop:
            break;
          }
          else {      // if you got a newline, then clear currentLine:
            currentLine = "";
          }
        }
        else if (c != '\r') {    // if you got anything else but a carriage return character,
          currentLine += c;      // add it to the end of the currentLine
        }

        // Check to see if the client request was "GET /H" or "GET /L":
        if (currentLine.endsWith("GET /H")) {
          digitalWrite(6, HIGH);               // GET /H turns the LED on
        }
        if (currentLine.endsWith("GET /L")) {
          digitalWrite(6, LOW);                // GET /L turns the LED off
        }
      }
    }
    // close the connection:
    client.stop();
    Serial.println("client disonnected.");
  }
}

void printWifiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
  // print where to go in a browser:
  Serial.print("To see this page in action, open a browser to http://");
  Serial.println(ip);
}

void verifyWifiConnection() {
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);

    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
    // wait 10 seconds for connection:
    delay(10000);
    ++connectionCount;
    printWifiStatus();                        // you're connected now, so print out the status
  }
}

well, started the second test right now...

[RamonRibes]

@sandeepmistry
Just to comment state of the 2 long term tests (both are still alive):
-First one got > 560 accesses. Working time about 35 hours (since sunday evening).
-Second one got > 160 accesses. Working time about 12 hours (since monday evening). 1 wifi connection.
I've noticed again a delay in each first access to the cards (after a long period of inactivity): When i typed the board's address in the web browser, the server page displays fast but, when i select to turn on the led, it turns on first, but the web browser keeps waiting about 10 seconds before refreshing the page displayed.
This behaviour is found in both cards, each time after a long unaccessing time.

EDIT: Both tests still alive and working fine.
First one: 800 accesses. 3 days working.
Second one: 400 accesses, 2 days working, 1 wifi connection.

Does it mean the end of the problematic behaviour?? ...Let's hope yes!

[RamonRibes]

Well, bad news...
After the good results in the previous tests, i disconnected the boards and started a new test in other PC, adding patches #77 and #154. This test failed after few hours, so i modified the sketch a bit more to catch the point where the program fails.
-Added client.flush() before each client.stop(), to prevent unreaded entries.
-Added a Serial.print("·") on each loop() function pass (useful to see if program is stopped or it is stuck in a part).
The sketch follows:

/*
  WiFi Web Server LED Blink

  A simple web server that lets you blink an LED via the web.
  This sketch will print the IP address of your WiFi Shield (once connected)
  to the Serial monitor. From there, you can open that address in a web browser
  to turn on and off the LED on pin 6.

  If the IP address of your shield is yourAddress:
  http://yourAddress/H turns the LED on
  http://yourAddress/L turns it off

  This example is written for a network using WPA encryption. For
  WEP or WPA, change the Wifi.begin() call accordingly.

  Circuit:
   WiFi shield attached
   LED attached to pin 6

  created 25 Nov 2012
  by Tom Igoe
*/
#include <SPI.h>
#include <WiFi101.h>

char ssid[] = "pepephone_ADSL2GWA";      //  your network SSID (name)
char pass[] = "WNQVU58TTSRJQBT";   // your network password

int accessCount = 0;                      // will count page access.
int connectionCount = 0;                  // will count wifi re-connections.
byte lineCounter = 0;                     // Used to format serial output.
int status = WL_IDLE_STATUS;
WiFiServer server(80);

void setup() {
  Serial.begin(9600);      // initialize serial communication
  pinMode(6, OUTPUT);      // set the LED pin mode
  verifyWifiConnection();
  server.begin();                           // start the web server on port 80
}

void loop() {
  WiFiClient client;

  verifyWifiConnection();
  if (++lineCounter >= 100) {
    lineCounter = 0;
    Serial.println(".");
  }
  else {
    Serial.print(".");
  }
  client = server.available();   // listen for incoming clients
  if (client) {                             // if you get a client,
    Serial.println();
    Serial.println("new client");             // print a message out the serial port
    String currentLine = "";                  // make a String to hold incoming data from the client
    while (client.connected()) {              // loop while the client's connected
      if (client.available()) {               // if there's bytes to read from the client,
        char c = client.read();               // read a byte, then
        Serial.write(c);                      // print it out the serial monitor
        if (c == '\n') {                      // if the byte is a newline character

          // if the current line is blank, you got two newline characters in a row.
          // that's the end of the client HTTP request, so send a response:
          if (currentLine.length() == 0) {
            // HTTP headers always start with a response code (e.g. HTTP/1.1 200 OK)
            // and a content-type so the client knows what's coming, then a blank line:
            client.println("HTTP/1.1 200 OK");
            client.println("Content-type:text/html");
            client.println();

            // the content of the HTTP response follows the header:
            client.print("Number of accesses so far: ");
            client.print(++accessCount);
            client.print("<br>");
            client.print("Number of wifi connections so far: ");
            client.print(connectionCount);
            client.print("<br>");
            client.print("Click <a href=\"/H\">here</a> turn the LED on pin 6 on<br>");
            client.print("Click <a href=\"/L\">here</a> turn the LED on pin 6 off<br>");

            // The HTTP response ends with another blank line:
            client.println();
            // break out of the while loop:
            break;
          }
          else {      // if you got a newline, then clear currentLine:
            currentLine = "";
          }
        }
        else if (c != '\r') {    // if you got anything else but a carriage return character,
          currentLine += c;      // add it to the end of the currentLine
        }

        // Check to see if the client request was "GET /H" or "GET /L":
        if (currentLine.endsWith("GET /H")) {
          digitalWrite(6, HIGH);               // GET /H turns the LED on
        }
        if (currentLine.endsWith("GET /L")) {
          digitalWrite(6, LOW);                // GET /L turns the LED off
        }
      }
    }
    client.flush();
    // close the connection:
    client.stop();
    Serial.println("client disonnected.");
  }
  delay(100);
}

void printWifiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
  // print where to go in a browser:
  Serial.print("To see this page in action, open a browser to http://");
  Serial.println(ip);
}

void verifyWifiConnection() {
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);

    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
    // wait 10 seconds for connection:
    delay(10000);
    ++connectionCount;
    printWifiStatus();                        // you're connected now, so print out the status
  }
}

Well, after a series of tests, it seems the problem is in the sentence client = server.available();, because when a fail occurs, the sketch goes on running through loop(), but don't catches the client's requests (don't enters into the if (client) { sentence).
Another attempt i've made is trying the 'Arduino Create' option (use of the web editor to compile & load the sketch). This attempt was made in the hope it overlays the Windows-10 related problems. This try also ended in failure, at the same point described.
So, at this moment, i'm very dissapointed. The worst thing is i don't now if it's my mistake, board error, library bug or operating system issue (or a combination of all....).
I'm thinking about buying some ESP8266 to do some tests, but don't like starting from nothing again.
Regards.

[sandeepmistry]

Hi @RamonRibes,

I just noticed you are using status != WL_CONNECTED inside verifyWifiConnection. This should actually be changed to while ( WiFi.status() != WL_CONNECTED) { because the status variable will not update.

[RamonRibes]

Hi, @sandeepmistry, thank you for your comment. The bug is now fixed in this way:

void verifyWifiConnection() {
  // attempt to connect to Wifi network:
  while ( WiFi.status() != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    if (WiFi.begin(ssid, pass) == WL_CONNECTED) {
      ++connectionCount;
      printWifiStatus();                        // you're connected now, so print out the status
    }
    // wait 10 seconds for connection:
    delay(10000);
  }
}

I'll comment the test results later.

[q2dg]

@RamonRibes Hello. Any (good) news? Thanks

[RamonRibes]

Hi, @q2dg,
No news at this point.
I use 2 Win10 PC on different locations, and the results are very different:
-On the first location, the sketch fails almost immediatelly, on my 2 boards. The failure point seems to be the boards don't acknowledge the client's request (acServerClient = server.available();), and don't enters inside the sentence if (acServerClient) {.
-On the second location, the same sketch (just changing SSID&Password) it's been working for days.
So, at this point, i'm not able to affirm if the problem is on me (probably is!), on the board (i got 8 MKR1000!), on the libraries, on the OS (Win10)...
The only thing i'm sure is the fact that this board it's not reliable for me, at this moment.
So i just ordered 2 nodemcu boards (ESP8266-12E), to start playing with, in the hope that at least, i could succeed having a reliable system.
Regards.

[trlafleur]

In my testing with the wifi101 stack and wifi modem, I can't keep a simple udp connection up for more that a day or two...

Similar problem as you're having

~~ _/) ~~~~ _/) ~~~~ _/) ~~~~ _/) ~~

Tom Lafleur

On Aug 4, 2016, at 3:26 PM, Ramon Ribes notifications@github.com wrote:

Hi, @q2dg,
No news at this point.
I use 2 Win10 PC on different locations, and the results are very different:
-On the first location, the sketch fails almost immediatelly, on my 2 boards. The failure point seems to be the boards don't acknowledge the client's request (acServerClient = server.available();), and don't enters inside the sentence if (acServerClient) {.
-On the second location, the same sketch (just changing SSID&Password) it's been working for days.
So, at this point, i'm not able to affirm if the problem is on me (probably is!), on the board (i got 8 MKR1000!), on the libraries, on the OS (Win10)...
The only thing i'm sure is the fact that this board it's not reliable for me, at this moment.
So i just ordered 2 nodemcu boards (ESP8266-12E), to start playing with, in the hope that at least, i could succeed having a reliable system.
Regards.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[mamama1]

Hello!
i tried this with issue #89 but actually the rework made it a bit worse:
my, in the issue stated, fix doesn't work anymore. the issue still occurs, but less predictable. however with your buffer rework, when the issue occurs, nothing helps but reset the whlole thing.
i'd be happy to help you tracking that down. were you even able to reproduce my problem? you can reach me on skype IM or email during normal CEST hours. cobtact me, if you wish to work ob this.

[sandeepmistry]

Closing in favour of #204 for now.