Slow wifi client.print() spi operations

[JoeEarlam]

Hi there, I'm trying to upload data to a server via HTTP post. I'm pre-assembling the data into a single char array and sending that in one go using client.print(&array) using the below code, which works perfectly at very low data rates (one line of ~250 characters every 5 seconds):

//Send data via https POST
void httpsSend() {
  uint16_t numChars;                                      //length of HTTP post content
  static char postData[8192];                             //HTTP post data buffer
  static char headerBuffer[sizeof(postData) + 256];       //HTTP header + post data
  static uint8_t lineCnt;                                 //number of lines to send to server
  uint32_t startMillis = millis();                        //upload timestamp in millis since boot
  uint64_t startTime = timeClient.getEpochTime() * 1000;  //upload timestamp in unix millis

  //iterate through some lines and write them to the HTTP buffer
  lineCnt++;  //increment number of lines to send
  if (lineCnt > 20) lineCnt = 1;
  for (uint8_t line = 0; line < lineCnt; line++) {
    uint64_t timeStamp = startTime + startMillis - millis();  //iteration timestamp is initial unix time plus milliseconds elapsed(ish)
    numChars += sprintf(postData + numChars, "%s,MEAS_SRC=%s ", measName, measSource);

    for (uint8_t i = 0; i < (sizeof(dataBuffer) / sizeof(dataBuffer[0])); i++) {
      if (i > 0) numChars += sprintf(postData + numChars, ",");        //leading comma after first field
      numChars += sprintf(postData + numChars, "%s", fields[i]);       //field tag so influxDB knows what this data point is called
      numChars += sprintf(postData + numChars, "%lu", dataBuffer[i]);  //data as a uint32_t. Note that Arduino's implementation of sprintf does not like float - multiply them suitably to preserve precision and transmit as uint32_t (e.g 1.23 becomes 123)
    }
    
    numChars += sprintf(postData + numChars, " %llu\n", timeStamp);  //append unix timestamp
    delay(2);                                                        //delay so every line has a different timestamp
  }

  //assemble our POST packet into the header buffer to send to the server
  sprintf(headerBuffer, "POST /influxDB_endpoint/?precision=ms HTTP/1.1\n"   //second precision, this can be set to milliseconds but that requires millisecond unix time
                        "Host: example.com\n"                        //address of server to upload to
                        "Accept: */*\n"                                      //content type supoprted by the client (all)
                        "Connection: keep-alive\n"                           //keep connection alive
                        "Content-Type: application/x-www-form-urlencoded\n"  //content type for postdata
                        "Content-Length: %d\n"                               //number of characters in postdata
                        "Authorization: Basic %s\n\n"                        //credentials
                        "%s\n",                                              //postdata
          numChars, credentials, postData);

  startMillis = millis();
  client.print(headerBuffer);  //send data to server
  Serial.printf("%d chars send time: %dmS\n", numChars, millis() - startMillis);

  //start checking for received data
  recvTask.setIterations(10);
  recvTask.restart();
}

I connect to the server using the following routine:

boolean httpsConnect() {
  client.setInsecure();
  client.setTimeout(3000);

  //Attempt to connect with five retry attempts
  //return true if connected, false if not
  int r = 0;  //retry counter
  while ((!client.connect("example.com", 443)) && (r < 5)) {
    delay(100);
    r++;
  }
  if (r == 5) {
    Serial.println("HTTPS connection failed");
    return false;
  } else {
    Serial.println("HTTPS connection successful");
    return true;
  }
}

However, when I run the code five times and average the time taken to send each packet I get the following result:

Chars sent | Mean Time (mS) | Rate (KB/s)
173 | 29.4 | 5.9
346 | 97 | 3.6
519 | 98.2 | 5.3
692 | 92.4 | 7.5
865 | 164.2 | 5.3
1038 | 186.6 | 5.6
1211 | 174.4 | 6.9
1384 | 258 | 5.4
1557 | 259.2 | 6.0
1730 | 291.4 | 5.9
1903 | 509.2 | 3.7
2076 | 568.8 | 3.6
2249 | 317.2 | 7.1
2422 | 475.2 | 5.1
2595 | 951.2 | 2.7
2768 | 679.2 | 4.1
2941 | 613.8 | 4.8
3114 | 442.6 | 7.0
3287 | 472.6 | 7.0
3460 | 631.4 | 5.5

If I hook up a scope to the wifi chip's clock pin (R6) I see the following trace:

You can see that after every SPI transaction there's an almost immediate reply from the wifi chip... Then 60ms where the device does nothing. This seems like an excessive delay to me, but I can't track down where it's coming from in the source. How can I optimise my HTTP code to increase throughput?

Thanks!

[earlephilhower]

You don't have the full source, so it's not easy to say, but I have an idea looking at what's here.

It looks like you are running a new HTTPS connection for each message. HTTPS requires a SSL handshake which is very slow on these embedded systems. There is no specialized encryption or certificate hardware so it takes a lot of cycles just to do the initial TLS handshake.

Maybe you should consider WebSockets? https://github.com/Links2004/arduinoWebSockets . That keeps the connection open and lets you do your comms w/o requiring a handshake every message. The actual encrypt/decrypt isn't too bad, only the TLS handshaking, so that might give massive improvements.

Alternatively look into TLS sessions. Basically they let you short-circuit the TLS handshake by storing the last parameters.

[JoeEarlam]

Thanks for the reply. I'm actually not closing the connection with client.close() or flush() in the receive operation (just checking the response buffer), is it being closed automatically? I will have a play with those solutions and see what I can come up with.

[earlephilhower]

Sorry, those were my best guesses w/o a complete MCVE . If you can reproduce this problem with a 1-screen MCVE I might be able to give some other suggestions. But I obviously won't be able to run your whole app or have your other infrastructure/hw/etc.