Async RTDB write concurrent with multipath RTDB stream read

[mdailey]

Thanks for a really great library @mobizt.

I am confused about the RTDB async write methods.

I have an ESP8266 PID application where I read the setpoint from RTDB along with some other things using the multipath stream setup and concurrently write sensor values to the RTDB using setInt() and setFloat() at a particular frequency.

This works perfectly with setInt() and setFloat(), but if the wifi is sketchy, the synchronous writes can take a long time to complete or timeout, disturbing my control loop. So I would like to use setFloatAsync() and setIntAsync().

I'm not sure I understand the documentation correctly. I separated my readstream and writes to use separate FirebaseData objects. Then, if I just replace setFloat() and setInt() with setFloatAsync() and setIntAsync() in the control loop at the required frequency (every two seconds), the methods return immediately as expected without any error code, but there is no update to the database.

I must be missing something about how to give the async writes time to execute. Do I need to do something more to let them run? My control loop does not have any delays; it just reads sensors, does the PID calculation, then checks if it's time to update the display or the RTDB.

[mobizt]

This example comment stated how about async action

Firebase-ESP-Client/examples/RTDB/FastSend/FastSend.ino

Lines 114 to 116 in cb7ea94

	//The response is ignored in this async function, it may return true as long as the connection is established.
	//The purpose for this async function is to set, push and update data instantly.
	Firebase.RTDB.setIntAsync(&fbdo, "/test/int", count);

It returns true as long as the connection established (socket is stay open or session is alive) then you can continuously write data to database without to wait for response which error will not known.

The operation is not like async TCP client that can handle multiple sockets connection as it does not support in core WiFiClientSecure.

The async function returns immediately after the request sent and the server response is ignored.

If the connection closed due to internet connection issue or timed out, the new connection process takes time due to SSL handshake process as usual.

[mobizt]

To read/store data more quickly, you may consider the following concerns.

The speed of data transfer depends on many factors as this library works with REST API that used http protocol to transport the data. It involved sending the request and receiving the sever response. The request and response consisted of header and payload. The authentication token when is used as the part of payload will be increased the request header size.

If you set the security rules of your RTDB to allow public read and write i.e. .read and .write keys have the value of true, which no auth token is needed you can rapidly read and write database data but it is not supported in this library because of the privacy concerns.

[mdailey]

Thanks @mobizt ! I did read that comment. Also, I'm not sure, but I don't think speed of transfer is a problem. The problem is that the async methods do not update the database at all.

I was able to reproduce the problem with your FastSend example. The main difference is that I am using a service account/oauth. Is that a problem? Please see below.

/* Modified version of Mobizt's FastSend.ino */

#include <ESP8266WiFi.h>
#include <Firebase_ESP_Client.h>

#include <addons/TokenHelper.h>
#include <addons/RTDBHelper.h>

#define WIFI_SSID "my-ssid"
#define WIFI_PASSWORD "my-wpa-password"
#define DATABASE_URL "https://myapp-default-rtdb.asia-southeast1.firebasedatabase.app"

// Firebase objects

FirebaseData fbdo;
FirebaseAuth auth;
FirebaseConfig config;

unsigned long sendDataPrevMillis = 0;
int count = 0;

void setup()
{
  Serial.begin(115200);
  Serial.println();
  Serial.println();

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to Wi-Fi");
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(".");
    delay(300);
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();

  Serial.printf("Firebase Client v%s\n\n", FIREBASE_CLIENT_VERSION);

  // Authenticate using service account

  config.service_account.json.path = "/service-account.json";
  config.service_account.json.storage_type = mem_storage_type_flash;
  auth.token.uid = "";
  config.database_url = DATABASE_URL;
  config.token_status_callback = tokenStatusCallback;
  config.max_token_generation_retry = 5;

  // Connect to Firebase

  Firebase.begin(&config, &auth);
  Firebase.reconnectWiFi(true);
  fbdo.setBSSLBufferSize(512, 2048);
}

void loop()
{
  if (Firebase.ready() && (millis() - sendDataPrevMillis > 15000 || sendDataPrevMillis == 0))
  {
    sendDataPrevMillis = millis();

#define ASYNC
#ifdef ASYNC
    Serial.print("Set int async...");
#else
    Serial.print("Set int sync...");
#endif

    for (size_t i = 0; i < 10; i++)
    {
#ifdef ASYNC
      // Doesn't work
      Firebase.RTDB.setIntAsync(&fbdo, "/test/int", count);
#else
      // Works perfectly
      Firebase.RTDB.setInt(&fbdo, "/test/int", count);
#endif
      count++;
    }
    Serial.println("ok");
  }
}

With setIntAsync(), there is no update to the database at all. With setInt(), the program works perfectly as expected.

[mobizt]

As I said, the async ignored the server response, if the server returns http 200 code with some error message in authentication or invalid or bad request, you will not know this.

You should use non-sync function to see what is return from server using errorReason().

[mdailey]

@mobizt thanks for being patient with me. I do understand that there is no feedback with the async functions. The problem is that I am not getting any error when I use the non-async functions. They work perfectly and the database is updated correctly. I get the same behavior in my program and with your FastSend example. If I change the synchronous version of your FastSend example as follows:

      // Works perfectly
      bool ok = Firebase.RTDB.setInt(&fbdo, "/test/int", count);
      Serial.printf("Return value: %d\n", ok);
      Serial.printf("Current error code: %s\n", fbdo.errorReason().c_str());

Then the output is

...
Return value: 1
Current error code: 
Return value: 1
Current error code: 
...

i.e., the set is success (OK) with no error. I can see the values changing correctly in the Firebase RTDB console. When I switch back to setIntAsync() I don't see any change. I guess I need to go deeper into the async functions' source code to debug this. Or maybe I am missing something simple that I don't understand.

[mobizt]

If you think this is an issue, open the issue and follow the issue templates because I don't have enough info from your system.

[mdailey]

I had no idea at first, but it does look like an issue when I dig deeper. I hacked the async path in handleResponse() to actually wait for the server response code, and it's 404, whereas it's 200 if the async bit is not set.

[mobizt]

The authentication method is not an issue.

The error which I just discovered is the changes in Google server side which breaks the operation of using silent print parameter to return empty response payload and it should return 204 No Content status code but it is currently return 404 Not Found status code which is not valid return.

Now all functions that use this silent print parameter will be affected by this Google issue included updateNodeSilent, all async functions, pathExisted, setBlob, setBlobAsync, setFile, setFileAsync and restore.

[mdailey]

Aha! Is it possible to fix? Or we must use synchronous set.

[mobizt]

I will temporary remove silent print parameter from those functions to fix the issue in the next update, the side effect is the data bandwidth was not reduced as usual.

[mdailey]

Awesome! Thanks for the great help.

[mdailey]

BTW I confirm that removing &print=silent in common.h makes my async writes work. Thanks again @mobizt.