NimBLE + SPIFFS - SPIFFS will stop working over longer periods with fewer writes

[BrugerX]

Board

ESP32-S3

Device Description

NodeMCU v1.1

Hardware Configuration

I have three triggers that determine how my experiment works:

1. Printing_on GPIO: 19
2. Xperiment_on GPIO: 18
3. Restart GPIO: 17

Other than that, I have two LEDS connected:

1. XPERIMENT_ON_LED: 25
2. BLE_FLASH: 32

Version

v2.0.14

IDE Name

PlatformIO with CLion

Operating System

Windows 10

Flash frequency

Unknown, but flashing is nto the issue

PSRAM enabled

yes

Upload speed

115200

Description

I am trying to create a BLE based soil water content datalogger.

Filesystem: I am using a custom-made wrapper class based around the SPIFFS filesystem. I have used it for a previous project on an esp32 with LoRa where it went through exhaustive unit testing.
BLE: I am using NimBLE-Arduino for my BLE stack module.

Expected behaviour:

When the experiment on:

The server is expected to go to sleep for 30 minutes at a time, wake up for a limited amount of time (5 seconds.) The server has a SWC-sensor attached, which it sends to the client.

The client is constantly on, because it is connected to a power outlet. I receives the SWC reading and stores it plus a timestamp, which it generates locally, to a SWC_Value_Array and a Timestamp_Value_Array.

Saving:

The way the saving works currently is for the client to load the whole file into an array in the heap. It then finds the index where it wishes to insert the value. It writes the value and increments a counter, that keeps track of where the next write-index is.

Each value array has a specific index counter.

The timestamps are saved as their bits in 4 unsigned chars. The SWC is saved in the format "XX.XX" as unsigned chars as well. All SWC values are formatted to be exactly 5 chars long. E.g "1.24" -> "01.24", "4.54" -> "04.54".

All arrays are initialized to be filled with "l"'s.

Interrupts:

Xperiment_on: If this is triggered to be true, we start the experiment. If it is true and triggered to be false, we stop scanning and we stop the experiment.

Restart: This will overwrite the value arrays with l's and trigger a system abort.

Print_on: This will start a while loop, which continues to print the value arrays as unsigned chars or longs. There are vTask delays inbetween to ensure no WDT is triggered.

Error:

I can write a lot of values in a row into the SPIFFS. That is, I can write 20 values in a matter of 2 minutes, and nothing bad happens.

However, if I write 7 values over a span of half an hour I get an erroneous output. It will succesfully read the SWC from the client, however the SWC_value_array turns into unintelligble characters. In the specific screenshot the value is written in the middle of the array. But all previous values have been replaced by unintelligble characters. (The first screenshot is the 6th write, the second is the 7th write. I have highlighted the part where the client receives the characteristic value of the SWC characteristic, and where it can be found in the value_array)

Sketch

1. Start NimBLE as a client
2. Read server response and store it in an unsigned char array in SPIFFS in the manner described above
3. Try to save this unsigned char with
4. Put the server to sleep after 5 seconds from 2) and have it wake up long after
5. Repeat from 2-5

Otherwise, I will share my main.cpp here. I am hoping someone can give me an "analytical" explanation, and I will try to test it out on my own codebase :)

#include <NimBLEDevice.h>
#include "Arduino.h"
#include <BLEMacros.h>
#include <FileManager.h>
#include <DataLogging.h>
#include <STATES_MACROS.h>
#include <nvs_flash.h>


static volatile bool print_on = false;
static volatile bool xperiment_on = false;
static volatile bool restart_on = false;
bool were_scanning_print = false;




void scanEndedCB(NimBLEScanResults results);

static NimBLEAdvertisedDevice* advDevice;

static bool doConnect = false;
static uint32_t scanTime = 0; /** 0 = scan forever */


static int SWC_counter = 0;
static int timestep_count = 0;

static unsigned char * SWC_read = nullptr;
static unsigned long time_last_read = 0L;
unsigned long time_start = millis();

static int state = STATE_IDLE;



void IRAM_ATTR printon_interrupt()
{
    print_on = !print_on;
}

void IRAM_ATTR xperimenton_interrupt()
{
    xperiment_on = !xperiment_on;
    state = STATE_IDLE;
}

void IRAM_ATTR restart_interrupt()
{
    restart_on = true;
}

NimBLEClient * pClient_SWC;
SPIFFSFileManager& fileMane = SPIFFSFileManager::get_instance();


/**  None of these are required as they will be handled by the library with defaults. **
 **                       Remove as you see fit for your needs                        */
class ClientCallbacks : public NimBLEClientCallbacks {
    void onConnect(NimBLEClient* pClient) {
        Serial.println("Connected");
        /** After connection we should change the parameters if we don't need fast response times.
         *  These settings are 150ms interval, 0 latency, 450ms timout.
         *  Timeout should be a multiple of the interval, minimum is 100ms.
         *  I find a multiple of 3-5 * the interval works best for quick response/reconnect.
         *  Min interval: 120 * 1.25ms = 150, Max interval: 120 * 1.25ms = 150, 0 latency, 60 * 10ms = 600ms timeout
         */
        pClient->updateConnParams(120,120,0,60);
    };

    void onDisconnect(NimBLEClient* pClient) {
        Serial.print(pClient->getPeerAddress().toString().c_str());
        Serial.println(" Disconnected ");
        pClient->disconnect();
        NimBLEDevice::getScan()->start(scanTime, scanEndedCB);
    };

    /** Called when the peripheral requests a change to the connection parameters.
     *  Return true to accept and apply them or false to reject and keep
     *  the currently used parameters. Default will return true.
     */
    bool onConnParamsUpdateRequest(NimBLEClient* pClient, const ble_gap_upd_params* params) {
        if(params->itvl_min < 24) { /** 1.25ms units */
            return false;
        } else if(params->itvl_max > 40) { /** 1.25ms units */
            return false;
        } else if(params->latency > 2) { /** Number of intervals allowed to skip */
            return false;
        } else if(params->supervision_timeout > 100) { /** 10ms units */
            return false;
        }

        return true;
    };

    /********************* Security handled here **********************
    ****** Note: these are the same return values as defaults ********/
    uint32_t onPassKeyRequest(){
        Serial.println("Client Passkey Request");
        /** return the passkey to send to the server */
        return 123456;
    };

    bool onConfirmPIN(uint32_t pass_key){
        Serial.print("The passkey YES/NO number: ");
        Serial.println(pass_key);
        /** Return false if passkeys don't match. */
        return true;
    };

    /** Pairing process complete, we can check the results in ble_gap_conn_desc */
    void onAuthenticationComplete(ble_gap_conn_desc* desc){
        if(!desc->sec_state.encrypted) {
            Serial.println("Encrypt connection failed - disconnecting");
            /** Find the client with the connection handle provided in desc */
            NimBLEDevice::getClientByID(desc->conn_handle)->disconnect();
            return;
        }
    };
};


/** Define a class to handle the callbacks when advertisments are received */
class AdvertisedDeviceCallbacks: public NimBLEAdvertisedDeviceCallbacks {

    void onResult(NimBLEAdvertisedDevice* advertisedDevice) {
        if(advertisedDevice->isAdvertisingService(NimBLEUUID(SWC_SRVI_UUID)))
        {
            Serial.println("Found Our Service");
            /** stop scan before connecting */
            NimBLEDevice::getScan()->stop();
            /** Save the device reference in a global for the client to use*/
            advDevice = advertisedDevice;

            /** Ready to connect now */
            doConnect = true;
            Serial.println(doConnect);
        }
    };
};


/** Notification / Indication receiving handler callback */
void notifyCB(NimBLERemoteCharacteristic* pRemoteCharacteristic, uint8_t* pData, size_t length, bool isNotify){
    std::string str = (isNotify == true) ? "Notification" : "Indication";
    str += " from ";
    /** NimBLEAddress and NimBLEUUID have std::string operators */
    str += std::string(pRemoteCharacteristic->getRemoteService()->getClient()->getPeerAddress());
    str += ": Service = " + std::string(pRemoteCharacteristic->getRemoteService()->getUUID());
    str += ", Characteristic = " + std::string(pRemoteCharacteristic->getUUID());
    str += ", Value = " + std::string((char*)pData, length);
    Serial.println(str.c_str());
}

/** Callback to process the results of the last scan or restart it */
void scanEndedCB(NimBLEScanResults results){
    Serial.println("Scan Ended");
}


/** Create a single global instance of the callback class to be used by all clients */
static ClientCallbacks clientCB;


/** Handles the provisioning of clients and connects / interfaces with the server */
bool connectToServer() {
    NimBLEClient* pClient = nullptr;

    /** Check if we have a client we should reuse first **/
    if(NimBLEDevice::getClientListSize()) {
        /** Special case when we already know this device, we send false as the
         *  second argument in connect() to prevent refreshing the service database.
         *  This saves considerable time and power.
         */
        pClient = NimBLEDevice::getClientByPeerAddress(advDevice->getAddress());
        if(pClient){
            if(!pClient->connect(advDevice, false)) {
                Serial.println("Reconnect failed");
                return false;
            }
            Serial.println("Reconnected client");
        }
            /** We don't already have a client that knows this device,
             *  we will check for a client that is disconnected that we can use.
             */
        else {
            pClient = NimBLEDevice::getDisconnectedClient();
        }
    }

    /** No client to reuse? Create a new one. */
    if(!pClient) {
        if(NimBLEDevice::getClientListSize() >= NIMBLE_MAX_CONNECTIONS) {
            Serial.println("Max clients reached - no more connections available");
            return false;
        }

        pClient = NimBLEDevice::createClient();

        Serial.println("New client created");

        pClient->setClientCallbacks(&clientCB, false);
        /** Set initial connection parameters: These settings are 15ms interval, 0 latency, 120ms timout.
         *  These settings are safe for 3 clients to connect reliably, can go faster if you have less
         *  connections. Timeout should be a multiple of the interval, minimum is 100ms.
         *  Min interval: 12 * 1.25ms = 15, Max interval: 12 * 1.25ms = 15, 0 latency, 51 * 10ms = 510ms timeout
         */
        pClient->setConnectionParams(12,12,0,51);
        /** Set how long we are willing to wait for the connection to complete (seconds), default is 30. */
        pClient->setConnectTimeout(6);


        if (!pClient->connect(advDevice)) {
            /** Created a client but failed to connect, don't need to keep it as it has no data */
            NimBLEDevice::deleteClient(pClient);
            Serial.println("Failed to connect, deleted client");
            return false;
        }
    }

    if(!pClient->isConnected()) {
        if (!pClient->connect(advDevice)) {
            Serial.println("Failed to connect");
            return false;
        }
    }

    Serial.print("Connected to: ");
    Serial.println(pClient->getPeerAddress().toString().c_str());
    Serial.print("RSSI: ");
    Serial.println(pClient->getRssi());

    /** Now we can read/write/subscribe the charateristics of the services we are interested in */
    NimBLERemoteService* pSvc = nullptr;
    NimBLERemoteCharacteristic* pChr = nullptr;
    NimBLERemoteDescriptor* pDsc = nullptr;

    pSvc = pClient->getService(SWC_SRVI_UUID);
    if(pSvc){
        NimBLERemoteCharacteristic * VWC_chr = pSvc->getCharacteristic(SWC_VWC_CHAR_UUID);
        String SWC_read_string = (String) VWC_chr->readValue().c_str();
        SWC_read = format_SWC( (unsigned char*) SWC_read_string.c_str());
        //SWC_read = const_cast<unsigned char*>(reinterpret_cast<const unsigned char*>("1.23"));

        time_last_read = millis();
        //time_last_read = 69L;
        unsigned long timestamp = time_last_read-time_start;
        //timestamp = time_last_read;
        printf("Will try to save the SWC: %c%c%c%c%c and the timestamp %lu \n",SWC_read[0],SWC_read[1],SWC_read[2],SWC_read[3],SWC_read[4],timestamp);

    }


    return true;
}

void setup (){
    Serial.begin(115200);
    Serial.println("Starting NimBLE Client");
    fileMane.mount();
    nvs_flash_init();

    //Interrupts and pins
    pinMode(PRINTING_ON_TRIGGER_PIN,INPUT);
    attachInterrupt(digitalPinToInterrupt(PRINTING_ON_TRIGGER_PIN),printon_interrupt,HIGH);

    pinMode(XPERIMENT_ON_TRIGGER_PIN,INPUT);
    attachInterrupt(digitalPinToInterrupt(XPERIMENT_ON_TRIGGER_PIN),xperimenton_interrupt,HIGH);

    pinMode(RESTART_TRIGGER_PIN,INPUT);
    attachInterrupt(digitalPinToInterrupt(RESTART_TRIGGER_PIN),restart_interrupt,HIGH);

    //LED BLINKERS
    pinMode(BLE_FLASH_PIN,OUTPUT);
    pinMode(XPERIMENT_ARRAY_FULL, OUTPUT);
    pinMode(XPERIMENT_ON_PIN,OUTPUT);

    /** Initialize NimBLE, no device name spcified as we are not advertising */
    NimBLEDevice::init(BLE_CLIENT_NAME);

    /** Set the IO capabilities of the device, each option will trigger a different pairing method.
     *  BLE_HS_IO_KEYBOARD_ONLY    - Passkey pairing
     *  BLE_HS_IO_DISPLAY_YESNO   - Numeric comparison pairing
     *  BLE_HS_IO_NO_INPUT_OUTPUT - DEFAULT setting - just works pairing
     */
    //NimBLEDevice::setSecurityIOCap(BLE_HS_IO_KEYBOARD_ONLY); // use passkey
    //NimBLEDevice::setSecurityIOCap(BLE_HS_IO_DISPLAY_YESNO); //use numeric comparison

    /** 2 different ways to set security - both calls achieve the same result.
     *  no bonding, no man in the middle protection, secure connections.
     *
     *  These are the default values, only shown here for demonstration.
     */
    //NimBLEDevice::setSecurityAuth(false, false, true);
    NimBLEDevice::setSecurityAuth(/*BLE_SM_PAIR_AUTHREQ_BOND | BLE_SM_PAIR_AUTHREQ_MITM |*/ BLE_SM_PAIR_AUTHREQ_SC);

    /** Optional: set the transmit power, default is 3db */
#ifdef ESP_PLATFORM
    NimBLEDevice::setPower(ESP_PWR_LVL_P9); /** +9db */
#else
    NimBLEDevice::setPower(9); /** +9db */
#endif

    /** Optional: set any devices you don't want to get advertisments from */
    // NimBLEDevice::addIgnored(NimBLEAddress ("aa:bb:cc:dd:ee:ff"));

    /** create new scan */
    NimBLEScan* pScan = NimBLEDevice::getScan();

    /** create a callback that gets called when advertisers are found */
    pScan->setAdvertisedDeviceCallbacks(new AdvertisedDeviceCallbacks());

    /** Set scan interval (how often) and window (how long) in milliseconds */
    pScan->setInterval(0);
    pScan->setWindow(15);

    /** Active scan will gather scan response data from advertisers
     *  but will use more energy from both devices
     */
    pScan->setActiveScan(true);
}


void loop (){

    if(restart_on)
    {
        log_e("Restarting ESP32");
        bool res1,res2,finish = false;
        do{
        delay(500);
        res1 = overwrite_value_array(SWC_VALUE_ARRAY_SIZE,SWC_VALUE_ARRAY_PATH);
        delay(1500);
        res2 = overwrite_value_array(TIMESTEP_VALUE_ARRAY_SIZE,TIMESTEP_VALUE_ARRAY_PATH);
        finish = res1 && res2;
        }
        while(!finish);


        esp_system_abort("Restarting experiment");

    }

    while(print_on)
    {
        if(NimBLEDevice::getScan()->isScanning())
        {
            were_scanning_print = true;
             NimBLEDevice::getScan()->stop();}

        unsigned char * print_TIMESTEP = (unsigned char* )malloc(sizeof(char) * TIMESTEP_VALUE_ARRAY_SIZE);
        fileMane.load_file(TIMESTEP_VALUE_ARRAY_PATH,reinterpret_cast<unsigned char *>(print_TIMESTEP),TIMESTEP_VALUE_ARRAY_SIZE-1);
        for(int i = 0; i<(SIZE_OF_TIMESTAMP_AFTER_FORMATTING*200)/4;i += 4)
        {
            unsigned long long_rep = (print_TIMESTEP[i]) | (print_TIMESTEP[i+1]<< 8) | (print_TIMESTEP[i+2] << 16) | (print_TIMESTEP[i+3] << 24);
            printf("%lu ",long_rep);
            delay(10);
        }

        vTaskDelay(1000/portTICK_PERIOD_MS);

        log_e("\nNumber of writes to timestep: %i\n",timestep_count);

        free(print_TIMESTEP);

        char * print_SWC = (char* )malloc(sizeof(char) * SWC_VALUE_ARRAY_SIZE);
        fileMane.load_file(SWC_VALUE_ARRAY_PATH,reinterpret_cast<unsigned char *> (print_SWC),SWC_VALUE_ARRAY_SIZE-1);
        for(int i = 0; i<200*SIZE_OF_SWC_AFTER_FORMATTING;i++)
        {
            printf("%c",print_SWC[i]);
            delay(10);
        }

        free(print_SWC);

        vTaskDelay(1000/portTICK_PERIOD_MS);

        log_e("\nNumber of writes to SWC: %i\n",SWC_counter);

        vTaskDelay(TIME_DELAY_AFTER_PRINTING_mS/portTICK_PERIOD_MS);
        if(!print_on && were_scanning_print)
        {
            NimBLEDevice::getScan()->start(scanTime,scanEndedCB);
            log_e("Done printing\n");
        }
    }



    if(xperiment_on)
    {
        digitalWrite(XPERIMENT_ON_PIN,LOW);
        switch(state)
        {
            case STATE_IDLE:

                log_e("%i",xperiment_on);
                log_e("State: Idle");

                if(!print_on)
                {

                    NimBLEDevice::getScan()->start(scanTime,scanEndedCB);
                    state = STATE_LISTENING;
                }

                break;

            case STATE_LISTENING:


                //If we can connect to our server
                if(doConnect)
                {
                    connectToServer();
                    doConnect = false;

                    digitalWrite(BLE_FLASH_PIN,HIGH);
                    vTaskDelay(100/portTICK_PERIOD_MS);
                    digitalWrite(BLE_FLASH_PIN,LOW);

                    if(xperiment_on){
                    insert_at_carriage_return_and_save(TIMESTEP_VALUE_ARRAY_PATH,(time_last_read-time_start),SIZE_OF_TIMESTAMP_AFTER_FORMATTING,TIMESTEP_VALUE_ARRAY_SIZE,timestep_count*SIZE_OF_TIMESTAMP_AFTER_FORMATTING,&timestep_count);
                    insert_at_carriage_return_and_save(SWC_VALUE_ARRAY_PATH,SWC_read,SIZE_OF_SWC_AFTER_FORMATTING,SWC_VALUE_ARRAY_SIZE,SWC_counter*SIZE_OF_SWC_AFTER_FORMATTING,&SWC_counter);
                    state = STATE_VALUE_RECEIVED;}
                }

                break;

            case STATE_VALUE_RECEIVED:

                printf("State: Received\n");

                //TODO: Check if array is full


                if(xperiment_on)
                {
                    log_e("State: Listening\n");
                    state = STATE_LISTENING;
                }
                else
                {
                    state = STATE_IDLE;
                }

                break;
        }

    }
    else{
        //log_e("Experiment off");
        digitalWrite(XPERIMENT_ON_PIN,HIGH);
        doConnect = false;
        vTaskDelay(1000/portTICK_PERIOD_MS);
        if(NimBLEDevice::getScan()->isScanning())
        {
            //We add delays, so that the GATT server has time to either startup- or shutdown before th button can pressed again
            NimBLEDevice::getScan()->stop();
            vTaskDelay(1000/portTICK_PERIOD_MS);
        }
        state = STATE_IDLE;
    }



}

Debug Message

There was no debug message. See the screenshots above for the semantical error.

I even include multiple exceptions in my saving method:

bool SPIFFSFileManager::save_file_with_retries(const char *filePath, const unsigned char *dataToWrite,int size_of_file, int attempts, int mS_delay_between) {
    int success = 0;
    int retries = 0;

    SPIFFSFileManager& fileMan = SPIFFSFileManager::get_instance();

    while (retries < attempts) {
        // Try to save the file
        success = fileMan.save_file(filePath, dataToWrite,size_of_file);

        // Check if save was successful
        if (success) {
            log_i("Succesfully inserted value into value array.\n");
            return true;
        } else {
            log_i("Failed to insert value array into path %s for the %i 'th time. Retrying again in %i milliseconds",filePath,retries,mS_delay_between);
            vTaskDelay(mS_delay_between / portTICK_PERIOD_MS);
            retries++;
        }
    }

    return false;
}

And loading:

bool SPIFFSFileManager::load_file(const char * filePath, unsigned char * resultArray, size_t endIdx){
    File f1 = fileSystem.open(filePath,FILE_READ);
    if(!f1 || f1.isDirectory()){
        throw std::logic_error("− failed to open file for reading. You possibly tried to load a directory");
    }

    unsigned char res;

    for (int i = 0; i<endIdx; i++){
        if(!f1.available()){
            break;
        }
        res = f1.read();
        resultArray[i] = res;
    }

    resultArray[endIdx] = '\0';

    return true;

}

I have run unit tests, that confirm that these methods work without NimBLE activated.

Other Steps to Reproduce

My Pio.ini file, if it helps:

; PlatformIO Project Configuration File
;
; Build options: build flags, source filter
; Upload options: custom upload port, speed and extra flags
; Library options: dependencies, extra library storages
; Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:esp32dev]
platform = espressif32 @ 6.3.0
board = NodeMCU-32S
framework = arduino
monitor_speed = 115200
monitor_filters = esp32_exception_decoder
upload_port = COM3
monitor_port = COM3
test_port = COM3

debug_tool = esp-prog
debug_init_break = tbreak setup

lib_deps =
h2zero/NimBLE-Arduino@^1.4.1
extentsoftware/TBeamPower@^2.0.4
milesburton/DallasTemperature@^3.11.0
4-20ma/ModbusMaster@^2.0.1
thebigpotatoe/Effortless-SPIFFS@^2.3.0
lib_ldf_mode = deep+
board_build.partitions = SPIFFSoneMPartition.csv
board_build.flash_mode = dio

build_type = debug

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lbernstone]

This forum is for issues with the code in this repository. It is not for issues with your code, or 3rd party libraries. You can ask your question at https://esp32.com , but expecting someone to guess at your problem without providing a minimal example that can be copied and compiled is not going to get you many answers.

[SuGlider]

@BrugerX - The project seems very nice. If the issue is related to SPIFFS, maybe you can isolate the code, create some timed writing method and check if it fails. This could result in a smaller sketch that can be used to reproduce the issue.

As an alternative, I would suggest using LitleFS instead of SPIFFS. It may solve the issue with no collateral effect for the project.

LittleFS is crash proof so every append first makes a copy, appends to the copy, then replaces the original. In this way a crash during append can not corrupt the original file. It is a bit slower because of it, but it may be safer than SPIFFS.

[VojtechBartoska]

Closing this as expired due to no answer from the author of the issue.