RingBuffer file implementation.

[conkerkh]

I'm trying to transition from FatFS to littlefs, because of several benefits comming from wear-leveling and other things, I have ring buffer implemented as a file storing head and tail as first 8 bytes of data, then there is correct data. From what I noticed and what is mentioned in #27 it seems that littlefs performance isn't very good when writing at the beginnig of the file as the rest of the content has to be rewritten. In my case I store head and tail at the beginnin of the file, the file size is around 4MB. Obviously writing to such massive file in worst case scenario is around couple of minutes on quite slow SPI Flash. Specifically in case of littlefs what would be advised option for storing data? I need to use it in LIFO fashion where there is logging task writing data and reading task consuming this data, but from what I know it's not possible to "remove" data at the beginning of the file? Shifting data is not an option as it takes to much time. I could truncate file but it isn't very convinient as the data is sent over the network and new data would be prioritised in this case which shouldn't be the case.

[geky]

Hi @conkerkh, I'm glad you asked this question. As you mentioned FIFOs as a files are a bit tricky and littlefs's append-only COW structure doesn't work well with the obvious solution. Which is extra unfortunate since logs are a very common use case for embedded filesystems.

What does work well is to do something similar to the logrotate utility on Unix:

1. Split the log into multiple files
2. Rename log files when full
3. Delete oldest log file

The simples logrotate would have only two files. This means the log takes up 2x the space. This could be improved by using more log files, but adds complexity.

Also I'm not sure the best place to store the offset. The best place would probably be to store it in its own file.

I put together a quick snippet showing how this could work. Note I haven't tested this, and it doesn't consider the case where your data is not aligned to your log size. But hopefully it's a good starting point:

#include "lfs.h"

// logrotate configuration
#define LOGROTATE_SIZE    (4*1024*1024)
#define LOGROTATE_FILE0   "logrotate.0"
#define LOGROTATE_FILE1   "logrotate.1"
#define LOGROTATE_FILEOFF "logrotate.off"

// logrotate state, may be nicer in a struct if you want to support multiple logs
lfs_t lfs;
lfs_file_t file;
lfs_off_t off;
struct lfs_info info;

static int logrotate_getoff(void) {
    // try to read offset file
    int err = lfs_file_open(&lfs, &file, LOGROTATE_FILEOFF, LFS_O_RDONLY);
    if (err && err != LFS_ERR_NOENT) {
        return err;
    }

    if (err == LFS_ERR_NOENT) {
        // default to offset that skips to file 1
        off = LOGROTATE_SIZE;
        return 0;
    }

    // read offset if it exists
    err = lfs_file_read(&lfs, &file, &off, sizeof(off));
    if (err < 0) {
        return err;
    }

    err = lfs_file_close(&lfs, &file);
    if (err) {
        return err;
    }

    return 0;
}

static int logrotate_setoff(void) {
    // write offset file
    int err = lfs_file_open(&lfs, &file, LOGROTATE_FILEOFF,
            LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC);
    if (err) {
        return err;
    }

    err = lfs_file_write(&lfs, &file, &off, sizeof(off));
    if (err) {
        return err;
    }

    err = lfs_file_close(&lfs, &file);
    if (err) {
        return err;
    }

    return 0;
}


int logrotate_init(void) {
    int err = logrotate_getoff();
    if (err) {
        return err;
    }

    return 0;
}

int logrotate_deinit(void) {
    // noop
    return 0;
}

int logrotate_append(const void *data, lfs_size_t size) {
    // first check if we need to rotate, we rotate when our log is half full
    int err = lfs_stat(&lfs, LOGROTATE_FILE0, &info);
    if (err && err != LFS_ERR_NOENT) {
        return err;
    }

    if (err != LFS_ERR_NOENT && info.size >= LOGROTATE_SIZE) {
        if (off < LOGROTATE_SIZE) {
            // can't rotate, haven't depleted the oldest log file
            return LFS_ERR_NOSPC;
        }

        // rotate using rename
        err = lfs_rename(&lfs, LOGROTATE_FILE0, LOGROTATE_FILE1);
        if (err) {
            return err;
        }

        // update offset
        off -= LOGROTATE_SIZE;
        err = logrotate_setoff();
        if (err) {
            return err;
        }
    }

    // open for append
    err = lfs_file_open(&lfs, &file, LOGROTATE_FILE0,
            LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND);
    if (err) {
        return err;
    }

    err = lfs_file_write(&lfs, &file, data, size);
    if (err < 0) {
        return err;
    }

    err = lfs_file_close(&lfs, &file);
    if (err) {
        return err;
    }

    return 0;
}

int logrotate_pop(void *data, lfs_size_t size) {
    // which file do we pop from?
    const char *filepath;
    lfs_off_t fileoff;
    if (off < LOGROTATE_SIZE) {
        // pop from older file
        fileoff = off;
        filepath = LOGROTATE_FILE1;
    } else {
        // pop from newer file
        fileoff = off - (LOGROTATE_SIZE);
        filepath = LOGROTATE_FILE0;
    }

    // read our data
    int err = lfs_file_open(&lfs, &file, filepath, LFS_O_RDONLY);
    if (err) {
        return err;
    }

    err = lfs_file_seek(&lfs, &file, fileoff, LFS_SEEK_SET);
    if (err < 0) {
        return err;
    }

    err = lfs_file_read(&lfs, &file, data, size);
    if (err < 0) {
        return err;
    }

    err = lfs_file_close(&lfs, &file);
    if (err) {
        return err;
    }

    // update offset
    off += size;
    err = logrotate_setoff();
    if (err) {
        return err;
    }

    return 0;
}

[geky]

Actually I just noticed you said LIFO not FIFO, was this intentional? A LIFO would actually be simpler, though less common.

[conkerkh]

@geky thanks for answer, I had FIFO in mind was quite late when I was writing this post and in the meantime I got a bit confused😜 I really like this concept of rotating two files with saving offset in the third one, looks like really good idea. I’ll put this together and write UT to test all the possibilities. Thanks a lot!

[conkerkh]

Also perhaps since you said yourself this type of logging is very common we could include something like this in utils folder?

[geky]

That's a good idea. I was thinking actually create an examples section in the readme. Putting the code in a utils folder would mean we need to worry about backwards compatibility, since it's effectively a part of that API at that point, and this sort of thing may end up heavily customized by the user.