File content rollover

[espruino-discuss]

Posted at 2020-04-09 by msdeibel

Hello everyone,

I just started working on a data recorder widget/app for the Bangle and want to limit the file size to keep a maximum of one week worth of data. Which amounts to around 1000 lines in a text file (1 line/10 min).

Now that the storage size is limited and flash mem has a limited number of write cycles I was wondering what would be the best way to handle this.

In a simple solution I would

open the data file
read all lines into an array
append the new line to the array
write all lines, but the first, into the file

Since I'm quite new to developing for the Espruino I don't want to mess up the flash within a few days of the widget repeatedly writing to the flash.

Do you have any suggestions how the proposed solution could be improved?

[gfwilliams]

Posted at 2020-04-09 by @gfwilliams

Well, using just Storage.write will result in the whole file re-written each time, which is obviously not good. Storage.open(file,"a") creates a slightly different file type which allows easy appends, but dropping the first few lines of the file and rewriting will still be bad.

What I'd suggest is to have one StorageFile (with Storage.open(file,"a")) for each day of the week, and then at midnight when you swap over you do Storage.open(file,"r").erase() to erase the previous day (7 days ago) before starting afresh.

That way you're actually being super kind to the flash, and it should be reasonably easy to implement?

[espruino-discuss]

Posted at 2020-04-13 by msdeibel

Here's what I implemented. I'll leave it here for further reference.
The full code can be found at https://github.com/msdeibel/BangleApps/blob/master/apps/batchart/widget.js

function logBatteryData() {
      const previousWriteLogName = "bcprvday";
      const previousWriteDay = require("Storage").read(previousWriteLogName);
      const currentWriteDay = new Date().getDay();

      const logFileName = "bclog" + currentWriteDay;

      // Change log target on day change
      if (previousWriteDay != currentWriteDay) {
        //Remove a log file containing data from a week ago
        require("Storage").erase(logFileName);
        require("Storage").write(previousWriteLogName, currentWriteDay);
      }

      var bcLogFileA = require("Storage").open(logFileName, "a");
      if (bcLogFileA) {
        console.log([getTime().toFixed(0), E.getBattery(), E.getTemperature(), getEnabledConsumersValue()].join(","));
        bcLogFileA.write([[getTime().toFixed(0), E.getBattery(), E.getTemperature(), getEnabledConsumersValue()].join(",")].join(",")+"\n");
      }
  }

[gfwilliams]

Posted at 2020-04-14 by @gfwilliams

Great - thanks for posting it up!

However it looks like you need require("Storage").open(logFileName,"r").erase() instead of require("Storage").erase(logFileName);

It's a bit tricky at the moment but StorageFile and the files created by require("Storage").read/write/erase are a bit different.