Help with reading data from csv in BangleJs2

[espruino-discuss2]

Posted at 2023-06-06 by user155657

Hello,

I am a student who is to new JS and recently got my hands on to bangleJS2. As part of our student project we are planning to acquire motion and heartrate data from bangleJS2.
The goal of the following code is to save outputs of accelerometer, magnetometer and heartrate sensor into a csv file. Due to storage limitation we are stopping it to record data when Flash is less than 0.75MB.

We want to record at least two days worth of data.

fileName = "rawData";
Bangle.setCompassPower(1);
var rawData = require("Storage").open(fileName+".csv", "a");
Bangle.setHRMPower(1);
var timePast = Date.now();
Bangle.on('HRM-raw', function(hrm) {
  var memory = require("Storage").getStats();
  var hrmRaw = [
    "HRM:",
    hrm.raw,
    hrm.filt,
    hrm.bpm,
    hrm.confidence
  ];
  var c = Bangle.getCompass();
  var a = Bangle.getAccel();
  //Data order in the csv
  //timestamp, accelerometer[x,y,z], magentometr [x, y, g, dx, dy, dz], hrm [raw, filter, bpm, confidence]
  if(Date.now()-timePast>100){
    if (memory > 750000)
    {
    //1000000 Bytes = 1 MB (in decimal)
rawData.write([Math.floor(Date.now()),a.x,a.y,a.z,c.x,c.y,c.z,c.dx,c.dy,c.dz,hrm.raw,hrm.filt,hrm.bpm,hrm.confidence].map((o)=>parseInt(o*1000)/1000).join(","));
  rawData.write("\n");
  timePast = Date.now();}
  else{
   g.drawString("Memory full", 50, 50)
  }}
});

When we are trying to call the data we are only retrieving only 550 rows of data, even though the csv is having more data, we aren't able to get more values
We are using the following code in our script to read data

//getting Data
var getData = require('Storage').read('rawData.csv\1');
var array = getData.split("\n");
Bluetooth.println("<data>\n"+array[d]+"\n</data>");
//Removing Data
require("Storage").erase('rawData.csv\1');
require("Storage").open("rawData.csv", "w");

Is there a way to read all the rows from the csv file to array and send using ''Bluetooth.print''?

Apologies for my clumsy code.
Thank you in advance.

[espruino-discuss2]

Posted at 2023-06-06 by rigrig

var getData = require('Storage').read('rawData.csv\1');

This only reads chunk 1 of the Storage file, see here for an explanation about those.

You probably want something like this:

var dataFile = require('Storage').readopen('rawData.csv', 'r');
var line = dataFile.readLine();
while (line != undefined) {
        Bluetooth.println(`<data>\n${line}\n</data>`);
        line = dataFile.readLine();
}
dataFile.erase();

Also:

var memory = require("Storage").getStats();

Shouldn't this be var memory = require("Storage").getFree(); ?

---

Posted at 2023-06-06 by user155657

Thank you very much.
I will try it today and update you

Also var memory = require("Storage").getFree(); is a very good idea.

---

Posted at 2023-06-06 by @gfwilliams

Looks like @rigrig's got you sorted.

Just some notes though:

• getStats gives you a bunch of other info, not just free space.
• getStats/getFree can be slow since it scans all of storage to work out how much data there is - maybe think about just calling it every minute, or even better call it once at startup to work out how much free there is, and keep track of how much you're writing. It won't be exact because there can be some overheads but since you're leaving 0.75MB of leeway you'll be fine
• It's worth doing as few calls to .write as possible to make it a bit faster - so try and do rawData.write([....].map((o)=>parseInt(o*1000)/1000).join(",")+"\n");

So maybe more like:

var memory = require("Storage").getFree();
Bangle.on('HRM-raw', function(hrm) {
  // ...
  var line = [Math.floor(Date.now()),a.x,a.y,a.z,c.x,c.y,c.z,c.dx,c.dy,c.dz,hrm.raw,hrm.filt,hrm.bpm,hrm.confidence].map((o)=>parseInt(o*1000)/1000).join(",")+"\n";
  memory -= line.length;
  rawData.write(line);

---

Posted at 2023-06-06 by user155657

Thanks @gfwilliams
Another quick question, do you think my approach of using Date.now()-timePast>100 for sampling rate is efficient or you recommend any other way?
I took inspiration from Arduino millis() function

Thanks again

---

Posted at 2023-06-06 by user155657

Thank you very much.
It is working :-)

---

Posted at 2023-06-08 by @gfwilliams

do you think my approach of using Date.now()-timePast>100 for sampling rate is efficient or you recommend any other way?

That'll be ok but it wouldn't give exactly 10Hz since you're sampling only when the delay is 100ms or more - but the HRM by default runs at 25Hz, so you might get steadier readings by just averaging two readings. It'd give you 12.5Hz but it'd be better to stay at that than to drop samples and get 10ish Hz.