Fitness data protocol

[JF002]

For now, we only expose the live heart rate values. Exposing steps the same way won't be difficult.

For more advanced features, companion app need more than just the current value, they'll need an historic of all the data acquired during the day/night, even if the watch was not connected to the companion app when the data was acquired.

This goal of this discussion is to define how the data will be stored and the API InfiniTime will expose to the companion app.

I asked Adam from Amazfish how fitness data were managed on other devices :

• The watch stores the data with a specific granularity along with a timestamp. Ex : number of steps counted every 10 minutes, HR value per minute,...
• The data should be stored in some kind of circular buffer so that we ensure we don't fill the memory
• Using BLE, the companion app request the data from a specific date-time, and of course, InfiniTime sends them if they are available in the buffer.

Questions:

• What would be an ideal granularity for the data/timestamp?
• Should the data be stored in RAM? External flash? Temporarily in RAM and flushed in external flash from time to time (pay attention to wear of the flash memories) ?

Settings

In settings, we define the granularity (the delay between 2 samples). Ex : 10 minutes

Data acquisition

InfiniTime monitores the data (steps + HR if enabled) and accumulates them during the "granularity" period.

When the period elapses, the accumulated data are appended in the circular buffer in the form

struct FitnessData {
  uint64_t timestamp;
  uint8_t nbSteps;
  uint8_t heartRate;
}

We could also add average/min/max for the heart rate value. We can also store a single reference timestamp, and then store a delta from this timestamp in FitnessData to save a few bytes for each sample.

BLE API

Service : FitnessDataService
Characteristics:

• Period (Read/Write) : the companion app can read the current acquisition period and set a new one.
• Commands (Write)
  • Request data from a specified timestamp
  • Clear data (from / to a specific timestamp)
• Data (Read) : requested data will be written on this characteristic
• Data available (Notify) : notifies the companion app that new values are available

The protocol should be designed in such a way it will be possible to extend it (add a version characteristic or field in the data sent to the companion app).

Let's talk about this ;)

These are just some ideas out of my head, feel free to give your opinion.
I'm not sure that 'fitnessData' is the most appropriate name for the data...

[piggz]

One idea to start with, in the reply to the request for data, part of that return could be the format version. That would allow the protocol to be extended without hacks. This is something huami devices -do not- do.

[JF002]

One idea to start with, in the reply to the request for data, part of that return could be the format version. That would allow the protocol to be extended without hacks. This is something huami devices -do not- do.

I've just added a note in the original post.

[infinitytec]

10 minutes seems like a pretty reasonable time to me. Depending on some exercise routines people may do there may be reasons to do more frequent logging, but I imagine those could be implemented later.

Another way of doing it would be an option that let's the user set the interval time.

It would be nice to log it to external flash every now and then. That way if the watch restarts the daily step count could stay.

[piggz]

Id like to have accuracy down to 1minute. There is an algorithm called Personal Activity Intelligence (PAI) based on heartrate, and to get data on the huami watches requires that all-day HRM scanning is on 1m accuracy. I imagine there is a battery impact on that, so the default could certainly be higher.

[infinitytec]

If the watch has a stable connection to a companion app the data could stay in RAM and then sent to the app rather than having to save it to flash. Then if the connection drops it could start writing data to flash.

[lman0]

@JF002 i think that for HR at least it should be configurable on pinetime , and maybe from gadgetbrigdge as well ,
with setting like once by minute , every 5 / 15 /30 /60 minute.
because some sport like HIIT need the HR to be very fast (the exercise last less than 5 minute ) so a stting with one by minute is needed.

maybe in order to make it not take to much battery on some case , you could tied the specific case like with an app inside infintime to allow to configure preset for exexrcise / laps with the alarm or timer as base :
the user would set the timer duration for the the hr reading , and the user would be able to save preset in external memory:
example of user defined preset:

• preset 1 :30 minute with HR every 1 minutes
• preset 2 : 60 minute with HR every 5 minutes
  ...
  and the app would be able to extend the duration , if needed

this would allow to have a default setting (like every 5 or 15 minute for HR reading)
and specific case with the app inside infinitme.

if it would be able to be setted from gadget bridge or from fitotrack it would become configurable from samrtphone.

for the motion part , it shouls be interesting for you to see what needed for sleep as android and gadgetbrige to make use the most of it

[snowsquizy]

What frequency do you we expected users to connect to download the data, daily, intraday or greater than daily? I think that periodicity would be used to set the storage volume.
Making it configurable is probably easiest upfront and providing rough time of storage for each setting. User chooses

[infinitytec]

1.7.0 says it has motion service in it, but it doesn't seem to send data to Gadgetbridge.

[Avamander]

@infinitytec wait for the Gadgetbridge changelog to say that it has added support for it. InfiniTime and Gadgetbridge are two separate projects.

[Oblomov]

I'm very interested in this feature. I've been thinking about what would be an efficient way to store the data, and the first thought I've had is that some users may only be interested in one of the values (personally, for example, I only care about steps, not HR).
This might allow some storage optimization (be it in RAM and/or in flash).

Brainstorming now, ignore the ring buffer and consider this: at least for steps the storage could actually be made at the granularity of the individual steps, by encoding the timestamp delta since the previous (individual) step. Since even the fastest sprinters don't take more than 5/6 steps for second, we could have a granularity of something like an 8th of a second for the deltas, so that a uint8_t delta would be enough to store steps less than 32 seconds apart (after which we'd need to store the full timestamp). In fact, during activity we could probably store a delta per nibble. In this case, the storage requirement would depend on the number of steps taken. At two steps per byte, 16K steps would require around 8KB of RAM, which is admittedly a bit much, but hey, you're recording every single step! For comparison, a full day (24h) of 1 minute sampling with 10 bytes per sample would take 14KB, so this could actually be considered a net gain ;-)

Of course the comparison is unfair: with 1m samples you don't actually need 64-bit timestamps for each packet, you cold do one full packet every e.g. 64 samples, and the others could get away with storing a 16-bit timestamp delta too; this however makes the ring buffer implementation more complex.

On a more serious note, with regular sampling a lot of memory could be spared by making “large” packets, with a single timestamp and several associated data points. For example, something like:

struct FitnessData60 {
uint64_t timestamp;
uint8_t nbSteps[60];
uint8_t heartRate[60];
}

could store 60 samples (e.g. 1 full hour of 1m samples) in 128 bytes, compared to only 12 samples in 120 bytes. This means that 3KB could store a full day of data at 1m sampling rate. As a bonus, if the user is only interested in one of the values, the other field could be used to double the sampling rate within the same amount of memory (even more so if for heartRate we plan on storing more than one value, e.g. max, min and average => 4x sampling of steps when only storing that).

[ITCactus]

@infinitytec wait for the Gadgetbridge changelog to say that it has added support for it. InfiniTime and Gadgetbridge are two separate projects.

just in case, do you know if someone works on it? i don't see a pull-request or task for that, so i created the feature request for this...

update:
and as of 16-Dec, the step counter sync to Gadgetbridge on F-Droid is live...

[minacode]

I would like this feature very much, just because I am loosing my step count too often, because I miss syncing it before midnight.

Also, what about storing steps and heart rate in different data structures and enabling both independently? I do not care about heart rate and would like to only store steps. Having a struct with both inside would waste a lot of memory.

[LF-fr]

Interested in this feature too.

Some question on my side : if we don't upload every data, probably for HR we are interested in 3 values : lowest, average, highest.

Collecting data like HR every minutes doesn't make sens when doing nothing, but collecting lowest/highest/average values is interesting.

When doing sport or any other activity collecting each data is interesting.

Thus a mix of collecting & gathering data on a period (5/0/15/30/60 minutes) makes sens only if the 3 values are available. This should be activated for people who are interested.

For people who what to collect info during specific activities maybe apps can collect store and export larte more datas.

[watertrainer]

I think another problem with storing HeartRate data and step data in the same "batch" relative to one timestamp would be the frequency of each data aqquistion. You can't really pull the step number much less than once a minute, because then you'd risk the step number exceeding 512 (because the step number is always a multiple of 2 we can actually store up to 512 steps in 8 bits at max)
The user might however want to have their heart rate measured less often (or not at all as discussed), because heart rate measurments are quite power intenisve. So either this option of having less heart rate measurments would have to be ignored or we'd need to store heart rate and steps seperatly.

[watertrainer]

There is a bluetooth specification for how to transfer data like this, see https://www.bluetooth.com/specifications/specs/physical-activity-monitor-service-1-0/

[JF002]

There is a bluetooth specification for how to transfer data like this, see https://www.bluetooth.com/specifications/specs/physical-activity-monitor-service-1-0/

That's interesting, I didn't know a standard service existed for that kind of application. But... is this a "classical Bluetooth or a Bluetooth Low-Energy service?

[watertrainer]

I think it defines behaviour for both. In 3.2.1 (which is the only time they spell Bluetooth low energy) they define how to transfer data in multiple packages because ble size is too small if all fields are present (which they aren't in our case, but it seems that they had ble in mind)

[dozadoesit]

I just want to add That as a runner I also need to be able to determine in an exercise period not only my heart rate, and steps but distance travel. I recommend that we have this be an app in infinitime. Where you could choose running, set a goal, and then the watch will display steps, distances, and heart rate zones while your are exercising.

[stephanlachnit]

Seems like a duplicate of #214

[khimaros]

FYI there is some interest in this space at https://codeberg.org/Freeyourgadget/Gadgetbridge/issues/2383 and some work under way to expose the current heart rate data in Gadgetbridge here https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/3093

[jmlich]

btw I have also experimental branch of Amazfish which collects heart rate and steps data and shows them in graphs
piggz/harbour-amazfish#303

I have recently found the blog post about health tracking on AsteroidOS.
https://dodorad.io/blogs/2023-07-17:_Health_trackers,_a_non-trivial_problem.md

[ildar]

Greetings! I'm too late to this party yet wanted to share my ideas here: 1. @JF002 plz make a check list on top of what's done and what's planned. This thread is already quite long 2. I imagine that IT can be a great provider of health (and much more!) information. Privacy concerns are offtopic here. 3. On storing the values. 1. Storing only regular measured values. Mesure interval is configurable. In case of physical exercises the HR+steps should be sent *continuously* but not stored. It's very important to track HR during such activities. It's a good idea to turn the HR on forcibly when activity is detected (e.g. running) 2. I think that the ring-buffer in RAM is the best solution. The timestamp is good to be stored modified, e.g. divided by the interval and e.g. only the least significant 12 bit taken (for 24+hours with 10 sec interval). 3. If the IT sends the values above + current time ref and the interval value, then the companion app can easily calculate all the values in relative times and store true values in its DB. 4. Using standard BLE command etc. is a very good idea 4. For the possible values to be passed from IT to the companion app: 1. Status: off-hand, sleeping, sitting (low activity), moving, sporting. 2. Steps 3. HR 5. HR sensors we have (in P8) are of poor quality. That's why it's best to get 3-4 measures and get the average of the sane values (ignoring those which aren't in the line).