Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inaccurate sleep/wake recognition #232

Open
ghost opened this issue Feb 24, 2016 · 29 comments

Comments

Projects
None yet
9 participants
@ghost
Copy link

commented Feb 24, 2016

I'm not sure if Gadgetbridge processes raw data from Mi Band or just simply fetches it, but Mi Band doesn't recognise wakeup properly, it shows that my activity is 'light sleep' not 'activity'. There are sporadic short 'activity' graphs among sleep graphs, but no sign of fully 'waking up'. This inaccurate sleep detection is resulted in much longer sleep hour, such as 12 hours of sleep instead of 8 hours of actual sleep.

Or maybe it's because my Mi Band is broken. Any ideas?

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2016

@official-jkim we merely collect the raw data (minute samples) from the device and put them on the graphs.

We started collecting ideas / algorithms for interpreting the data in the following wiki page:
https://github.com/Freeyourgadget/Gadgetbridge/wiki/Activity-analysis

Any further ideas and contribution is much welcome.

See also #135 and other issues...

@Samarium151

This comment has been minimized.

Copy link

commented Feb 25, 2016

All of these devices are not particularly accurate for measurements of sleep. One big issue with them is the manufactures mostly implicitly over sell their accuracy. (heck rigorous checks of the pedometer functions also do not pan out well either. ) I added one reference to the wiki page above. Gadgetbridge documentation should in my opinion have some explanation of this.

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2016

I can happily provide more details on how raw* data is coming from the different devices (original miband and pebble health). That would be a description of how things are, not an explanation on how activity tracking works.

Thanks for contributing to our wiki! Do you think that what I proposed above could help you giving the explanation you have in mind?

*We usually refer to this as "raw" data but in fact they are already manipulated on the devices. For both miband and pebble we get per-minute aggregated data, for instance.

@ghost

This comment has been minimized.

Copy link
Author

commented Feb 25, 2016

I think my Mi Band is broken, as you said even Mi Band's raw data is manipulated on the device. Now it doesn't recognise 'wakeup' at all, recognises any activity as 'sleep' constantly. After even most intensive shake action and highest activity value, Mi Band recognises it as very short 'activity' and then go back to recognising it as 'light sleep'. Oh well, time to throw it away. Seems it's not a Gadgetbridge's fault.

@Samarium151

This comment has been minimized.

Copy link

commented Feb 26, 2016

I think that's a good start. I'll try to find some more objective stuff on the accuracy to reference. The manufacturers have no incentive to publish data that says there devices arent' accurate so it may be har dto come by. I'd be interested to see what data the devices are putting out as that would help in trying to figure out what if any analysis gadgetbridge should do with it.

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2016

I added a paragraph about raw data to the same page of the wiki.

@Samarium151

This comment has been minimized.

Copy link

commented Feb 27, 2016

Thanks that's really helpful. I did find some methods for using the raw actigraphy data.
It looks like the devices don't report enough for gadgetbridge to use it's own method, so may not be worth more time.

Gradeing of sleep via actigraphy alone is controvesial at best. For sleep no sleep accuracy often around 90% for a given epoch it's like spell check where even 99% accuracy can lead to lots of unreadable sentences. It's really hard to gauge the actual utility of a given method.

I suspect they are using Sadeh's algorithm as it's been around for a while, widely and is fairly simple to calculate on hardware like the MI band.:

SI(sleep index)( = 7.601 - 0.065 * mu - 0.056_sigma - 0.0703_LogAct- 1.08 nat

Where:
SI = sleep index (>1 is considered asleep)
mu = mean activity in an 11 minute window centered on the current epoch
sigma = standard deviation of activity for the last 6 min
LogAct= natural log of the activity in the current epoch increased by 1
nat = the number of epochs in the 11 minute window 50<= epoch activity <= 100

ref: Sadeh and Sharky Sleep 1994:17: 201-207

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2016

Really interesting! Thanks! Some post processing that happens for sure in other apps is the identification of deep sleep periods, are there known ways to calculate these? Currently we have no clean way to show deep sleep to the user because we only have punctual data, and adjacent deep sleep samples aren't good enough.
If you could share some algorithm I can apply it to the pebble health samples: on the watch I can see the deep sleep periods but they are not transferred to the phone (only the total amount of deep sleep seconds is transmitted, without temporal coordinates) so I could "easily" check if they use the same algo.

@ghost

This comment has been minimized.

Copy link
Author

commented Feb 28, 2016

img_20160229_083542

As you see, no wakeup detection, no sleep detection. If I try reboot(in debug menu), Mi Band starts tracking physical movements as 'activity' normally(green graph), but after one night, it goes back to sleep-only mode.

So, it's not Gadgetbridge's fault, it's just that my Mi Band is in limbo, right?

@Samarium151

This comment has been minimized.

Copy link

commented Mar 1, 2016

I will look for a method. There are no good validated methods I'm aware of for deep vs light sleep. Sleep stages are defined formally by EEG findings which no wearable collects. Apparently a common method is to look for periods of complete inactivity and call that REM. While a reasonable thought it has problems as some just don't move much in their sleep. Certain sleep disorders such as Sleep paralysis and rem behavior disorder violate these assumptions.

Is there an easy way to see a raw data file or can you post one here? I'm trying to understand better what exactly the devices are providing.

thanks

@ghost

This comment has been minimized.

Copy link
Author

commented Mar 4, 2016

I realised that it's neither Gadgetbridge's fault, nor my Mi Band's fault. It's Xiaomi's fault. I returned my Mi Band and got a new one, and now it's happening again. No wakeup(activity) data, only sleep(light/deep) data. Different band, same firmware version, same problem. I think latest firmware is not processing sleep/awake data properly.

@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2016

You could downgrade to a different version with Gadetbridge. There is very preliminary information how to do it at https://github.com/Freeyourgadget/Gadgetbridge/wiki/User-Documentation. For Mi Band 1A, firmware version 5.15.7.14 (MiFit 1.5.912-1539) works pretty well.

@ghost

This comment has been minimized.

Copy link
Author

commented Mar 4, 2016

Thanks, but I don't know where to get the firmware. It seems Mi Fit APK file itself contains a firmware, but I couldn't find neither 5.15.7.14 firmware nor 1.5.912-1539 Mi Fit app. Can you do me a favor?

@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented Mar 5, 2016

I can, but unfortunately not now, because I'm away for the weekend. I think you can find it on apkmirror.com, though.

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Mar 16, 2016

@official-jkim I deleted your comment because I believe there may be issues distributing firmware files. Further there is a bug on github that make shared files from forked repositories look like they are part of the original repo.

@computerlyrik

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2016

HR sensing is totally OK for my MiBand 1s

FW: 4.15.12.10
HR: 1.3.74.64

How can this improve sleep-detection?

cpfeiffer added a commit that referenced this issue Apr 3, 2016

Small fixes to PR 273 #232
- dynamically toggle hr sleep support when preference changes
- check hr support dynaically after device info is available to avoid false error message
@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2016

On my 1S, activity and sleep detection is totally bogus. HR did not work at all until I upgraded the firmware and the Mi Band firmware gave me > 23h of sleep per day. With my new firmware 4.15.11.20, things look a lot better.

cpfeiffer added a commit that referenced this issue Apr 4, 2016

Enable heart rate charts #232
(And fix some charting issues)
@0nse

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2016

“All of these devices are not particularly accurate for measurements of sleep”@Samarium151

I was really surprised to see just how accurate the MiFit app recognises my sleep lengths. When using the official app for a bit more than the last two months, I never had an incident, where the “fell asleep”- and “wake up”-times were wrong.

“This inaccurate sleep detection is resulted in much longer sleep hour, such as 12 hours of sleep instead of 8 hours of actual sleep.” — @official-jkim

However, I have witnessed the opposite with Gadgetbridge. The software will display me as being awake long after falling asleep and long before actually waking up. Thus, I get much fewer sleeping hours per app as I actually do.

I'm wearing my tracking band (MiFit 1S) quite loosely.

@cpfeiffer cpfeiffer added the device mi label Apr 10, 2016

@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2016

@0nse That's interesting results. Which firmware version do you use? The version that came with my 1S was utterly broken, it reported > 23h sleep per day.

But if you use the same device and firmware both with Mi Fit and Gadgetbridge, and the results are really different, then I'd love to analyze that.

Gadgetbridge has an option in the Mi Band settings to keep activity data on the device. If you enable that, you could

  • sleep a night
  • then fetch activity data with Gadgetbridge
  • then fetch activity data with Mi Fit
    and compare the results. I'd love to hear them.
@0nse

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2016

  • FW: 4.15.12.10
  • HR: 1.3.74.64

I will report back when I have the results of the same night by both apps.

Edit: For some reason, I cannot get MiFit and GB to both read out the info although I have set GB to keep the data on the device and I open up GB before MiFit. What happens most of the time is that neither of the two apps will be able to fetch any data. The update to MiFit 2.0 didn't make things any better. So for now, I cannot report any comparisons.

I can only say that GB will already think that I am awake when I am in very light sleep where MiFit would ignore this little light sleep peak. Here's an example with Sleep as Android as reference.
screenshot_20160415-120412
I guess this correlates with the MiBand ignoring sleep when it happens anytime but at night. That is, neither MiFit nor GB would detect sleep patterns (although one can clearly recognise them when looking at the activity graph) when sleep started in the morning, which is bad for people working night shifts.

The image also shows that GB will irregularly recognise my band as being not worn although that is false.

@danielegobbetti

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2016

Since a few weeks we have two helper functions that allow for changing the stored sample type. See this method and the one that follows.
The former overwrites any stored type with the desired one, the latter changes only from a specific sample type to another (use case: change from "awake" to "sleeping" but preserve eventual "deep sleep" samples.

The GUI to let users change their samples is still missing, though.

@v1r0x

This comment has been minimized.

Copy link

commented May 8, 2016

Just wanted to share my results with you. I experience similar results. E.g. last night I slept for around 7 hours but GB only shows around 2 hours. In general I get far less sleep time in GB as in MiFit. I'll try to track one night with GB and MiFit. Is there any good method to provide the results beside the activity graphs?

@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented May 8, 2016

Some notes:

  1. always provide the mi band fw version, and if you have a heart rate sensor, then indicate whether you enabled the sensor to measure during sleep or not.

  2. Also check if the settings are correct (e.g. if you wear it on the left or right wrist).

  3. do you wear it loosely or have it tight?

  4. so far, we don't do any post processing of the data that the band provides. Maybe we need to do that.

The data measured with my 1S with fw version 4.5.12.10 look pretty realistic to me, wearing the band tightly.

@cpfeiffer

This comment has been minimized.

Copy link
Contributor

commented May 8, 2016

@v1r0x what kind of activity does GB show when it's not sleep? Green activity? Also, did you use the same data for GB and mifit by syncing with GB first, but keeping the data on the band?

@v1r0x

This comment has been minimized.

Copy link

commented May 8, 2016

  1. Sorry, forgot to mention my hard- and software specs. I have the Mi1S (FW 4.16.3.7) and use the HR sensor for sleep improvement
  2. Is set correctly
  3. Rather loosely, I tried it a bit tighter but that doesn't seem to improve measurements.

I'll try to downgrade the FW after doing some tests with MiFit. Maybe the FW you mentioned works better with GB.

@v1r0x what kind of activity does GB show when it's not sleep? Green activity? Also, did you use the same data for GB and mifit by syncing with GB first, but keeping the data on the band?

Yes, green activity. I didn't compare GB and MiFit, but the results from MiFit seemed to be more precise before I switched to GB.

@walkjivefly

This comment has been minimized.

Copy link
Contributor

commented Jan 11, 2017

Sleep records from Mi Band 2 significantly different than what Mi Fit says they are.

I've been doing an experiment with both Gadgetbridge (0.16 self-compiled) and Mi Fit (2.2.9) connected to the same Mi Band 2 (HW v0.1.3.2, FW v1.0.1.28). I know the Wiki says this doesn't work but it seems to work for me. The sleep records from both apps are significantly different. I am careful to not use either app after going to bed, and try to remember to sync first Gadgetbridge followed by Mi Fit in the morning. Gadgetbridge is configured to leave data on the device. I got the order wrong 5 mornings out of 14 but None of the numbers match for the other days. For about 3 days they are vaguely in the same ballpark after allowing for Gadgetbridge having the "light" and "dark" categories the wrong way round.

Compare and contrast
gb_sleep
mifit_sleep

These are OOcalc generated from the following data (manually extracted from both apps)
image

The temporal distribution of light and deep sleep each night varies even more between both apps which makes me wonder if Gadgetbridge interpretation of the "raw" data is correct, or if Mi Fit does additional processing.

@Vebryn Vebryn referenced this issue May 11, 2017

Closed

Sleep not detected correctly #650

2 of 2 tasks complete
@JonasT

This comment has been minimized.

Copy link

commented Jul 1, 2018

I have the same issue, and the HR is between 60-90 with the HR detection used for sleep state enabled - still it shows as deep sleep(!) which I think given the HR is kind of obviously not true...?

...unless I'm dreaming this right now and not actually typing a GitHub comment 😄

FW: 1.0.1.53, HW: V0.1.3.3 (Xiaomi Mi Band 2), Gadgetbridge 0.26.5 (f-droid)

Now as I'm typing this, my HR seems to have reached 100 though and it went from graphing deep sleep right into activity (even though I am sitting and typing, not even standing up). The pulse looks kind of correct to me, but the sleep thing is just completely off.

@Dikay900

This comment has been minimized.

Copy link
Contributor

commented Jul 26, 2018

dropping #686 since this seems to be same problem so we can find related issues

@JonasT

This comment has been minimized.

Copy link

commented Jul 26, 2018

Edit: oops, ignore this, mixed up the tickets. 😆

@Dikay900 Dikay900 referenced this issue Jul 27, 2018

Open

Improvements to sleep detection #1

2 of 12 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.