feat: add logic for parsing sleep data and viz panels

[k0rventen]

No description provided.

[fabiosangregorio]

I'm interested in this as well! 🤩
Would it be possible to also show sleep cycle data? (amount of Core, REM, Awake time, sleep efficiency %, etc) 👀

[k0rventen]

I'll try to finalise this first iteration this weekend !
Unfortunately my series 3 does not return as much sleep-related metrics as the latest ones, I'm limited by the fields I find in my exports..

But if you have a newer watch that does have those metrics, would you be willing to help out ? I'll see how we could export the fields names while ditching the actual metric for the sake of privacy.

that would be very cool to expose all those new metrics as well !

[fabiosangregorio]

I have an Apple Watch SE, and I would be willing to share and validate a sample of my sleep data with you!

[MarshalX]

yay! awesome!

[k0rventen]

yay! awesome!

@MarshalX yeah finally 😅. I'll test it out this week on other exports, and the next stable version with this feature should land next sunday. If you want to try it beforehand you can use the :rolling tag on the ingester image.
Regards !

[MarshalX]

@k0rventen i tried for the first time. rolling tag. i cant find anything about debug mode of debug logging so that's all what i can provide

throws after "Inserted 1760000 records"

apple-health-grafana-ingester-1  | Traceback (most recent call last):
apple-health-grafana-ingester-1  |   File "//app.py", line 189, in <module>
apple-health-grafana-ingester-1  |     process_health_data(client)
apple-health-grafana-ingester-1  |   File "//app.py", line 162, in process_health_data
apple-health-grafana-ingester-1  |     client.write_points(records, time_precision="s")
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 603, in write_points
apple-health-grafana-ingester-1  |     return self._write_points(points=points,
apple-health-grafana-ingester-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 681, in _write_points
apple-health-grafana-ingester-1  |     self.write(
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 413, in write
apple-health-grafana-ingester-1  |     self.request(
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 378, in request
apple-health-grafana-ingester-1  |     raise InfluxDBClientError(err_msg, response.status_code)
apple-health-grafana-ingester-1  | influxdb.exceptions.InfluxDBClientError: 413: {"error":"Request Entity Too Large"}

[MarshalX]

@k0rventen should i try decrease the batch from 10k to like 5k? is it safe to run the ingester a second time without dropping db?

upd. see "drop_database"
upd2. same with 5k
upd3. same with 1k
upd4. same with 100. just 100 xd

upd5.
okay i just added INFLUXDB_HTTP_MAX_BODY_SIZE=0 to env for influxdb service

[MarshalX]

Do i understand correctly that there is no sense in providing "All" in sleep sources as we have in "Apple health specific metrics"?

[MarshalX]

@k0rventen more info about the bug

apple-health-grafana-ingester-1  | Inserted 1740000 records
apple-health-grafana-ingester-1  | Inserted 1750000 records
apple-health-grafana-ingester-1  | Inserted 1760000 records
apple-health-grafana-ingester-1  | Total number of records: 3913329
apple-health-grafana-ingester-1  | All done! You can now check grafana.

well... i think batching is broken now. because format_record could return any list size. so we can't count +1 for each iteration.

at least if len(records) == 10000: should be changed to if len(records) >= 10000: and total_count += len(records) instead of total_count += 10000

and i hope that format_record can't return more than X records where X is the batch size (10k as default)

upd. yeah, that was the problem. sent a fix #23

[k0rventen]

@k0rventen i tried for the first time. rolling tag. i cant find anything about debug mode of debug logging so that's all what i can provide

throws after "Inserted 1760000 records"

apple-health-grafana-ingester-1  | Traceback (most recent call last):
apple-health-grafana-ingester-1  |   File "//app.py", line 189, in <module>
apple-health-grafana-ingester-1  |     process_health_data(client)
apple-health-grafana-ingester-1  |   File "//app.py", line 162, in process_health_data
apple-health-grafana-ingester-1  |     client.write_points(records, time_precision="s")
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 603, in write_points
apple-health-grafana-ingester-1  |     return self._write_points(points=points,
apple-health-grafana-ingester-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 681, in _write_points
apple-health-grafana-ingester-1  |     self.write(
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 413, in write
apple-health-grafana-ingester-1  |     self.request(
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.12/site-packages/influxdb/client.py", line 378, in request
apple-health-grafana-ingester-1  |     raise InfluxDBClientError(err_msg, response.status_code)
apple-health-grafana-ingester-1  | influxdb.exceptions.InfluxDBClientError: 413: {"error":"Request Entity Too Large"}

that's funny, because that's the exact bug I encountered once (and why I wanted to test more before releasing a new version 🐛 )

If there are other problems related to this feature I'll open a dedicated issue, rather than bumping a closed PR 😉