Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document how to set up Nightscout #129

Closed
bewest opened this issue Apr 27, 2016 · 19 comments

Comments

Projects
None yet
5 participants
@bewest
Copy link
Member

commented Apr 27, 2016

Per updates to http://openaps.readthedocs.org/en/latest/docs/walkthrough/phase-1/visualization.html

Once Nightscout is running how do we send data into it?

Let's try nightscout:

$ nightscout -h
  Usage:
nightscout <cmd>

* latest-openaps-treatment
* cull-latest-openaps-treatments

* get
* upload
* dedupe-treatments
* hash-api-secret
* status
* upload-entries
* autoconfigure-device-crud


The autoconfigure-device-crud feature will allow us to create an easy to use ns device:

$ nightscout autoconfigure-device-crud https://my.nightscout.host averylongplainsecret
added process://ns/nightscout/ns NIGHTSCOUT_HOST API_SECRET
process://ns/nightscout/ns https://my.nightscout.host e6fc892e8e88235a476d197de3dfbef3f2db53d0

It added a new ns device to our uses menu:

openaps use ns shell get entries.json 'count=10'
openaps use ns shell upload treatments.json recently/combined-treatments.json

So we now have various uses for ns: get, upload, latest-treatment-time, format-recent-history-treatments, upload-non-empty-treatments.

Get records from Nightscout

Use the get feature which takes two arguments: the name of the endpoint (entries, devicestatus, treatments, profiles) and any query arguments to append to the argument string. 'count=10' is a reasonable debugging value.

openaps use ns shell get $endpoint $query-params

Unifying pump treatments in Nightscout

To upload treatments data to Nightscout, prepare you zoned glucose, and pump model reports, and use the following two reports:

openaps report add nightscout/recent-treatments.json JSON ns shell  format-recent-history-treatments monitor/pump-history.json model.json
openaps report add nightscout/uploaded.json JSON  ns shell upload-non-empty-treatments  nightscout/recent-treatments.json

Here are the equivalent uses:

openaps use ns shell format-recent-history-treatments monitor/pump-history.json model.json
openaps use ns shell upload-non-empty-treatments nightscout/recent-treatments.json

The first report runs the format-recent-history-treatments use, which fetches data from Nightscout and determines which of the latest deltas from openaps need to be sent. The second one uses the upload-non-empty-treatments use to upload treatments to Nightscout, if there is any data to upload.

@bewest bewest added the help wanted label Apr 27, 2016

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 1, 2016

oref0_glucose

New dexcom use to format:

# with pre-existing data
openaps use cgm oref0_glucose  --glucose raw-cgm/glucose-raw.json --sensor sensor_raw.json
# without raw data
openaps use cgm oref0_glucose --no-raw  --glucose raw-cgm/glucose-raw.json
# with fresh/new data:
openaps use cgm oref0_glucose --no-raw
# or, by default it will get 1 hour of data
openaps use cgm oref0_glucose .20
openaps use cgm oref0_glucose 24

Make sure to save the report to eg raw-cgm/entries.json and ensure to the use tz rezone --date dateString --date display_time and save the report as eg monitor/glucose.json or similar.

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 2, 2016

 2085  openaps report show --json raw-cgm/glucose-raw.json 
 2086  openaps use tz select --prev "2016-05-02T11:44:10-07:00" some-glucose-zoned.json  | json
 2087  openaps use tz select --prev "2016-05-02T11:44:10-07:00" some-glucose-zoned.json  | tee selected.json
 2088  cat selected.json 
 2089  openaps us ns shell upload entries.json selected.json
 2090  openaps use ns shell upload entries.json selected.json
 2091  openaps report show monitor/glucose
 2092  openaps report show monitor/glucose.json 
 2093  openaps report show monitor/glucose.json  --json
 2094  openaps report show monitor/glucose.json  --ini
 2095  openaps report show raw-cgm/glucose-raw.json  --ini
 2096  openaps report add raw-cgm/glucose-raw.json  cgm oref0_glucose --hours  2
 2097  openaps report add raw-cgm/glucose-raw.json JSON  cgm oref0_glucose --hours  2
 2098  openaps report show raw-cgm/glucose-raw.json  --ini
 2099  ls nightscout/
 2100  history
 2101  openaps use ns shell upload 
 2102  openaps use ns shell lsgaps tz 
 2103  openaps use ns shell lsgaps tz  |  json -a prev current
 2104  openaps invoke raw-cgm/glucose-raw.json monitor/glucose.json
 2105  openaps report invoke raw-cgm/glucose-raw.json monitor/glucose.json
 2106  openaps use ns shell lsgaps tz entries  |  openaps use tz select --gaps - monitor/glucose.json 
 2107  openaps use ns shell lsgaps tz entries  |  openaps use tz select --gaps - monitor/glucose.json  | json -a dateString
 2108  openaps use ns shell lsgaps tz entries  |  openaps use tz select --gaps - monitor/glucose.json  | json -a dateString |wc -l
 2109  openaps use ns shell format-recent-type tz entries monitor/glucose.json 
 2110  openaps use ns shell format-recent-type tz entries monitor/glucose.json  | json -a dateString | wc -l
 2111  openaps report invoke raw-cgm/glucose-raw.json monitor/glucose.json
 2112  openaps use ns shell format-recent-type tz entries monitor/glucose.json  | json -a dateString | wc -l
 2113  openaps report add nightscout/recent-missing-entries.json JSON ns shell format-recent-type tz entries monitor/glucose.json  
 2114  openaps report invoke nightscout/recent-missing-entries.json
 2115  openaps report add nightscout/uploaded-entries.json JSON  ns shell upload entries.json nightscout/recent-missing-entries.json 
 2116  openaps report invoke nightscout/recent-missing-entries.json 
 2117  openaps report invoke nightscout/recent-missing-entries.json
 2118  cat nightscout/recent-missing-entries.json 
 2119  openaps report invoke nightscout/recent-missing-entries.json
 2120  cat nightscout/recent-missing-entries.json  | json -a dateString | wc -l
 2121  openaps report invoke nightscout/uploaded-entries.json 
 2122  openaps report invoke nightscout/recent-missing-entries.json
 2123  cat nightscout/recent-missing-entries.json  | json -a dateString | wc -l
 2124  history


@bewest

This comment has been minimized.

Copy link
Member Author

commented May 2, 2016

+++ b/openaps.ini
@@ -17,7 +17,7 @@ ping = ! bash -c "echo PONG! $*" --
 pong = ! echo PONG
 hello = ! echo hello
 okbar = ! bash -c "echo $1" --
-report-nightscout = report invoke  nightscout/recent-treatments.json nightscout/uploaded.json
+report-nightscout = report invoke nightscout/recent-treatments.json nightscout/uploaded.json ni

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 3, 2016

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 12, 2016

These uses require openaps-utils with lsgaps and select.

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 12, 2016

Preferably, version of 0.0.9 of openaps-contrib (pip install openaps-contrib), which was just released.

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 16, 2016

Also depends on oref0 0.2.0.

@baughaw

This comment has been minimized.

Copy link

commented May 17, 2016

Here is some sample glucose data to use,
glucose-raw.json.txt

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 20, 2016

Use mm-format-ns-glucose.

@baughaw

This comment has been minimized.

Copy link

commented May 20, 2016

Is there any chance of mm-format-ns-glucose calculating the trend arrows? I am assuming it is just the difference of the current glucose value and the previous.

@scottleibrand

This comment has been minimized.

Copy link
Contributor

commented May 20, 2016

Trend arrows on the Dexcom are calculated based on the 15-minute rate of change in sensor values. You could certainly add some logic to calculate them similarly using Medtronic data.

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 28, 2016

Please help contribute by adding these notes, perhaps supplemented by your own to the docs. 😺

@bewest

This comment has been minimized.

Copy link
Member Author

commented May 29, 2016

nightscout tools in openaps

openaps use ns shell get entries.json 'count=10'
openaps use ns shell upload treatments.json recently/combined-treatments.json
  -h                                  This message.
  get type args                                  Get records of type from
                                                 Nightscout matching args.

  upload endpoint file                           Upload a file to the Nightscout endpoint.
  latest-treatment-time                          - get latest treatment time from Nightscout
  format-recent-history-treatments history model - Formats medtronic pump
                                                 history and model into
                                                 Nightscout compatible
                                                 treatments.

  format-recent-type ZONE type file              - Selects elements from the
                                                 file where the elements would
                                                 satisfy a gap in the last 1000
                                                 Nightscout records.

  upload-non-empty-treatments file               - Upload a non empty treatments
                                                 file to Nightscout.
  lsgaps tz entries                              - Re-use openaps timezone device
                                                 to find gaps in a type (entries)
                                                 by default.
  upload-non-empty-type type file
  status                                         - Retrieve status
  preflight                                      - NS preflight

Nightscout Endpoints

  • entries.json - Glucose values, mbgs, sensor data.
  • treatments.json - Pump history, bolus, treatments, temp basals.
  • devicestatus.json - Battery levels, reservoir.
  • profile.json - Planned rates/settings/ratios/sensitivities.
  • status.json - Server status.

Examples

Get records from Nightscout

Use the get feature which takes two arguments: the name of the endpoint
(entries, devicestatus, treatments, profiles) and any query arguments to append
to the argument string. 'count=10' is a reasonable debugging value.
The query-params can be used to generate any query Nightscout can respond to.

openaps use ns shell get $endpoint $query-params

Unifying pump treatments in Nightscout

To upload treatments data to Nightscout, prepare you zoned glucose, and pump
model reports, and use the following two reports:

openaps report add nightscout/recent-treatments.json JSON ns shell  format-recent-history-treatments monitor/pump-history.json model.json
openaps report add nightscout/uploaded.json JSON  ns shell upload-non-empty-treatments  nightscout/recent-treatments.json

Here are the equivalent uses:

openaps use ns shell format-recent-history-treatments monitor/pump-history.json model.json
openaps use ns shell upload-non-empty-treatments nightscout/recent-treatments.json

The first report runs the format-recent-history-treatments use, which fetches
data from Nightscout and determines which of the latest deltas from openaps
need to be sent. The second one uses the upload-non-empty-treatments use to
upload treatments to Nightscout, if there is any data to upload.

Uploading glucose values to Nightscout

Format potential entries (glucose values) for Nightscout.

openaps use ns shell format-recent-type tz entries monitor/glucose.json  | json -a dateString | wc -l
# Add it as a report
openaps report add nightscout/recent-missing-entries.json JSON ns shell format-recent-type tz entries monitor/glucose.json  
# fetch data for first time
openaps report invoke nightscout/recent-missing-entries.json

# add report for uploading to NS
openaps report add nightscout/uploaded-entries.json JSON  ns shell upload entries.json nightscout/recent-missing-entries.json 
# upload for fist time.
openaps report invoke nightscout/uploaded-entries.json
@bewest

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

Combined with #132 (comment), here's how I did the status pill:

# add ns-status as a device:
oref0 device-helper ns-status 'ns-status $*' | openaps import
# try using new ns-status device:
openaps use ns-status shell monitor/clock.json oref0-monitor/iob.json oref0-predict/oref0.json oref0-enacted/enacted-temp-basal.json monitor/battery.json monitor/reservoir.json  monitor/status.json
# add as a report:
openaps report add nightscout/openaps-status.json JSON ns-status shell monitor/clock.json oref0-monitor/iob.json oref0-predict/oref0.json oref0-enacted/enacted-temp-basal.json monitor/battery.json monitor/reservoir.json  monitor/status.json 
# pull data in first time to help prime the system
openaps report invoke nightscout/openaps-status.json

# test uploading device status
openaps use ns shell upload  devicestatus.json nightscout/openaps-status.json
# add as report
openaps report add nightscout/uploaded-recent-devicestatus.json JSON ns shell upload  devicestatus.json nightscout/openaps-status.json

# add report-nightscout alias
openaps alias add report-nightscout "report invoke nightscout/preflight.json nightscout/recent-treatments.json nightscout/uploaded.json nightscout/recent-missing-entries.json nightscout/uploaded-entries.json nightscout/openaps-status.json nightscout/uploaded-recent-devicestatus.json"


bewest added a commit to bewest/docs that referenced this issue Jun 5, 2016

new-style recommendations
Mainly removing some older cruft, importing some docs from
openaps#129 and bringing in
openaps#219.

bewest added a commit to bewest/docs that referenced this issue Jun 6, 2016

@bewest

This comment has been minimized.

Copy link
Member Author

commented Jun 28, 2016

openaps use ns shell get entries.json 'count=10'
openaps use ns shell upload treatments.json recently/combined-treatments.json
  -h                                  This message.
  get type args                                  Get records of type from
                                                 Nightscout matching args.

  oref0_glucose [tz] [args]                      Get records matching oref0
                                                 requirements according to args
                                                 from Nightscout.
                                                 tz should be the name of the
                                                 timezones device (default with
                                                 no args is tz).
                                                 args are ampersand separated
                                                 arguments to append to the
                                                 search query for Nightscout.
  oref0_glucose_without_zone [args]              Like oref0_glucose but without
                                                 rezoning.
  upload endpoint file                           Upload a file to the Nightscout endpoint.
  latest-treatment-time                          - get latest treatment time from Nightscout
  format-recent-history-treatments history model - Formats medtronic pump
                                                 history and model into
                                                 Nightscout compatible
                                                 treatments.

  format-recent-type ZONE type file              - Selects elements from the
                                                 file where the elements would
                                                 satisfy a gap in the last 1000
                                                 Nightscout records.

  upload-non-empty-treatments file               - Upload a non empty treatments
                                                 file to Nightscout.
  lsgaps tz entries                              - Re-use openaps timezone device
                                                 to find gaps in a type (entries)
                                                 by default.
  upload-non-empty-type type file
  status                                         - ns-status
  get-status                                     - status - get NS status
  preflight                                      - NS preflight

Nightscout Endpoints

  • entries.json - Glucose values, mbgs, sensor data.
  • treatments.json - Pump history, bolus, treatments, temp basals.
  • devicestatus.json - Battery levels, reservoir.
  • profile.json - Planned rates/settings/ratios/sensitivities.
  • status.json - Server status.

Examples

Get records from Nightscout

Use the get feature which takes two arguments: the name of the endpoint
(entries, devicestatus, treatments, profiles) and any query arguments to append
to the argument string. 'count=10' is a reasonable debugging value.
The query-params can be used to generate any query Nightscout can respond to.

openaps use ns shell get $endpoint $query-params

Unifying pump treatments in Nightscout

To upload treatments data to Nightscout, prepare you zoned glucose, and pump
model reports, and use the following two reports:

openaps report add nightscout/recent-treatments.json JSON ns shell  format-recent-history-treatments monitor/pump-history.json model.json
openaps report add nightscout/uploaded.json JSON  ns shell upload-non-empty-treatments  nightscout/recent-treatments.json

Here are the equivalent uses:

openaps use ns shell format-recent-history-treatments monitor/pump-history.json model.json
openaps use ns shell upload-non-empty-treatments nightscout/recent-treatments.json

The first report runs the format-recent-history-treatments use, which fetches
data from Nightscout and determines which of the latest deltas from openaps
need to be sent. The second one uses the upload-non-empty-treatments use to
upload treatments to Nightscout, if there is any data to upload.

Uploading glucose values to Nightscout

Format potential entries (glucose values) for Nightscout.

openaps use ns shell format-recent-type tz entries monitor/glucose.json  | json -a dateString | wc -l
# Add it as a report
openaps report add nightscout/recent-missing-entries.json JSON ns shell format-recent-type tz entries monitor/glucose.json  
# fetch data for first time
openaps report invoke nightscout/recent-missing-entries.json

# add report for uploading to NS
openaps report add nightscout/uploaded-entries.json JSON  ns shell upload entries.json nightscout/recent-missing-entries.json 
# upload for fist time.
openaps report invoke nightscout/uploaded-entries.json
@bewest

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2016

openaps report show --json | json -g | json -c "this[this.name].device == 'ns'" | json -a name | while read report ; do echo "## report $report"; echo ; echo 'use:' ; echo '```'; openaps report show $report --cli ; echo '```'; echo ; echo;  done

report nightscout/recent-treatments.json

use:

openaps use ns shell format-recent-history-treatments monitor/pump-history.json model.json

report nightscout/uploaded.json

use:

openaps use ns shell upload-non-empty-treatments nightscout/recent-treatments.json

report nightscout/recent-missing-entries.json

use:

openaps use ns shell format-recent-type tz entries monitor/glucose.json

report nightscout/uploaded-entries.json

use:

openaps use ns shell upload-non-empty-type entries.json nightscout/recent-missing-entries.json

report nightscout/preflight.json

use:

openaps use ns shell preflight

report nightscout/uploaded-recent-devicestatus.json

use:

openaps use ns shell --not-json-default upload devicestatus.json nightscout/openaps-status.json
@bewest

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2016

@mwilliams

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

Having issues. Guessing it's because it's data from my enlite CGM.

I'm using tool to convert the minimed data, the output of openaps use ns shell format-recent-type tz entries monitor/glucose.json | json -a dateString | wc -l is simply an empty collection, [].

Just running the json -a dateString against my converted glucose file does show content:

2016-09-08T20:57:00-0400
2016-09-08T20:52:00-0400
2016-09-08T20:47:00-0400
2016-09-08T20:42:00-0400
2016-09-08T20:44:00-0400
2016-09-08T20:39:00-0400

Therefore, my nightscout/recent-missing-entries.json report ends up empty.

Here's sample content from my mm-glucose.json

[
  {
    "packet_size": 0,
    "name": "DataEnd",
    "date": 1473382620000,
    "date_type": "none",
    "_tell": 1,
    "op": "0x01",
    "dateString": "2016-09-08T20:57:00-0400",
    "type": "pumpdata",
    "device": "openaps://medtronic/pump/cgm"
  },
  {
    "name": "GlucoseSensorData",
    "date_type": "prevTimestamp",
    "_tell": 2,
    "sgv": 106,
    "date": 1473382320000,
    "packet_size": 0,
    "op": 53,
    "dateString": "2016-09-08T20:52:00-0400",
    "type": "sgv",
    "device": "openaps://medtronic/pump/cgm"
  },
  {
    "name": "GlucoseSensorData",
    "date_type": "prevTimestamp",
    "_tell": 3,
    "sgv": 104,
    "date": 1473382020000,
    "packet_size": 0,
    "op": 52,
    "dateString": "2016-09-08T20:47:00-0400",
    "type": "sgv",
    "device": "openaps://medtronic/pump/cgm"
  },
  {
    "packet_size": 6,
    "name": "SensorCalFactor",
    "raw": "94-6a-08-10-1a-af",
    "date": 1473381720000,
    "factor": 6.831,
    "date_type": "minSpecific",
    "_tell": 4,
    "op": "0x0f",
    "dateString": "2016-09-08T20:42:00-0400",
    "type": "pumpdata",
    "device": "openaps://medtronic/pump/cgm"
  },
  {
    "name": "GlucoseSensorData",
    "date_type": "prevTimestamp",
    "_tell": 11,
    "sgv": 104,
    "date": 1473381840000,
    "packet_size": 0,
    "op": 52,
    "dateString": "2016-09-08T20:44:00-0400",
    "type": "sgv",
    "device": "openaps://medtronic/pump/cgm"
  },
  {
    "name": "GlucoseSensorData",
    "date_type": "prevTimestamp",
    "_tell": 12,
    "sgv": 92,
    "date": 1473381540000,
    "packet_size": 0,
    "op": 46,
    "dateString": "2016-09-08T20:39:00-0400",
    "type": "sgv",
    "device": "openaps://medtronic/pump/cgm"
  }
]

I'm running from your alias's as well, my do-everything runs into:

openaps report invoke oref0-predict/oref0.json
determine-basal://text/shell/oref0-predict/oref0.json

{"iob":0.008,"activity":-0.0015,"bolussnooze":0,"basaliob":0.008,"netbasalinsulin":0.15,"hightempinsulin":0.15,"time":"2016-09-08T19:51:38.000Z"}
{"delta":null,"glucose":null,"short_avgdelta":null,"long_avgdelta":null}
{"duration":30,"rate":0,"temp":"absolute"}
{"max_iob":5,"type":"current","max_daily_safety_multiplier":3,"current_basal_safety_multiplier":4,"autosens_max":1.5,"autosens_min":0.7,"autosens_adjust_targets":true,"override_high_target_with_low":false,"bolussnooze_dia_divisor":2,"min_5m_carbimpact":3,"carbratio_adjustmentratio":1,"dia":2,"model":{},"current_basal":2.6,"basalprofile":[{"i":0,"start":"00:00:00","rate":1.1500000000000001,"minutes":0},{"i":1,"start":"04:30:00","rate":1.55,"minutes":270},{"i":2,"start":"08:00:00","rate":2.6,"minutes":480},{"i":3,"start":"15:00:00","rate":2.6,"minutes":900},{"i":4,"start":"21:00:00","rate":1.4000000000000001,"minutes":1260}],"max_daily_basal":2.6,"max_basal":3,"out_units":"mg/dL","min_bg":110,"max_bg":145,"sens":16}

Adjusting target_bg from 127.5 to 128
Error: could not calculate eventualBG
reporting oref0-predict/oref0.json

I think i've stared at things for too long - no data is making it to my NS instance no matter how I slice it :). Any thoughts that could be applied to improving this ticket?

@danamlewis

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2016

I believe this is resolved between the new setup scripts and the new Nightscout setup instruction page (http://openaps.readthedocs.io/en/dev/docs/walkthrough/phase-1/nightscout-setup.html). Closing for now, someone can reopen if we find it's not fully resolved.

@danamlewis danamlewis closed this Oct 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.