-
Notifications
You must be signed in to change notification settings - Fork 46
Tableau API
There are two API endpoints: the Tableau WDC endpoint and the JSON endpoint. Both use:
- API keys: These are different from the API keys used to download data. Generate an API key by going to the Manage Credentials page, then clicking "Generate A New API Key" under "Manage Tableau Credentials", and copying the Access Key and Secret Key that appear at the top of the page. The Secret Key will never be displayed again, so make sure you copy it somewhere!
-
Study IDs: The way you reference a study (
<study_object_id>) is the 24-character random string visible on the View Study page, right under the study's name.
Path: https://<server>/api/v0/studies/<study_object_id>/summary-statistics/daily/wdc
This is designed to only be accessed from Tableau.
To set up a web data connector in Tableau, click on "More" under "To a Server" in the "Connect" menu.
Then, enter the URL corresponding to your study, substituting and <study_object_id>.
For example, to access the data for a study with ID "4J9BnPs2Y8T6EuNXhcYnMOaX" on the studies deployment, you would use:
https://studies.beiwe.org/api/v0/studies/4J9BnPs2Y8T6EuNXhcYnMOaX/summary-statistics/daily/wdc
After entering this URL, you will need to enter your Tableau Access Key and Secret Key.
Path: /api/v0/studies/<study_object_id>/summary-statistics/daily
You must pass your Access Key and Secret Key in via the headers X-Access-Key-Id and X-Access-Key-Secret.
You can call this endpoint using cURL. For the studies.beiwe.org deployment, here's how you'd pull all summary statistics from an entire study and save then in a json file:
curl "https://studies.beiwe.org/api/v0/studies/<study_object_id>/summary-statistics/daily" -H "X-Access-Key-Id: abcd..." -H "X-Access-Key-Secret: efgh..." > my_filename.json
Note that using > my_filename.json stores the output of the cURL in a file rather than printing them to your terminal.
It is possible to complete curl requests in a variety of languages, such as R, Python, and Matlab. This website is a helpful tool for getting started.
You can filter by date, participant ID, and which fields you want. This will make the query run faster. If the API query takes longer than 60 seconds to run, you'll get a 504 timeout error instead of a response. Add filters as GET parameters to the URL, using standard URL parameter format (add the first param to the URL like this: ?<parameter_name>=<parameter_value>, and all subsequent parameters like this: &<parameter_name>=<parameter_value>.
-
Start date:
start_date. Must be in YYYY-MM-DD format. Example:?start_date=2021-03-15 -
End date:
end_date. Must be in YYYY-MM-DD format. Example:?end_date=2021-04-01 -
Participant IDs:
participant_ids. Comma-separated list, with no other characters wrapping the list or each item. Example for two participant IDs:?participant_ids=ouq7r382,r3h9qp2o. -
Fields:
fields. Comma-separated list. Example:&fields=date,participant_id,beiwe_accelerometer_bytes,jasmine_distance_from_home -
Limit:
limit. An integer value. The query will return only the first N matches of results, where N is the limit.
Example usage: let's say we want to get data from the study with ID "R69zae1Y7Lw6yuwVUR4BOALY", for March 15, 2021 through April 1, 2021, for participant "ouq7r382", and we only want the values of "beiwe_accelerometer_bytes" and "jasmine_distance_from_home". Here's how to make that API call with cURL:
curl "https://studies.beiwe.org/api/v0/studies/R69zae1Y7Lw6yuwVUR4BOALY/summary-statistics/daily?start_date=2021-03-15&end_date=2021-04-01&participant_ids=ouq7r382&fields=date,participant_id,beiwe_accelerometer_bytes,jasmine_distance_from_home" -H "X-Access-Key-Id: abcd..." -H "X-Access-Key-Secret: efgh..." > my_filename.json
You can also use filtering with the Tableau web data connector (WDC). Just add the filtering fields after the end of the URL, as in:
https://studies.beiwe.org/api/v0/studies/4J9BnPs2Y8T6EuNXhcYnMOaX/summary-statistics/daily/wdc?fields=date,participant_id,beiwe_accelerometer_bytes
After storing data in a .json file, you can connect to the file from Tableau. Select .json under "from a file", then navigate to your json file.
For complete documentation on how to use the API endpoint, copy the file beiwe-backend/api/tableau_api/spec.yaml and paste it into any OpenAPI viewer, like
Swagger Editor.
The JSON API and the Tableau WDC API should return the same data; the WDC API is just a wrapper on the JSON API.
The JSON API returns a list of dictionaries, where each dictionary represents the summary statistics for one participant on one day. If a dictionary doesn't get returned, it means there are no summary statistics data for that participant on that day.
If a summary statistic is null, that means it's either 0 or not calculated.
There are two types of summary statistics:
- Forest-generated statistics: these are calculated by running Forest, and include interesting statistics like "maximum distance from home" and "number of incoming phone calls".
-
Data quantity/volume statistics: these are calculated automatically, whether or not Forest is run. These tell you the number of bytes of each type of data, for each day.
- Each statistic is the integer number of bytes of decrypted data.
- A "day" is defined as midnight to midnight in the study's time zone. So for US Eastern Daylight Time, that'll be 4am UTC one day to 4am UTC the next. (When you download the data, it's all timestamped in UTC.)
- This is based on the quantity of batched data, not raw data. So if data has been uploaded from the phone but not yet batched (maybe the data processing/batching script crashed), it won't be reflected in this total. But as soon as data gets batched, it should show up in the data quantity statistics.
Example: here's data the API would return on one participant, for two days:
[{'date': '2021-09-28',
'participant_id': 'ouq7r382',
'study_id': 'R69zae1Y7Lw6yuwVUR4BOALY',
'beiwe_accelerometer_bytes': 29003797,
'gps_bytes': 836203,
'distance_traveled': 179.668,
'home_duration': 14.4905700000127,
...},
{'date': '2019-09-29',
'participant_id': 'ouq7r382',
'study_id': 'R69zae1Y7Lw6yuwVUR4BOALY',
'accelerometer_bytes': 15627650,
'gps_bytes': 337579,
'distance_traveled': 4.007,
'home_duration': 4.27461527115769,
...}]
date
participant_id
study_id
beiwe_accelerometer_bytes
beiwe_ambient_audio_bytes
beiwe_app_log_bytes
beiwe_bluetooth_bytes
beiwe_calls_bytes
beiwe_devicemotion_bytes
beiwe_gps_bytes
beiwe_gyro_bytes
beiwe_identifiers_bytes
beiwe_image_survey_bytes
beiwe_ios_log_bytes
beiwe_magnetometer_bytes
beiwe_power_state_bytes
beiwe_proximity_bytes
beiwe_reachability_bytes
beiwe_survey_answers_bytes
beiwe_survey_timings_bytes
beiwe_texts_bytes
beiwe_audio_recordings_bytes
beiwe_wifi_bytes
jasmine_distance_diameter
jasmine_distance_from_home
jasmine_distance_traveled
jasmine_flight_distance_average
jasmine_flight_distance_stddev
jasmine_flight_duration_average
jasmine_flight_duration_stddev
jasmine_gps_data_missing_duration
jasmine_home_duration
jasmine_gyration_radius
jasmine_significant_location_count
jasmine_significant_location_entropy
jasmine_pause_time
jasmine_obs_duration
jasmine_obs_day
jasmine_obs_night
jasmine_total_flight_time
jasmine_av_pause_duration
jasmine_sd_pause_duration
willow_incoming_text_count
willow_incoming_text_degree
willow_incoming_text_length
willow_outgoing_text_count
willow_outgoing_text_degree
willow_outgoing_text_length
willow_incoming_text_reciprocity
willow_outgoing_text_reciprocity
willow_outgoing_MMS_count
willow_incoming_MMS_count
willow_incoming_call_count
willow_incoming_call_degree
willow_incoming_call_duration
willow_outgoing_call_count
willow_outgoing_call_degree
willow_outgoing_call_duration
willow_missed_call_count
willow_missed_callers
sycamore_total_surveys
sycamore_total_completed_surveys
sycamore_total_opened_surveys
sycamore_average_time_to_submit
sycamore_average_time_to_open
sycamore_average_duration