SenseDoc

The SenseDoc is a research grade multisensor device, made by MobySens, and used for mobility (GPS) and physical activity (accelerometer) tracking. These data are collected continuously and allow us to measure location-based physical activity and infer transportation mode.

The data fields collected by the SenseDoc can be found on INTERACT's Data Dictionary

Collection/Extraction

Data is pulled from the individual devices by the regional coordinator, using an extraction tool provided by MobySens. While this extraction includes the proprietary raw data files, they are also coalesced into a SQLite3 DB file (and given a .sdb extension) for more efficient manipulation. Our migration process preserves those raw data files, but the ingest process is built against the SDB files, since they are in a much more consumable form.

Migration

When a participant has completed their collection period, the SenseDoc is returned to the regional coordinator, who then extracts the data to their local computer. Since the device is always recording, this means that it may include some movement data contributed by the coordinator, either before or after it was given to the participant. So to filter the data properly, the coordinator maintains a start and end timestamp for when the device was actually being worn by the participant, known as the "wear date window." This data is kept in the participant metadata table.

From time to time during the collection period, batches of these extracted data files are uploaded to ComputeCanada over SSH. Those incoming files are staged in a folder at /projects/def-dfuller/interact/incoming_data/{CITYNAME}/Wave{WAVENUM}/.

Uploaded batches include the telemetry data itself, plus an MD5 checksum for each file. The checksums were computed on the coordinator's local machine prior to upload, so it can be used by the data manager to verify the success of the upload once it is received on ComputeCanada. If any verification problems are found, the damaged files are re-uploaded and reverified, and only marked as accepted once the problem has been corrected and the checksums match.

After accepting an uploaded file, it is then given a "provenance sidecar," based on the checksum provided by the coordinator. This sidecar then moves around the system along with the data file itself and can be used at any time to verify that the file has not been altered since it was first produced. Additionally, if a change is made to the file, the sidecar can be updated with a new checksum, as well as an explanation of what changed. This is handled by our ProvLog system, and provides provenance tracking for the lifecycle of all the data files that are part of our pipeline.

Manual Ingest Prep

Before data is actually ingested, a number of verification and normalization steps are conducted first to ensure that the ingest can proceed successfully.

• files migrated to the incoming_data folder are copied to the permanent_archive area ** /projects/def-dfuller/interact/permanent_archive/{CITYNAME}/Wave{WAVENUM}/SenseDoc

• This permanent_archive folder is then organized into a canonical file hierarchy, with data from each contributor stored in its own folder, named as {IID}_{DEVICEID}

• Within each user-device folder, we verify that the crucial ingest file is present: SD{DEVID}fw{OSVER}_{TIMESTAMP}.sdb

• The normalized permanent_archive files are then added to our ProvLog system, which scans every night to ensure that all files always match the checksums they were uploaded with and have not been deleted or altered on disk during the course of working with them

  • This is done by running: provlog -T DIRNAME
  • If any changes are ever detected in any logged files on disk, a message is sent to the data manager, who investigates and either restores the data files from backup, or updates the ProvLog record to explain the change, thus ensuring a complete manifest of data changes

• The participant metadata is placed into the permanent_archive folder, in a file called /projects/def-dfuller/interact/permanent_archive/{CITYNAME}/Wave{WAVENUM}/linkage.csv ** It's called this because it provides the linkage between the participants and the device ids they were issued

Guided Ingest Prep

With the data files all in the correct place, with the expected names, the data manager can then begin to process the Jupyter Notebook that governs the ingest process.

• Edit the parameter assignments in the first code block of the notebook to set the wave_id and city_id being ingested

The next few blocks of the notebook will conduct some additional analyses to find gaps and conflicts in the data so they can be fixed prior to ingest.

• All expected files are confirmed to be present and named correctly
• The incoming linkage data is confirmed to be well-formed
• Each expected participant has corresponding telemetry data in the permanent_archive folder
• All telemetry data found in the permanent_archive folder corresponds to a known participant

Any problems found by these tests are reported to the data manager who then consults with the regional coordinator to fix the discrepancies. The most common problem found here is to find a user in the linkage table who produced no data in the telemetry folders. These are usually cases where a coordinator created a dummy account to use for testing. But since they did not actually wear a device, there is no telemetry to go with the account. In these cases, the user record must be marked by putting the word 'ignore' in the data_disposition field of the linkage table, which instructs the ingest system to ignore that user record entirely.

Once the validation block of the Jupyter Notebook passes cleanly, reporting no unexpected conditions in the data, the actual ingest blocks can be run.

Ingest

First, the linkage data will be loaded into the DB table (portal_dev.sensedoc_assignments). This is a straightforward process of...

Then the notebook will proceed to loading the telemetry files.

Loading telemetry files is a bit more complicated...

• The last few sections perform the actual ingest
  • In the first pass, the raw telemetry files are loaded into a temporary DB table
  • In the next block, that temporary table is cross-linked with the proper IID, based on the mapping from the device id found in the linkage table
  • Finally, the cross-linked telemetry data is added to the final telemetry tables (sd_gps, sd_accel, '''and others?''')
• Once the ingest has completed successfully, a few housekeeping tasks are required:
  • Delete the temporary tables '''(called?)'''
  • Export the JupyterNotebook as a PDF, which provides a complete document of the ingest process, as it happened.
  • If any substantive code was changed in the notebook (aside from setting parameters), clear all the output blocks, save the notebook, and commit the changes to the git repo, describing what improvements or corrections were made to the code
• Congratulations, you have now completed an ingest cycle.