Skip to content

Protocols

Jump to bottom
Jefferson Smith edited this page May 6, 2021 · 21 revisions

Rough sketch of documentation describing the protocols, scripts, processes, and procedures

Table of Contents

Interact IDs

  1. Initial generation scheme
  2. TrekSoft generation scheme
  3. Harmonizer integration of IDs across partners

SenseDoc Telemetry

  1. SenseDoc devices are collected from participants by regional coordinator
  2. Telemetry data is extracted by coordinator from each device and stored on local computer or institutional data server
  3. From time to time during the collection period, a batch of these local data files are uploaded to ComputeCanada over SSH
  4. Incoming files are placed in /projects/def-dfuller/interact/incoming_data/{CITYNAME}/Wave{WAVENUM}/
  5. Each batch includes the telemetry data itself, plus an MD5 checksum for each file, computed on the local machine prior to upload
  6. After receiving data files on CC, those MD5 checksums are verified by the data manager, and if any problems are found, the necessary files are re-uploaded
  7. Once the files have been fully verified, they are normalized for ingest:
    1. all files are copied to /projects/def-dfuller/interact/permanent_archive/{CITYNAME}/Wave{WAVENUM}/SenseDoc
    2. This permanent_archive is then organized into a canonical file hierarchy, with data from each contributor stored in its own folder, named as {IID}_{DEVICEID}
    3. Within each device folder, verify that the crucial file for ingest present: SD{DEVID}fw{OSVER}_{TIMESTAMP}.sdb
  8. The normalized permanent_archive files are then added to our ProvLog system, which monitors every night to ensure that all files still match the checksums they were uploaded with and have not been deleted or altered on disk during the course of working with them
    1. This is done by running: provlog -m "Adding to ProvLog after validating upload checksum" -T DIRNAME
    2. If any changes are detected to logged files on disk, a message is sent to the data manager, who investigates and either restores the data files from backup, or updates the ProvLog record to explain the change, thus ensuring a complete manifest of data changes
  • checksums: produced using platform independent ExactFile tool, although any MD5 tool will work
  • ProvLog: on ComputeCanada, provenance monitoring is done using our own ProvLog tool, in conjunction with the native crontab system

SenseDoc Participant Metadata

Ethica Telemetry

Ethica Participant Metadata

Ethica Survey Data

TrekSoft Participant Metadata

TrekSoft Survey Data

($sketched)

Clone this wiki locally