Bulk Sensor Datasets
Data from Waggle-based sensor nodes (AoT nodes, Waggle uglyboxes, Nano-Waggle lightweight sensors) are available for download and analysis.
Data (ontology) is assembled and exported daily, and made available via an archive published every day at midnight CST.
Are there alternate avenues to get Array of Things data?
There are two applications available to access AoT data:
- The AoT File Browser to download bulk volumes of data
- The AoT API to programmatically access recent data
The following chart illustrates the data availability across all the AoT applications -
Waggle-based nodes used by several projects, so archives are organized by Waggle ProjectID. Each archive is stored as a single tar file.
Files in the archive use this naming convention:
Untarring an archive will create a date-specfic directory:
Here, YYYY-MM-DD is the UTC date when the tar archive was produced.
Inside the archive directory are the following files:
data.csv.gz # compressed file of all data values nodes.csv # list of nodes in the dataset and their metadata README.md # An explaination of the database fields sensors.csv # A list of sensors and their metadata provenance.csv # Metadata on the entire dataset archive offsets.csv # data.csv.gz file byte offsets
The sensor data is compressed, and must be uncompressed to be used. It will expand roughly 10X in size after being uncompressed.
All data is stored in CSV format.
Several datasets split based on projects are available at this link.
- Download the archive -
$ wget https://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_Chicago.complete.latest.tar
- Untar the archive -
$ tar -xf AoT_Chicago.complete.latest.tar
- It should produce a directory -
$ ls AoT_Chicago.complete.2018-05-22 AoT_Chicago.complete.latest.tar
- Explore the README and other meta information -
$ cd AoT_Chicago.complete.2018-05-22 $ ls data.csv.gz nodes.csv provenance.csv README.md sensors.csv
- When ready, unpack the data archive. The archive decompressess to a CSV text file about 10 times its size -
$ gzip -d --keep data.csv.gz $ ls data.csv data.csv.gz nodes.csv provenance.csv README.md sensors.csv
Working with the Data
We have a few simple tools for to manipulate, extract, create convenient data products and analyze (plot) data. These tools can be found in the Waggle data-tools repository.