Creating a Data Archive

[jonathancallahan]

It is possible to create new 'pas' and 'pat' objects directly from the PurpleAir and ThingSpeak data sources. But this takes a long time.

For this article you will need to create a new archive for sensors in the Methow Valley. These sensors are seen with:

mvcca <- 
  pas_load() %>% 
  pas_filter(stringr::str_detect(label, "MV Clean Air Ambassador"))

pas_leaflet(mvcaa)

This document should go through the following steps:

flat directory structure

• use setArchiveBaseUrl() to get initial 'pas' object only
• get list of deviceDeploymentIDs for MVCAA sensors
• in a for loop, use pat_createNew() create 'pat' objects for each sensor. I think it will get the last week if you don't specify startdate and enddate
• save the created files all together in a single directory (i.e. ~/Data/my_project/)
• load files with base load()

use the archive structure

Do the same but create monthly 'pat' files and store both the 'pas' and the 'pat' files in the "well known" archive directory structure.

• create a directory with the "well known structure"
• use setArchiveBaseDir() instead of setArchiveBaseUrl()

[jeanmico]

Curious as to the archive directory structure - I pulled data from ~500 sensors and saved the data and meta for each as csv files (/project/deviceDeploymentID_data.csv, /project/deviceDeploymentID_meta.csv), and now can't figure out how to read them back in to a pa_timeseries object.

[jonathancallahan]

@jeanmico By creating csv files you have basically left the world view of the AirSensor package. It isn't currently straightforward to reenter that world view and create pat_timeseries objects.

As you can see from this GitHub issue, we are working on some documentation to demonstrate how to create your own archive of 'pas' and 'pat' objects as compressed binary files -- ".rda" files.

After @AstridSanna finishes this document, we will be curious to get your feedback.

[jonathancallahan]

@jeanmico We have just released a set of tutorials demonstrating how to create a local data archive:

https://working-with-data.mazamascience.com/2021/04/02/methow-valley-air-quality/