Data

The main principles we follow in storing data are summarized in chapter 4 of Code and Data for the Social Sciences. The key points are:

• Store all data in tables with unique, non-missing keys
• Keep data normalized as far into the code pipeline as possible

We store small to medium sized data files that relate to a single project directly in our repositories using Git LFS.

We store large data files and data that need to be shared across multiple projects on Dropbox when possible, or occasionally in other large-scale storage locations.

Raw Data

Raw data files must be stored in a raw data directory that follows specific rules. This is normally in the /raw/ directory of a Github repository or in an analogous directory on Dropbox.

Every raw directory must have a detailed readme.md file which includes the source of the data, when and how it was obtained, and other any information necessary to understand the provenance and meaning of the data.

Codebooks, data use agreements, and other documentation should be placed in a /docs/ subdirectory.

Raw directories can contain code to perform preprocessing steps necessary to produce files ready to be used downstream (e.g., file conversions, appending files together, etc.). In this case the data in its original form should be stored in an /orig/ subdirectory and the preprocessed data should be stored in an /output/ or /data/ subdirectory.