Data

The main principles we follow in storing data are summarized in chapter 4 of Code and Data for the Social Sciences. The key points are:

• Store all data in tables with unique, non-missing keys
• Keep data normalized as far into the code pipeline as possible

We store small to medium sized data files that relate to a single project directly in our repositories using Git LFS.

We store large data files and data that need to be shared across multiple projects on Dropbox.

Raw data files must be stored in a special raw data directory. This is normally in the /raw/ directory of a Github repository or in our main /raw/ directory on Dropbox. Raw data directories follow special rules which are detailed in our repository template here. The most important rule is that every raw data directory must have a detailed readme.txt file which includes the source of the data, when and how it was obtained, and other any information necessary to understand the provenance and meaning of the data.