-
Notifications
You must be signed in to change notification settings - Fork 43
Repository Structure
Overriding goal: All output must be replicable
A user with no special experience should be able to clone the repository, delete everything other than the code and raw input files, and reproduce all output including intermediate data files, statistical analysis, tables and figures, and PDFs of the paper draft and slides. Doing this should be a straightforward and intuitive process, and while the computational time may be substantial, the human time required should not. Not only must the current output be replicable, but all previous output must be as well (e.g., results in any previous draft or issue deliverable). The output not only needs to be replicable today; it needs to remain so in the future.
There are two key rules that are the bedrock of replicability.
Important commits are any commit merged to master, any commit that defines a final issue deliverable, and the final commit before a pull request. See discussion on the Workflow page here.
This is our version of the "one rule" of Github Flow -- that anything in the master branch is deployable.
Without this rule, a user has no way of knowing what combination of code and inputs produced a given set of output, and there is no guarantee that the code that produced it is the same as the code in the commit. Perhaps some scripts were run but not others. Perhaps edits were made to a script after it was run. Perhaps some output files were changed manually or by processes outside the repository.
Running the build scripts before committing eliminates this ambiguity. The first step in the script is to delete and recreate the relevant /output/. So long as no changes are made between the completion of the build script and the time of commit, the user can be confident that everything in /output/ was created by the build script and the code that it calls as of that commit. Re-running the build script at that commit must reproduce the output (provided rule #2 is followed as well!).
An external dependency is anything outside the repository that the code uses as inputs. This can include data files that are too large to be committed directly (even with Git LFS), code libraries / packages / modules for Python, R, or Stata, and output of other repositories.
When there are no external dependencies, following Rule #1 is sufficient to guarantee replicability. Otherwise, replicability may fail because a user does not know exactly what external resources were used to produce the output and/or the state of those resources has changed.
Dependencies are robustly documented if (i) a user can easily see what external resources are used as inputs; (ii) we record these resources' location and state at the time the build script is run; (iii) we provide enough provenance information that a user has a good chance of being able to locate or replicate these resources in the future.
This method is not foolproof. External resources are outside of the repository and so may be outside of our control, and in some cases we may not be able to guarantee that they do not disappear or change. The practices below are designed to minimize this risk.
A repository typically contains all work related to a single research project (e.g., a single journal article). This includes the data, analysis, paper, slides, and any supplementary information like data use agreements, notes from seminar presentations, and so on.
In some cases, a repository builds a dataset or other resources that are then used by multiple projects.
For small to medium scale projects we commit data directly to the repository (using GIT LFS). Large data and other large inputs / outputs are stored outside the repository, typically on Dropbox.
The key building block of our repositories is a module. A module is a directory with a build script at its top level, an /output/ subdirectory, and one or more input subdirectories such as /code/ or /input/.
A module is a self contained unit whose output can always be reproduced by running the build script. A module must declare all of its inputs (files from other parts of the repository that it requires to run) as well as its external dependencies (files from outside the repository that it requires to run). Provided the the inputs and external dependencies are available, the module should run successfully regardless of what machine it is on or where it is located on disk.
A build script is a script at the top level of a module that does the following things in sequence (i) deletes the contents of the /output/ directory; (ii) checks / records the state of all inputs and external dependencies; (iii) executes the code and other steps needed to produce the output.
Step (i) is the most important. Deleting the contents of /output/ guarantees that any output following Rule #1 is reproducible.
In our repository template build scripts are written in Python and are always named make.py. We use the simple code library gslab_make to automate steps (i) through (iii).
For some large projects we use a more sophisticated suite of build tools called SCons. In this case, we declare the relationship between inputs and outputs more explicitly, and this allows the builder to reproduce only the portions of /output/ that need to be updated given what has been changed. We discuss the use of SCons in the appendix here.
- raw
- data
- analysis
- paper_slides
- lib