-
Notifications
You must be signed in to change notification settings - Fork 43
Repository Structure
Overriding goal: All output must be replicable
A user with no special experience should be able to clone the repository, delete everything other than the code and raw input files, and reproduce all output including intermediate data files, statistical analysis, tables and figures, and PDFs of the paper draft and slides. Doing this should be a straightforward and intuitive process, and while the computational time may be substantial, the human time required should not. Not only must the current output be replicable, but all previous output must be as well (e.g., results in any previous draft or issue deliverable). The output not only needs to be replicable today; it needs to remain so in the future.
There are two key rules that are the bedrock of replicability.
Rule #1: Important commits must follow a complete run of the relevant build script
Important commits are any commit merged to master, any commit that defines a final issue deliverable, and the final commit before a pull request. See discussion on the Workflow page here.
This is our version of the "one rule" of Github Flow -- that anything in the master branch is deployable.
Without this rule, a user has no way of knowing what combination of code and inputs produced a given set of output. Perhaps some scripts were run but not others. Perhaps edits were made to a script after it was run. Perhaps some output files were changed manually or by processes outside the repository.
Running the build scripts before committing eliminates this ambiguity. The first step in the script is to delete and recreate the relevant /output/. So long as no changes are made between the completion of the build script and the time of commit, the user can be confident that everything in /output/ was created by the build script and the code that it calls as of that commit. Re-running the build script at that commit must reproduce the output (provided rule #2 is followed as well!).
A repository typically contains all work related to a single research project (e.g., a single journal article). This includes the data, analysis, paper draft, slides, and any supplementary information like data use agreements, notes from seminar presentations, and so on.
- Make.py
- data
- analysis
- paper_slides
- lib