Enabling the Verification of Computational Results

We conducted an experiment on the computational physics community to measure 1) how frequently authors shared their code and data directly in their article; 2) Whether authors who didn't would respond to a request for code and data; 3) How far a user skilled in computation, but not an expert in the field could get with the materials received.

procedures_and_descriptions/methodology.md is a description of our research methodology.

procedures_and_descriptions/column_descriptions.md is a description of the columns in our data format.

data/JCP_Dataset.csv an anonymized database of our findings which can be used to get the statistics we report in our article.

scripts/PRECSNumbers.py a script to produce the reported numbers from the included database

scripts/run.sh a script to run the numbers analysis of the database.

Computational Environment

The results published in our article were computed using python 3.6.4 in the Arch computing environment. The machine was a desktop with an Intel Core i7-6900K CPU, and two Nvidia GeForce GTX 1080 GPUs.