Class 5 Discussion Seeds

Noble 2009

• How are you currently organizing your bioinformatic work? Have you had issues with finding important files or is it working well?
• Noble's core principle is that "someone unfamiliar with your project should be able to look at your computer files and understand in detail what you did and why," and he notes that most often that someone is you, six months from now. Think of a specific analysis you ran more than six months ago. Could you reproduce it today? What specifically would block you — missing data, missing code, missing parameters, or missing intent?
• One of my favorite parts of this is that it preserves chronological order of analyses. What does/would your preferred organization look like?
• How would this work with documentation? Like a wiki or lab notebook? Do you store your notes in the same place as your code and data?
• What is the value of version control? Is it important only for "code" or also for analyses performed on the command line?
• AI will robustly annotate all of its code and incorporate numerous error-checks--often by default. Does this always make an analysis more reproducible and more "safe"? How can researchers validate AI-generated code that is more advanced than their own expertise, especially when the code runs and produces a reasonable-looking result?

Sandve 2013

• "Scientific papers commonly leave out experimental details essential for reproduction" - have you run into this?
• Have we developed the ‘‘culture of reproducibility’’ called for by this paper over 10 years ago? Why or why not?
• What do you think minimum standards should include?
• Which rules are going to be the most difficult to implement? Which are the most important? Are there any that are unnecessary or impossible in some cases?
• Sandve Rule 2: It is best practice to avoid manual changes to files. However, this is sometimes unavoidable. Is it sufficient to state when files were manually curated? What approaches can we take to reproducibly edit/curate files?
• Sandve Rule 6: How random are random seeds in AI-produced code? (Hint: The most common seed suggested is 42. Why?) Can you guess any other common seeds?
• Sandve Rule 7: Is it okay to tweak graphical elements in final plots instead of modifying the underlying source code? Why or why not?

Both

• Why do we care about reproducibility?
• Reproducibility is not equal to correctness...
  • reproducible vs replicable vs robust vs generalizable - what is the difference?
• These papers are showing their age. What technologies are being used today to solve these problems?
• If some or all of your analysis code is now AI-generated, what are the new provenance obligations? Is "prompt + model version + date" necessary and/or sufficient? Does AI-assisted coding make this easier or more difficult?
• Sandve Rule 5 says archive all intermediate results. What is the downside to this and when is it safe not to do this?
• Noble's lab notebook is external files that live in parallel to the driver script; Sandve Rule 9 suggests that results and interpretation must be tied back to the computation and output. How can this be done?
• AI is capable of higher level abstraction than humans, which also applies to code bases. Does code legibility impact reproducibility?