Analyze all Jupyter notebooks mentioned in PubMed Central

[Daniel-Mietchen]

Jupyter notebooks are a popular vehicle these days to share data science workflows. To get an idea of best practices in this regard, it would be good to analyze a good number of them in terms of their reproducibility and other aspects of usability (e.g. documentation, ease of reuse).

A search in PubMed Central (PMC) reveals the following results:

• ipynb OR jupyter

With currently just 102 hits, a systematic reanalysis seems entirely doable and could perhaps itself be documented by way of reproducible notebooks that might eventually end up being mentioned in PMC.

A good starting point here could be An open RNA-Seq data analysis pipeline tutorial with an example of reprocessing data from a recent Zika virus study, for which both a Jupyter notebook and a Docker image are available.

I plan to give a lightning talk on this. Some background is in this recent news piece.

[rossmounce]

A few notes...

With EuropePMC a search for ipynb OR jupyter gives 107 results:
http://europepmc.org/search?query=jupyter%20OR%20ipynb

I find it extremely interesting to note that EuropePMC has the full text for 102 of these 107 articles/preprints

(jupyter OR ipynb) AND (HAS_FT:Y)

Which demonstrates that jupyter/IPython notebooks are almost exclusively associated with open-friendly journals(?) Or perhaps this is a bias influenced by legally-enforced inability to do full text search on 'closed' journals where jupyter/ipynb might be mentioned but can't be found by EuropePMC because they are not allowed.

[rossmounce]

Rcode to get bibliographic metadata on each of those 107 hits from EuropePMC:

install.packages('europepmc')
library(europepmc)
hits <- epmc_search(query='jupyter%20OR%20ipynb&synonym=TRUE',limit=200)
dim(hits)
names(hits)
write.csv(hits,file="107hits.csv")

I've also made available the resulting CSV as an editable spreadsheet via GDocs:
https://docs.google.com/spreadsheets/d/1txg0u9zARHrLkY4MYuz5vmsVCZUOItgybEHqx13Bkbc/edit?usp=sharing

Perhaps with this sheet we can assign who takes responsibility for which papers?

[Daniel-Mietchen]

That's a great starting point — thanks!

[npscience]

+1 from me. Interested to contribute and to see the output.

[Daniel-Mietchen]

We've taken Ross' spreadsheet and added some columns for

• Notebook URL
• Code in problem cell
• Problem

The "Code in problem cell" column documents the notebook code causing the first problem, and the "Problem" column gives more details. So far, basically none of the notebooks ran through: We normally stopped after the first such error and went on to the next notebook, but for one rather complex notebook, we tried to go through to the end, which we have not reached yet.

[Daniel-Mietchen]

I've also added a column for the PMC URL to reduce the fiddling with URLs.

[Daniel-Mietchen]

I notified the Jupyter mailing list: https://groups.google.com/forum/#!topic/jupyter/6pQIarRmrsc .

[mrw34]

Here's a write-up of our efforts:

https://markwoodbridge.com/2017/03/05/jupyter-reproducible-science.html

Many thanks to @Daniel-Mietchen for the original idea, and for all the help over the weekend!

[Daniel-Mietchen]

@mrw34 Thanks - I'll go right into it.

[Daniel-Mietchen]

I found one that actually ran through, albeit after a warning about an old kernel:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4940747/bin/13742_2016_135_MOESM3_ESM.ipynb . A very simple notebook to test a random number generator, but hey, it works!

To celebrate the event, I introduced color coding to the spreadsheet: red for cases where the run resulted in an error, green when it did not.