Analyze all Jupyter notebooks mentioned in PubMed Central

[Daniel-Mietchen]

Jupyter notebooks are a popular vehicle these days to share data science workflows. To get an idea of best practices in this regard, it would be good to analyze a good number of them in terms of their reproducibility and other aspects of usability (e.g. documentation, ease of reuse).

A search in PubMed Central (PMC) reveals the following results:

• ipynb OR jupyter

With currently just 102 hits, a systematic reanalysis seems entirely doable and could perhaps itself be documented by way of reproducible notebooks that might eventually end up being mentioned in PMC.

A good starting point here could be An open RNA-Seq data analysis pipeline tutorial with an example of reprocessing data from a recent Zika virus study, for which both a Jupyter notebook and a Docker image are available.

I plan to give a lightning talk on this. Some background is in this recent news piece.

[rossmounce]

A few notes...

With EuropePMC a search for ipynb OR jupyter gives 107 results:
http://europepmc.org/search?query=jupyter%20OR%20ipynb

I find it extremely interesting to note that EuropePMC has the full text for 102 of these 107 articles/preprints

(jupyter OR ipynb) AND (HAS_FT:Y)

Which demonstrates that jupyter/IPython notebooks are almost exclusively associated with open-friendly journals(?) Or perhaps this is a bias influenced by legally-enforced inability to do full text search on 'closed' journals where jupyter/ipynb might be mentioned but can't be found by EuropePMC because they are not allowed.

[rossmounce]

Rcode to get bibliographic metadata on each of those 107 hits from EuropePMC:

install.packages('europepmc')
library(europepmc)
hits <- epmc_search(query='jupyter%20OR%20ipynb&synonym=TRUE',limit=200)
dim(hits)
names(hits)
write.csv(hits,file="107hits.csv")

I've also made available the resulting CSV as an editable spreadsheet via GDocs:
https://docs.google.com/spreadsheets/d/1txg0u9zARHrLkY4MYuz5vmsVCZUOItgybEHqx13Bkbc/edit?usp=sharing

Perhaps with this sheet we can assign who takes responsibility for which papers?

[Daniel-Mietchen]

That's a great starting point — thanks!

[npscience]

+1 from me. Interested to contribute and to see the output.

[Daniel-Mietchen]

We've taken Ross' spreadsheet and added some columns for

• Notebook URL
• Code in problem cell
• Problem

The "Code in problem cell" column documents the notebook code causing the first problem, and the "Problem" column gives more details. So far, basically none of the notebooks ran through: We normally stopped after the first such error and went on to the next notebook, but for one rather complex notebook, we tried to go through to the end, which we have not reached yet.

[Daniel-Mietchen]

I've also added a column for the PMC URL to reduce the fiddling with URLs.

[Daniel-Mietchen]

I notified the Jupyter mailing list: https://groups.google.com/forum/#!topic/jupyter/6pQIarRmrsc .

[mrw34]

Here's a write-up of our efforts:

https://markwoodbridge.com/2017/03/05/jupyter-reproducible-science.html

Many thanks to @Daniel-Mietchen for the original idea, and for all the help over the weekend!

[Daniel-Mietchen]

@mrw34 Thanks - I'll go right into it.

[Daniel-Mietchen]

I found one that actually ran through, albeit after a warning about an old kernel:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4940747/bin/13742_2016_135_MOESM3_ESM.ipynb . A very simple notebook to test a random number generator, but hey, it works!

To celebrate the event, I introduced color coding to the spreadsheet: red for cases where the run resulted in an error, green when it did not.

[Daniel-Mietchen]

Here's a notebook shared only as a screenshot, from a paper about reproducibility:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5014984/figure/figure1/ .

Just added yellow to the spreadsheet for cases like this where the notebook did not produce errors nor ran through, and where "n/a" is not applicable in the sense that there is no notebook.

[Daniel-Mietchen]

There is a nice "Ten simple rules" series in PLOS Computational Biology:
http://collections.plos.org/ten-simple-rules . Perhaps we should do one on "how to share Jupyter notebooks"?

They already have Ten Simple Rules for Reproducible Computational Research and Ten Simple Rules for Cultivating Open Science and Collaborative R&D as well as other somewhat related articles, but none of them seem to touch upon Jupyter notebooks.

[Daniel-Mietchen]

Some comments relevant for here are also in #41 (comment) .

[Daniel-Mietchen]

The above close was just as part of the wrap-up of the doathon. I will keep working on it and document my progress over at Daniel-Mietchen/ideas#2 .

[rossmounce]

@mrw34 @Daniel-Mietchen excellent write-up!

If licensing allows could you upload somewhere all the .ipynb notebooks you found that were related to those 107 papers?

[Daniel-Mietchen]

@rossmounce I plan to do that but haven't yet checked for licensing (added column AH for that).

The notebook URLs are in column AD, which currently has the following list:

• http://nbviewer.jupyter.org/github/biocore/American-Gut/blob/master/ipynb/module2_v1.0.ipynb
• http://nbviewer.jupyter.org/github/Gilles86/in_limbo/blob/master/notebooks/level2_simple.ipynb
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4150318/bin/rspb20141025supp2.txt
• http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3906378/bin/pone.0085777.s003.ipynb
• https://github.com/simkovic/wolfpackRevisited/blob/master/code/Revision1.ipynb
• http://nbviewer.jupyter.org/github/idekerlab/py2cytoscape/blob/develop/examples/New_wrapper_api_sample.ipynb
• http://nbviewer.jupyter.org/github/HullUni-bioinformatics/ReproPhylo/blob/master/Tutorial_files/Git/Git_Demo.ipynb
• http://nbviewer.jupyter.org/github/biocore/American-Gut/blob/master/ipynb/Alpha%20diversity%20notebook.ipynb
• http://nbviewer.jupyter.org/github/biocore/American-Gut/blob/master/ipynb/Power.ipynb
• http://nbviewer.jupyter.org/github/jmumford/orthogonalizaton_ipynb/blob/master/orthogonalization.ipynb
• http://nbviewer.jupyter.org/gist/fabianrost84/3cc58a27b5688f4e2eba
• http://nbviewer.ipython.org/github/germs-lab/frontiers-review-2015/blob/master/frontiers-nb-2015.ipynb
• http://nbviewer.jupyter.org/github/JonM4024/synthetic_lethals/blob/master/double_deletions_example.ipynb
• http://nbviewer.jupyter.org/github/a-slide/iPython-Notebook/blob/master/Notebooks/VCF_analysis.ipynb
• http://nbviewer.jupyter.org/github/shyamrallapalli/fraxinus_version1_data_analysis/blob/master/fraxinus_visits_model/Players-nonplayers.ipynb
• https://duke.app.box.com/s/9kphs2300kumjyowjq06
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4669674/bin/12918_2015_236_MOESM5_ESM.ipynb
• http://nbviewer.jupyter.org/github/maayanlab/Zika-RNAseq-Pipeline/blob/master/Zika.ipynb
• https://github.com/ajouary/VR_Zebrafish/blob/master/Code/FishTracking.ipynb
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4940747/bin/13742_2016_135_MOESM3_ESM.ipynb
• https://github.com/sulab/myvariant.info/blob/master/docs/ipynb/myvariant_R_miller.ipynb
• https://collab.humanbrainproject.eu/#/collab/704/
• http://goo.gl/UIhBjf
• https://github.com/scanisius/discover-notebooks
• https://github.com/linsalrob/PyFBA/blob/working/iPythonNotebooks/From_functional_roles_to_gap-filling.ipynb; https://github.com/linsalrob/PyFBA/blob/working/iPythonNotebooks/Using_an_SBML_model.ipynb
• https://github.com/mattloose/RUFigs
• https://doi.org/10.5281/zenodo.160333
• http://nbviewer.ipython.org/github/biocore/qiime/blob/1.9.1/examples/ipynb/illumina_overview_tutorial.ipynb
• https://github.com/PeterUlz/FocalAmplifications/tree/master/Focal_amplifications_in_R.ipynb
• https://github.com/LozRiviera/COMPASS_paper
• http://nbviewer.ipython.org/github/biocore/American-Gut/blob/master/ipynb/module2_v1.0.ipynb
• https://github.com/KarchinLab/Tokheim_PNAS_2016
• http://github.com/nickloman/ebov
• http://tinyurl.com/zq6uuja
• http://github.com/linnarsson-lab/ipynb-lamanno2016
• https://github.com/gregcaporaso/office-microbes
• https://github.com/MIT-LCP/mimic-iii-paper/blob/master/notebooks/tsne-vis.ipynb
• https://github.com/isidroc/bioalerts
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4980007/bin/pone.0159762.s001.html
• http://github.com/chewgl/uORF_repressiveness_supplemental/
• scipy/scipy#5232
• https://github.com/ipython/docker-notebook
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5189966/bin/ppat.1006114.s013.zip
• https://dx.doi.org/10.7554/eLife.12741.029; https://dx.doi.org/10.7554/eLife.12741.030
• https://github.com/biocore/american-gut/blob/master/ipynb/primary-processing/02-filter_sequences_for_blooms.md
• https://github.com/berenslab/pr_bc_connectivity
• https://github.com/Schork-Lab/np_single_nucleus_rnaseq/
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4979073/bin/NIHMS798127-supplement-Supplementary_Materials__Appendix_1.html
• Downloadable at http://www.phonetik.phoniatrie.rwth-aachen.de/bkroeger/documents/ipynb_InnerSpeechLoop.zip
• https://github.com/sofroniewn/2pRAM-paper
• Notebook attached as zip file
• Notebook is attached as additional files (in PDF)
• https://github.com/biocore/gneiss/blob/master/ipynb/88soils.ipynb and https://github.com/biocore/gneiss/blob/master/ipynb/cfstudy.ipynb
• https://github.com/chembl/mychembl/blob/master/ipython_notebooks/09_myChEMBL_web_services.ipynb; https://github.com/chembl/mychembl/tree/master/ipython_notebooks
• http://nbviewer.jupyter.org/github/peterwittek/ipython-notebooks/blob/master/Citation_Network_SDP.ipynb.
• http://doi.org/10.5281/zenodo.208230
• http://github.com/ThorntonLab/disease_sims; http://github.com/molpopgen/buRden; http://github.com/molpopgen/fwdpy; http://github.com/molpopgen/TennessenEAonly
• https://github.com/zhuangjun1981/retinotopic_mapping

[Daniel-Mietchen]

Mark's write-up is now up at https://markwoodbridge.com/2017/03/05/jupyter-reproducible-science.html .

[Daniel-Mietchen]

There is a validator tool for Jupyter notebooks: https://github.com/jupyter/nbformat/blob/master/nbformat/validator.py

[Daniel-Mietchen]

Here is a discussion about using Jupyter notebooks programmatically, with a useful demo.

[Daniel-Mietchen]

I am thinking of submitting this to JupyterCon — submission deadline March 14. Anyone in?

[rossmounce]

@Daniel-Mietchen I'd be happy to help you prepare the abstract submission, do a bit more analysis but I can't go to the meeting :) Does that count as 'in' ?

[Daniel-Mietchen]

That's "in enough" for my taste. Don't know whether I can go either, but the point is to reach out to the Jupyter community and to help do something about these issues, e.g. by refining the recommendations and perhaps offering some validation mechanism (think Schematron for XML).

[gbilder]

Hi. Daniel pointed me at this thread.

When you say that "DOIs, don't permit direct downloading today" I'm not sure if the word "permit" here is in reference to:

• access control restrictions
• the convention that Crossref/DataCite DOIs tend to resolve to a landing page instead of 'the thing itself."

Or both.

As far as I can tell, parties appear unanimous on the fact that, barring privacy/confidentiality issues, data should be made available openly. Access control issues should be minimal.

And DOIs would allow direct downloading today if, when possible and practical, those registering DOI metadata included links to direct downloads in their Crossref or DataCite metadata. At Crossref we are increasingly seeing publishers registering full text links in their metadata. For examples, see the text-mining links in these metadata records from PeerJ:

http://api.crossref.org/members/4443/works

[khinsen]

@gbilder My reference was to the landing-page issue. It's good to hear that people are discussion solutions, but as far as I know it remains impractical today to download a dataset automatically given a DOI.

[mrw34]

@khinsen @gbilder I guess best-practice here involves assigning DOIs to datasets (and not just their parent publication, if any), and resolving any ambiguity over if/how the data itself can be automatically retrieved given the relevant DOI. Lots more on the latter here: https://jcheminf.springeropen.com/articles/10.1186/s13321-015-0081-7

[Daniel-Mietchen]

I was just pinged about another validation tool:

• https://github.com/computationalmodelling/nbval

[Daniel-Mietchen]

Here's an interesting blog post on what can go wrong in terms of reproducibility (with a focus on R): http://blog.appsilondatascience.com/rstats/2017/03/28/reproducible-research-when-your-results-cant-be-reproduced.html .

[DCGenomics]

A demo for the original issue: https://www.ncbi.nlm.nih.gov/pubmed/27583132
If you click on the LinkOut link, would adding the Jupyter and Docker links here be helpful?

[mrw34]

More on reproducibility, including a few Jupyter mentions: https://www.practicereproducibleresearch.org

[Daniel-Mietchen]

@DCGenomics I think making the Jupyter, Docker, mybinder etc. versions of the code more discoverable is useful in principle, but conventional LinkOut (which is not shown by default) may not be the best mechanism to do this.

What I could imagine is a mechanism similar to the way images are currently being presented in PubMed, i.e. something that is shown by default if the paper comes with code shared in a standard fashion. That standard would have to be defined, though.

While necessary for reproducibility, discoverability alone is not sufficient, and this example paper highlights that, as explained in Mark's initial write-up.

[Daniel-Mietchen]

I asked for input from the JATS4R community on how such things could/ ought to be tagged in JATS.

[Daniel-Mietchen]

There is a JupyterCon talk about citing Jupyter notebooks. I have contacted the speakers.

[mpacer]

I'm not going to read all of this cause it's long. But, this is a cool idea and neat dataset. @eseiver mentioned you wanted to know how to read in notebooks, for which you can use nbformat specifically the nbformat.read() which you should probably use inside a context manager.

[Daniel-Mietchen]

At the WikiCite 2017 hackathon today, we made some further progress in terms of making this analysis itself more reproducible — a Jupyter notebook that runs the Jupyter notebooks listed in our Google spreadsheet and spits out the first error message: http://paws-public.wmflabs.org/paws-public/995/WikiCite%20notebook%20validator.ipynb . @mpacer - yes, it makes use of nbformat.read()

We also looked at Jupyter notebooks cited from Wikipedia — notes at https://meta.wikimedia.org/wiki/WikiCite_2017/Jupyter_notebooks_on_Wikimedia_sites .

[yuvipanda]

Hello! I'm part of the team that's working on beta.mybinder.org and related stuff, and am extremely interested in the idea of a 'badge that represents that your code is reproducible, and has been reproduced by CI'. Funnily, I also built the PAWS stuff, which is awesome to find in completely unrelated contexts :D

The part of the stack I've been attacking right now is the 'how do we reproduce the environment that the analysis took place in', as part of the mybinder work. You can see the project used for that here: https://github.com/jupyter/repo2docker. It takes a git repository and converts it into a Docker image, using conventions that should be easy to use for most people (and does not require them to understand or use Docker unless they want to). It's what powers the building bits of mybinder :)

As part of the CI for that project, you can see that we also build and validate some external repositories that are popular! We just represent these as YAML files here: https://github.com/jupyter/repo2docker/tree/master/tests/external and have them auto test on push so we make sure we can keep building them. This can be inverted too - in the repo's CI they can use repo2docker to make sure their changes don't break the build.

The part where we haven't made much progress yet is in actual validation. nbval mentioned here seems to be the one I like most - it integrates into pytest! We can possibly integrate repo2docker into pytest too, and use that to easily validate repos? Lots of possible avenues to work towards :)

One of the things I'd love to have is something like what https://www.ssllabs.com/ssltest/analyze.html?d=beta.mybinder.org does for HTTPS on websites - scores you on a bunch of factors, with clear ways of improving it. Doing something like that for git repos with notebooks would be great, and I believe we can do a fair amount of work towards it now.

I'll also be at JupyterCon giving a few talks, and would love to meet up if any of you are going to be there!

/ccing @choldgraf who also does a lot of these things with me :)

[danielskatz]

Hi!

@Daniel-Mietchen pointed me at this thread/project yesterday, and it seems quite interesting.

I wonder if it makes sense to think about short term and long term reproducibility for notebooks?

By short term, I mean that the notebook might depend on a python package that has to be installed, which could be done by pip before running the notebook, and this step could be automated by a notebook launcher perhaps.

And long term meaning that at some point, the dependency will not work, pip will be replaced by something new, etc., and the only way to solve this is to capture the full environment. This seems similar to what @yuvipanda describes, and what @TanuMalik is trying to do a bit differently in https://arxiv.org/abs/1707.05731 (though I don't think her code is available). And long term here might still have OS requirements, so maybe I really mean medium term.

Also, I thought I would cc some other people who I think will be interested in this topic, and could perhaps point to other work done in the context of making notebooks reproducible: @fperez @labarba @jennybc @katyhuff

[ThomasA]

@khinsen - sorry I lost track of this thread back in March... Yes, I think @Chroxvi's reproducibility can be carved out of the Magni package and I actually hope to do that sometime this fall. I hope to do that along with @ppeder08's validation (https://github.com/SIP-AAU/Magni/tree/master/magni/utils/validation) which can be used for in-/output-validation of for example somewhat more abstract "data types" than Python's built-in types.

[TanuMalik]

Hi Dan, Our code is available from https://bitbucket.org/geotrust/sciunit-cli Current documentation is available from http://geotrusthub.org/geotrust_html/GeoTrust.html Yes ours is restricted to Linux OS for now. We have a bare bones version for Mac OS X that is not in production use. We are currently working on enabling reproducibility for workflows and Jupyter notebooks through application virtualization. We need some more work to capture standard I/O. Tanu

…

On Tue, Aug 8, 2017 at 2:17 AM, Thomas Arildsen ***@***.***> wrote: @khinsen <https://github.com/khinsen> - sorry I lost track of this thread back in March... Yes, I think @Chroxvi <https://github.com/chroxvi>'s reproducibility can be carved out of the Magni package and I actually hope to do that sometime this fall. I hope to do that along with @ppeder08 <https://github.com/ppeder08>'s validation (https://github.com/SIP-AAU/ Magni/tree/master/magni/utils/validation) which can be used for in-/output-validation of for example somewhat more abstract "data types" than Python's built-in types. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANq42YxHXdtU_GCCGLGot-JRIZV9vo0hks5sWAuTgaJpZM4MMOSq> .

[Daniel-Mietchen]

Thanks for the additional comments. I have proposed to work on this further during the Wikimania hackathon: https://phabricator.wikimedia.org/T172848 .

[Daniel-Mietchen]

I got sick during the hackathon and haven't fully recoverd, but JupyterCon is just days away, so I have started to distill the discussion here into an outline for the talk next Friday:
https://github.com/Daniel-Mietchen/events/blob/master/JupyterCon-2017.md#outline-of-the-talk . Will work on it from Tuesday onwards, and your contributions to this are as always welcome.

[RomanGurinovich]

After chatting with @Daniel-Mietchen about this idea, we've implemented web app to autorun notebooks mentioned in the paper.

exe.sci.ai

Just add list of papers' URLs, like
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5322252/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4965465/
and run the executability validator.

It is a pre-pre-pre-alpha version done for fun and in the name of reproducibility. Please, report all the issues and suggest improvements.

Current setup might require additional horsepower to consume bigger datasets. Also we plan to implement whole repo autodeployment, too many fails because of lack of this feature at the moment.

List of current issues:

https://github.com/sciAI/exe/blob/master/Executability%20issues.md

Validator code:

https://github.com/sciAI/exe

All the credits are going to sci.AI team and, especially @AlexanderPashuk. Alex, thank you for the effort and fights with libs compatibility.

[RomanGurinovich]

@yuvipanda, nice job. High five! Dan mentioned, you are on the conference now, right? If you are interested, we can combine effort

[choldgraf]

We're both at the conference now and will be at the hack sessions tomorrow!

[Daniel-Mietchen]

I was traveling during the hackathon - had heard about it too late. In any case, I hope we can combine efforts with the Binder team. For those interested, the talk sits at
https://github.com/Daniel-Mietchen/events/blob/master/JupyterCon-2017.md#outline-of-the-talk , and the video shall hopefully become available in a few weeks.

[Daniel-Mietchen]

Seen at JupyterCon:
https://github.com/jupyter/repo2docker , a tool that can dockerize a git repo and provide a Jupyter notebook to explore the container.

[cgpu]

Hello! I'm part of the team that's working on beta.mybinder.org and related stuff, and am extremely interested in the idea of a 'badge that represents that your code is reproducible, and has been reproduced by CI'. Funnily, I also built the PAWS stuff, which is awesome to find in completely unrelated contexts :D

Apologies for reviving a closed issue. I am also interested in the reproducibility badge (which is not necessarily the same as the binder badge). I came across @mwoodbri 's jupyter-ci. Any plans for this to be used or people that currently use this?

(Also cc'ing @yuvipanda as he has been involved in repo2docker.)

[mwoodbri]

@cgpu Hi! At the time jupyter-ci was an answer to "what's the simplest possible way to generate a badge and notify of validation failure?". Simply porting the project to GitHub and GitHub Actions (updating to latest Jupyter best-practice in the process, if necessary) would be a great start. But a more general solution involving repo2docker and/or Binder would be even better!

[cgpu]

@cgpu Hi! At the time jupyter-ci was an answer to "what's the simplest possible way to generate a badge and notify of validation failure?". Simply porting the project to GitHub and GitHub Actions (updating to latest Jupyter best-practice in the process, if necessary) would be a great start. But a more general solution involving repo2docker and/or Binder would be even better!

Hi @mwoodbri , thank you for the prompt response and the background information! I am really fond of the idea of having a binary does it reproduce badge of honour, jupyter-ci sounds exactly like that. I am trying to reproduce .ipynb accompanied publications as part of a workshop exercise and I am really struggling. It would be nice to know before hands which ones to invest time on, hence the interest. Additionally, it would be nice to have/create a hub or awesome repo with the ones that are reproducible only (verified by ci).

jupyter-ci is currently only on gitlab, I know it's the same in essence, but the community is much more active in Github, it would be nice if you have plans to maintain the repo + idea long term to bring it over here. Just a thought.

Thanks once again!

[mwoodbri]

@cgpu Here's a version converted to use GitHub Actions: https://github.com/mwoodbri/jupyter-ci

[cgpu]

@cgpu Here's a version converted to use GitHub Actions: https://github.com/mwoodbri/jupyter-ci

@mwoodbri thank you! Time for me to test now :)

@Daniel-Mietchen thank you for providing the reproducibility cafe space for further discussions 2 years after the start of the initiative. Feel free to close this.