Repository navigation
Analyze all Jupyter notebooks mentioned in PubMed Central #25
Description
Activity
A few notes...
With EuropePMC a search for ipynb OR jupyter gives 107 results:
http://europepmc.org/search?query=jupyter%20OR%20ipynbI find it extremely interesting to note that EuropePMC has the full text for 102 of these 107 articles/preprints
(jupyter OR ipynb) AND (HAS_FT:Y)
Which demonstrates that jupyter/IPython notebooks are almost exclusively associated with open-friendly journals(?) Or perhaps this is a bias influenced by legally-enforced inability to do full text search on 'closed' journals where jupyter/ipynb might be mentioned but can't be found by EuropePMC because they are not allowed.
Reacted by Goodwin Gibbins and Paola MasuzzoRcode to get bibliographic metadata on each of those 107 hits from EuropePMC:
install.packages('europepmc') library(europepmc) hits <- epmc_search(query='jupyter%20OR%20ipynb&synonym=TRUE',limit=200) dim(hits) names(hits) write.csv(hits,file="107hits.csv")I've also made available the resulting CSV as an editable spreadsheet via GDocs:
https://docs.google.com/spreadsheets/d/1txg0u9zARHrLkY4MYuz5vmsVCZUOItgybEHqx13Bkbc/edit?usp=sharingPerhaps with this sheet we can assign who takes responsibility for which papers?
That's a great starting point — thanks!
+1 from me. Interested to contribute and to see the output.
We've taken Ross' spreadsheet and added some columns for
- Notebook URL
- Code in problem cell
- Problem
The "Code in problem cell" column documents the notebook code causing the first problem, and the "Problem" column gives more details. So far, basically none of the notebooks ran through: We normally stopped after the first such error and went on to the next notebook, but for one rather complex notebook, we tried to go through to the end, which we have not reached yet.
I've also added a column for the PMC URL to reduce the fiddling with URLs.
I notified the Jupyter mailing list: https://groups.google.com/forum/#!topic/jupyter/6pQIarRmrsc .
Here's a write-up of our efforts:
https://markwoodbridge.com/2017/03/05/jupyter-reproducible-science.html
Many thanks to @Daniel-Mietchen for the original idea, and for all the help over the weekend!
Reacted by David Haberthür@mrw34 Thanks - I'll go right into it.
I found one that actually ran through, albeit after a warning about an old kernel:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4940747/bin/13742_2016_135_MOESM3_ESM.ipynb . A very simple notebook to test a random number generator, but hey, it works!To celebrate the event, I introduced color coding to the spreadsheet: red for cases where the run resulted in an error, green when it did not.
Reacted by Joe McArthur, Mark Woodbridge and Scott EdmundsHere's a notebook shared only as a screenshot, from a paper about reproducibility:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5014984/figure/figure1/ .Just added yellow to the spreadsheet for cases like this where the notebook did not produce errors nor ran through, and where "n/a" is not applicable in the sense that there is no notebook.
There is a nice "Ten simple rules" series in PLOS Computational Biology:
http://collections.plos.org/ten-simple-rules . Perhaps we should do one on "how to share Jupyter notebooks"?They already have Ten Simple Rules for Reproducible Computational Research and Ten Simple Rules for Cultivating Open Science and Collaborative R&D as well as other somewhat related articles, but none of them seem to touch upon Jupyter notebooks.
Some comments relevant for here are also in #41 (comment) .
77 remaining items
Hello! I'm part of the team that's working on beta.mybinder.org and related stuff, and am extremely interested in the idea of a 'badge that represents that your code is reproducible, and has been reproduced by CI'. Funnily, I also built the PAWS stuff, which is awesome to find in completely unrelated contexts :D
Apologies for reviving a closed issue. I am also interested in the reproducibility badge (which is not necessarily the same as the
binderbadge). I came across @mwoodbri 's jupyter-ci. Any plans for this to be used or people that currently use this?(Also cc'ing @yuvipanda as he has been involved in
repo2docker.)@cgpu Hi! At the time
jupyter-ciwas an answer to "what's the simplest possible way to generate a badge and notify of validation failure?". Simply porting the project to GitHub and GitHub Actions (updating to latest Jupyter best-practice in the process, if necessary) would be a great start. But a more general solution involvingrepo2dockerand/or Binder would be even better!@cgpu Hi! At the time
jupyter-ciwas an answer to "what's the simplest possible way to generate a badge and notify of validation failure?". Simply porting the project to GitHub and GitHub Actions (updating to latest Jupyter best-practice in the process, if necessary) would be a great start. But a more general solution involvingrepo2dockerand/or Binder would be even better!Hi @mwoodbri , thank you for the prompt response and the background information! I am really fond of the idea of having a binary
does it reproducebadge of honour,jupyter-cisounds exactly like that. I am trying to reproduce.ipynbaccompanied publications as part of a workshop exercise and I am really struggling. It would be nice to know before hands which ones to invest time on, hence the interest. Additionally, it would be nice to have/create a hub orawesomerepo with the ones that are reproducible only (verified by ci).jupyter-ciis currently only on gitlab, I know it's the same in essence, but the community is much more active in Github, it would be nice if you have plans to maintain the repo + idea long term to bring it over here. Just a thought.Thanks once again!
@cgpu Here's a version converted to use GitHub Actions: https://github.com/mwoodbri/jupyter-ci
Reacted by Daniel Mietchen and Christina Chatzipantsiou@cgpu Here's a version converted to use GitHub Actions: https://github.com/mwoodbri/jupyter-ci
@mwoodbri thank you! Time for me to test now :)
@Daniel-Mietchen thank you for providing the reproducibility cafe space for further discussions 2 years after the start of the initiative. Feel free to close this.
Reacted by Lane RasberryHello everyone. It has been a while since the last post in this thread, but I am happy to report that there is now a preprint that reports on a reproducibility analysis of the Jupyter notebooks associated with publications available via PubMed Central: Computational reproducibility of Jupyter notebooks from biomedical publications — joint work with @Sheeba-Samuel . Here is the abstract:
Jupyter notebooks allow to bundle executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. Here, we analyze the computational reproducibility of 9625 Jupyter notebooks from 1117 GitHub repositories associated with 1419 publications indexed in the biomedical literature repository PubMed Central. 8160 of these were written in Python, including 4169 that had their dependencies declared in standard requirement files and that we attempted to re-run automatically. For 2684 of these, all declared dependencies could be installed successfully, and we re-ran them to assess reproducibility. Of these, 396 notebooks ran through without any errors, including 245 that produced results identical to those reported in the original. Running the other notebooks resulted in exceptions. We zoom in on common problems and practices, highlight trends and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.
For data and code, see https://doi.org/10.5281/zenodo.6802158 .
I'll keep this thread open until the paper is formally published, and invite your comments in the meantime. Extra pings to some of you who have contributed to this thread before: @mwoodbri @khinsen @yuvipanda @rossmounce @tompollard @RomanGurinovich @choldgraf @JosephMcArthur .
Reacted by Daniel S. KatzReacted by Daniel S. KatzCongrats @Daniel-Mietchen, excellent! I look forward to reading the paper, and will be sure to include it in my reading list for next year's reproducible research course I teach at Berkeley!
cc @facusapienza21.
Reacted by Daniel Mietchen and Lane RasberryLots of twitter interest in this preprint. Might be good to dig a bit deeper into all those unknown dependency resolution issues - or perhaps just feature a couple of examples as a panel. There were some comments regarding Docker.
Reacted by Daniel Mietchen and Lane RasberryThanks @Daniel-Mietchen for the update! The preprint is on my e-book reader.
Reacted by Daniel MietchenHoly shit this is AWESOME!
Reacted by Daniel Mietchen and Lane RasberryWe're still working on the revision of the paper but here are the slides of our JupyterCon 2023 talk: https://doi.org/10.5281/zenodo.7854503 .
Slide 23 — How you can get involved — asks for community input along a number of dimensions, which I am copying here.
- How shall we share the dataset?
- How would you use it?
- Which variables (article/ repo/ notebook etc.) to look at?
- Research questions
- Teaching/ learning opportunities
- What would be useful follow-ups?
- Anyone interested in helping turn this into a service? NumFOCUS?
- Who wants to do this for other languages?
- Would it be useful to publish reports about computational reproducibility?
- Would you like to publish such reports
We recently submitted the revision of the paper — see https://arxiv.org/abs/2308.07333 for the latest preprint, which describes a complete re-run of our pipeline and provides some more contextualization. In the discussion, we also briefly touch upon scaling issues with such reproducibility studies, mentioning ReScience (pinging @khinsen) as an example. We are keen on putting this dataset to use, e.g. in educational settings (cc @fperez ).
As always, comments etc. are welcome.
Reacted by Konrad Hinsen and Lane RasberryDear all,
thanks for your participation in this thread.The paper on this (with @Sheeba-Samuel) was published yesterday: Computational reproducibility of Jupyter notebooks from biomedical publications, https://doi.org/10.1093/gigascience/giad113 .
We remain interested in
- ways to integrate our data with research and education workflows around reproducibility
- collaborating on similar analyses for
- other research fields
- languages other than Python
- other types of notebooks
- exploring how such automated approaches can be integrated with peer review workflows
@mwoodbri @fperez @khinsen @yuvipanda @rossmounce @tompollard @RomanGurinovich @choldgraf @JosephMcArthur .
With that, I am closing this ticket after nearly 7 years - feel free to open up new ones in relevant places to discuss potential follow-ups.
Reacted by YuviReacted by Lane RasberryReacted by Konrad HinsenExtremely impressive feat! Well done @Sheeba-Samuel & @Daniel-Mietchen !
Reacted by Lane RasberryReacted by Daniel Mietchen
Jupyter notebooks are a popular vehicle these days to share data science workflows. To get an idea of best practices in this regard, it would be good to analyze a good number of them in terms of their reproducibility and other aspects of usability (e.g. documentation, ease of reuse).
A search in PubMed Central (PMC) reveals the following results:
With currently just 102 hits, a systematic reanalysis seems entirely doable and could perhaps itself be documented by way of reproducible notebooks that might eventually end up being mentioned in PMC.
A good starting point here could be An open RNA-Seq data analysis pipeline tutorial with an example of reprocessing data from a recent Zika virus study, for which both a Jupyter notebook and a Docker image are available.
I plan to give a lightning talk on this. Some background is in this recent news piece.