We thank the reviewer for the in-depth examination of our work and for his useful suggestions for the future evolution of ReScience.
Concerning what the reviewer calls a defeatist attitude towards computational reproducibility, that is not at all how we see our approach. The manuscript focuses on reporting our experiences with ReScience. Concerning reproducibility, we note that having it verified by reviewers is not sufficient. We are not aware of any practical experience demonstrating that some other approach is efficient, which is why we consider the reproducibility of software environments an unsolved problem. Many ideas are being explored at the moment, including containerization as the reviewer mentions. The effectiveness of these ideas cannot be judged yet. Nobody knows, for example, if five years from now, the then-current version of Docker will still be able to handle today's container images. We see our attitude not as defeatist, but as prudent. Many of the authors are contributing to the development and evaluation of new ideas, and we will certainly try some of them out in the context of ReScience.
We have reworded some of our statements to make clearer that our doubts about reproducibility apply only to today's state of the art. We have also added a new paragraph on the reproducibility of software environments to the Outlook section.
We do not discuss continuous analysis because ReScience remains a traditional journal in publishing "finished" articles with a time stamp and a DOI. Continuous analysis is part of a different line of improving on the current publishing model that proposes to replace static articles by dynamic and open development processes. This is an interesting approach but for now in a far more experimental state than any of the innovations in ReScience. ReScience repositories being open, anyone is free to base a continuous analysis project on a replication published in ReScience, and we would be happy to support such attempts, but it is not in the focus of our own work for the near future.
We have corrected our description of the Github pricing model, referring to public projects rather than Open Source projects.
We have also added a sentence stating that replication authors should base their work exclusively on the contents of the original article. In practice, we have not yet encountered a case where code for the original article was available.
The reviewer's proposal of a data-driven summary of ReScience operations is interesting but we do not wish to attempt any form of statistical analysis on a dataset of by now 17 entries. We did add a sentence stating the average duration and length (in numbers of comments) of the review process, but any more sophisticated analysis would be premature at this time.
We thank the reviewer for his enthusiastic comments, and we would be happy to welcome him as a contributor to ReScience.
We have corrected the name of Jonathan Buckheit.