Round 3, Editor's comments (decision: reject)

[labarba]

Received: February 15, 2022.

Your manuscript "High-productivity, high-performance workflow for virus-scale electrostatic simulations with Bempp-Exafmm" has now been seen by 3 referees (Reviewers # 3, # 5, and # 6), whose comments appear below. In the light of their advice, we have decided that we cannot offer to publish your manuscript in Nature Computational Science. [emphasis added]

I would like to briefly explain the rationale of our decision to you. As you know, your paper had originally 4 reviewers; Reviewer # 3 was enlisted specifically to give a more technical perspective on the paper, as this reviewer is an expert in Poisson-Boltzmann and numerical analysis. Given the initial disagreements between this reviewer and your team, and in order for us to make a decision as fair as possible, we decided to enlist 2 extra reviewers (Reviewers # 5 and # 6), who are also experts in Poisson-Boltzmann and numerical analysis. Based on the initial assessment given by Reviewers # 5 and # 6, we then decided to send a Revise decision.

The reviews are now back, and none of the reviewers (Reviewers # 3, # 5 and # 6) are satisfied with the revision. Given the previous disagreements between your team and Reviewer # 3, we initially focused our attention to Reviewers # 5 and # 6. Unfortunately, Reviewer # 5 only added their comments to the section that is visible to the editors, and they haven't replied to us about moving their comments to the section that is visible to the authors. But to summarize their comments, this reviewer mentioned that providing more evidence for the accuracy of the software is important, but that their suggestion was not entirely taken into account. Overall, they mentioned that they are not enthusiastic about the paper, and that they think that the paper does not present proper benchmarking and comparison. Regarding Reviewer # 6, this reviewer was also not entirely satisfied with the revision, pointing out that their validation suggestion was not properly addressed. This reviewer also
highlights similar issues as mentioned by Reviewer # 3 regarding the performance study and comparisons.

Therefore, given that none of these reviewers are supportive of the paper, we unfortunately will have to decline publication of your manuscript. I understand that this is disappointing, given all the effort and time that you and your team have put in this paper (which we really appreciated), but we need to make difficult decisions based on the reviews, and also based on the different expertise that we enlist.

PS: We did not understand Reviewer # 3's comments on the code (that the license prevents anonymous downloading and verification), but I just wanted to clarify that this particular comment was not taken into account in our decision.

I am sorry that we cannot be more positive on this occasion, but hope that you find the referees' comments helpful when preparing your paper for resubmission elsewhere.

[labarba]

On Sep 5, 2022, at 12:29 PM, Lorena Barba wrote:

Dear Fernando,

In February 2022, you sent us the editorial decision to not publish our manuscript "High-productivity, high-performance workflow for virus-scale electrostatic simulations with Bempp-Exafmm." We of course humbly accepted your decision.

It was our intention to attempt a transfer of the manuscript to Scientific Reports. As we learned that the transfer process would carry over the complete history of peer review, we recognized the need to address the final round of reviewer comments (attached to your decision letter, 2/15/22).

If you recall, the manuscript was originally seen by 4 referees, and after one revision, two of them were satisfied; Reviewer 1 was unsure of the target audience while Reviewer 3 remained negative. You requested additional information from us, and then brought two new referees for a third round of review. After receiving their comments, we submitted a second revision on December 2021.

The remaining reviewers (3, 5, 6) were unsatisfied with our second revision, in particular because we declined to follow Reviewer 3 in adding new experiments using binding energy. We had computed solvation energy up to that point; computing binding energy accurately is more difficult because it is the difference between two solvation energies, giving a small numerical result.

In view of the need to address this final round of critiques for the transfer of the manuscript to Scientific Reports, we decided to take extra time to perform a thorough study of binding energy calculations, comparing with other codes. This had been suggested by Reviewer 3, who referenced two papers: Nguyen et al. 2017 (https://doi.org/10.1002/jcc.24757), and Harris, Boschitsch and Fenley 2013 (https://doi.org/10.1021/ct300765w).

On completing the new study on the binding energy of several protein complexes computed with different codes, we found that results with our method fall in all cases within the range of others. A helpful way to visualize this finding is shown in Figure 3 of our new revision. Here, each vertical line corresponds to one protein complex. The different-color markers correspond to each code (MIBPB, Delphi, PBSA, APBS, Bempp, ACG). Clearly, the results with Bempp are up to standard.

Given the strength of this new evidence, before completing the transfer to Scientific Reports, we thought it was worth discussing with you if your team might be willing to consider this third revision for publication. We are certain that this revision addresses the remaining concerns of Reviewer 3 and 6.

As before, every result in our paper is fully reproducible, with all details to redo the calculations available on a GitHub repository.

I'm attaching here a "diff" of this revision compared to the decision version, the supplement, and a detailed response to the final round of reviewer comments.

Please let me know if you would like to discuss further. If your team is not willing to consider this, we will proceed to make the transfer.

[labarba]

Figure 3: Binding energies of 9 Barnase-Barstar complexes computed using various PB solvers.

[labarba]

On Sep 23, 2022, at 3:19 PM, Fernando Chirigati wrote:

Dear Lorena,

Thank you for your patience on this.

We had a meeting this morning about your paper. After looking at the new version of your manuscript and the new response to referees’ document, we agree that the new experiments do strengthen the paper and better address the referees’ concerns. However, we would depend mainly on the referees’ opinion to understand whether or not these concerns were satisfactorily addressed.

We would therefore be willing to re-consider your paper and send the new revision back to Referees #3 and #6 – because there were no comments for you to address from Referee #5, we would not contact this referee again. However, please note that, if either Referee #3 or #6 (or both referees) are unavailable at this time, we would need to contact new referees, which would likely delay the decision on your paper: we would need to give new referees more time to appropriately look at the manuscript and the review history (we would though ask them to focus on how the previous comments were addressed), and because of the start of the semester, it has been particularly challenging to find referees.

I’m mentioning this so we are all on the same page about the review process. In addition, if time of publication is very important to you and your co-authors, I should mention that, if you decide to transfer your paper to Scientific Reports, they might consider your paper ready for publication without sending it back to referees; I cannot really guarantee that this would happen, given that our teams are editorially independent – this is just based on my experience with papers transferred from our journal to theirs. Of course, I’m not suggesting here that you choose the potentially ‘quick’ publication route: I just want to better explain the different potential scenarios to you so you can make a more informed decision!

If you decide to have your new revision considered by us, please send me a new cover letter and a clean version of your new revision (meaning, without diffs or tracked changes), and I can start the appeal process.

Let me know if you have any questions.

[labarba]

On Nov 18, 2022, at 10:27 AM, Fernando Chirigati wrote:

Dear Lorena,

Just wanted to provide you with an update with respect to your manuscript NATCOMPUTSCI-21-0283C-Z, entitled "High-productivity, high-performance workflow for virus-scale electrostatic simulations with Bempp-Exafmm".

Referee #3 decided not to review the new revision given that they did not think it would be a fair and unbiased report -- a comment that I actually appreciated. We were able to find a new referee (#7), but Referee #6, who was originally responsive to our request (and reminders), has since not responded to us anymore. Our editorial team don't think it would be fair to you and your co-authors to make a decision on your manuscript solely based on one report (from Referee #7), so we are currently trying to find another referee.

We are really sorry for the delay, but hope you can understand that we want to be as fair as possible.

Let me know if you have any questions or concerns.

[labarba]

On Dec 5, 2022, at 6:52 PM, Fernando Chirigti wrote:

Dear Professor Barba,

Your manuscript "High-productivity, high-performance workflow for virus-scale electrostatic simulations with Bempp-Exafmm" has now been seen by 1 extra referee, whose comments appear below. Our original plan was to invite Referees #3 and #6 again to comment on your replies to their last concerns. However, Referee #3 decided not to review the new revision given that they did not think it would be a fair and unbiased report, and Referee #6 was unresponsive. We did find a new Referee (#7), with a similar expertise as Referees #3 and #6. We tried our best to have another referee to look at the paper and reviews; however, while we eventually found one, this new referee (#8) became unresponsive too.

In the light of Referee #7's advice, and taking into account the entire review process of this manuscript, unfortunately, we have decided that we cannot offer to publish your manuscript in Nature Computational Science.

Referee #7 has an expertise in developing and implementing Poisson-Boltzmann model-related algorithms and software (like Referees #3 and #6), and their comments were mostly in line with Referees #3 and #6 (except for some of the comments, such as the ones related to practical usage, which we did not take into account when making our decision). We feel that the criticisms related to novelty and numerical accuracy are sufficiently important as to preclude publication of your work in Nature Computational Science.

I am sorry that we cannot be more positive on this occasion. I also apologize for the delay in making a decision on your paper, as finding referees at this time proved to be particularly challenging.