[REVIEW]: Pigeons.jl: Distributed sampling from intractable distributions

[whedon]

Submitting author: @nikola-sur (Nikola Surjanovic)
Repository: https://github.com/Julia-Tempering/Pigeons-Paper
Branch with paper.md (empty if default branch):
Version:
Editor: @pitsianis
Reviewers: @nsailor, @georgebisbas
Archive: Pending

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d"><img src="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg)](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@nsailor & @georgebisbas, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @pitsianis know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @nsailor

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

Review checklist for @georgebisbas

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @nsailor, @georgebisbas it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Failed to discover a Statement of need section in paper

[whedon]

Wordcount for paper.tex is 308

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.10 s (503.0 files/s, 350305.9 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
TeX                                     37           1011           1624          30073
Windows Module Definition                1              0              0           1585
Julia                                   10             59             50            330
Ruby                                     1              8              4             45
YAML                                     1              0              0             30
---------------------------------------------------------------------------------------
SUM:                                    50           1078           1678          32063
---------------------------------------------------------------------------------------


Statistical information for the repository '301fc6098cae505466f5f2a7' was
gathered on 2023/11/18.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nikola Surjanovic                1            57              0          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nikola Surjanovic            57          100.0          0.0                7.02

[whedon]

Failed to discover a valid open source license.

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1007/s11222-012-9328-6 is OK
- 10.2307/2348941 is OK
- 10.1093/sysbio/syt022 is OK
- 10.1016/j.cma.2021.114264 is OK
- 10.1615/int.j.uncertaintyquantification.2022038478 is OK
- 10.1038/s41467-022-31830-2 is OK
- 10.1145/3314221.3314642 is OK
- 10.2307/2684568 is OK
- 10.21105/joss.02741 is OK
- 10.1186/s12918-017-0433-1 is OK
- 10.1103/physrevlett.57.2607 is OK
- 10.1111/rssb.12464 is OK
- 10.1093/bioinformatics/17.8.754 is OK
- 10.1093/sysbio/sys029 is OK
- 10.1109/wisp.2007.4447579 is OK
- 10.1007/s11222-008-9110-y is OK
- 10.3390/galaxies11010006 is OK
- 10.3847/1538-4357/ac09ee is OK
- 10.21105/joss.04457 is OK
- 10.1198/106186006x100579 is OK
- 10.1198/jcgs.2010.10039 is OK
- 10.1198/jcgs.2011.10167 is OK
- 10.1073/pnas.1408184111 is OK
- 10.4324/9781003289173-2 is OK
- 10.1111/rssb.12336 is OK
- 10.1145/62959.62969 is OK
- 10.1145/2714064.2660195 is OK
- 10.1080/01621459.1990.10474918 is OK
- 10.1093/sysbio/syq085 is OK
- 10.21105/joss.01816 is OK
- 10.1145/3551624.3555300 is OK

MISSING DOIs

- 10.1111/rssb.12464 may be a valid DOI for title: Non-reversible parallel tempering: a scalable highly parallel MCMC scheme

INVALID DOIs

- None

[whedon]

👋 @georgebisbas, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @nsailor, please update us on how your review is going (this is an automated reminder).

[nsailor]

This work is an implementation of the Parallel Tempering method in Julia, an improvement over MCMC for obtaining samples form otherwise difficult to sample probability distributions.
The package supports parallelization using both Julia's built-in multi-threading features, as well as MPI.jl, a Julia wrapper for MPI.

I greatly appreciate the attention given to what the authors term "strong parallelism invariance" (SPI), roughly meaning that the result should not depend on the degree of parallelism (especially considering the complexities of finite-precision floating-point arithmetic). This package also builds upon a Julia translation of a Java library called Splittable Random previously published by the authors, allowing deterministic random number generation across multiple threads.

Overall, I think this work is a great contribution to the Julia ecosystem, notwithstanding the following minor points:

• Although definitely not a requirement, I think it would be very interesting to see the scaling characteristics of this implementation, especially with respect to the other algorithm parameters, for instance, the number of chains.
  Additionally, the paper mentions Pigeons being able to run on "thousands of MPI-communicating machines". It may be worth clarifying if this is something that has been tested or a possibility given the package's design. Having said that, I was glad to find this issue so it seems this is something on the authors' radar.

• Unfortuantely, I was not able to get a speedup with the toy MVN example using multiple threads (see this issue in the project repository). In general, it would be very helpful to have some hints in the package's documentation for tuning the degree of parallelism for a given problem, especially if the speedup from parallelism is not linear.

• It would be great to have additional examples, as described in this issue

In writing the above, it is possible that I have misunderstood some part of this work, in which case, please feel free to correct me.

[nikola-sur]

Thank you for your comments! We will work on updating the manuscript, package, and documentation to account for your feedback.

[miguelbiron]

Dear @pitsianis -- we were wondering what the expected timeline for the reviews is? Should we perhaps just respond to @nsailor and not wait for @georgebisbas, or should we wait until we have both sets of comments? Thank you.

[pitsianis]

Please answer any outstanding issues; you don't need to do this sequentially. This exchange will also remind @georgebisbas to wrap up his review.

[miguelbiron]

Thanks for the clarification.

[georgebisbas]

Dear @miguelbiron and @pitsianis, my apologies for the delay here.
Feel free to respond to the outstanding issues; I aim to prioritize this review in my task list.

[miguelbiron]

No worries @georgebisbas, thank you for prioritizing

[georgebisbas]

Hi again, apologies for the delay, I have partly written a draft, aiming to complete my review in the next days.

[georgebisbas]

First, I would like to thank the authors for their patience over the last few months.

Pigeons.jl offers a high-level API to leverage shared- and distributed-memory parallelism via Julia's built-in multi-threading features and MPI.jl, a Julia wrapper for MPI.
Overall, this is a great work suitable for JuliaCon proceedings.
My review below lists a few weaknesses/questions that could help clarify my understanding of the work in a few places or act as constructive feedback to improve this work.

Strengths :

• Automated deployment to MPI DMP without necessary user experience
• Strong parallelism invariance is guaranteed.
• Applications are discussed in the Introduction.
• Smooth transition for the reader, problem formulation, and running examples help in that.
• Compatibility with other targets is a plus
• Section 2.2.1 helps to describe Figure 2
• I very much like how code is accompanied by text describing the symbolic math process

Weaknesses/Questions:
I understand and like that it "works out of the box." It feels that authors focus more on correctness rather than performance. A few questions I have, are:

• What are the memory requirements of a "typical large enough" problem to be tackled?
• What are the advantages of DMP versus SMP only?
• Why are any strong scaling graphs not included?
• After reading the paper, I feel that MPI works correctly, but with no clue on why it is needed and what performance benefits it brings to the table

Minor:

• Have any of the additional targets been implemented in the meantime? Could the paper be updated there?

[nikola-sur]

Thank you to both reviewers for their comments! Now that all reviews are in, we will get back to you shortly with updates and responses to the questions raised.