[REVIEW]: A Python module to combine p values arising from discrete tests

[editorialbot]

Submitting author: @Wrzlprmft (Gerrit Ansmann)
Repository: https://github.com/BPSB/combine-p-values-discrete
Branch with paper.md (empty if default branch):
Version: 1.2.2
Editor: @vissarion
Reviewers: @steppi, @mdhaber
Archive: 10.5281/zenodo.8338798

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f"><img src="https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f/status.svg)](https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@steppi & @mdhaber, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @vissarion know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @mdhaber

📝 Checklist for @steppi

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.21 s (119.7 files/s, 13481.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          14            398            393           1375
reStructuredText                 2             83             15            164
make                             1             28              6            143
YAML                             4             10              0             80
TeX                              1              5              0             44
Markdown                         1             13              0             43
JSON                             1              0              0             11
INI                              1              0              0              4
-------------------------------------------------------------------------------
SUM:                            25            537            414           1864
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 638

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[mdhaber]

Review checklist for @mdhaber

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/BPSB/combine-p-values-discrete?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
  Yes, here.
• Contribution and authorship: Has the submitting author (@Wrzlprmft) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
  Yes. @Wrzlprmft appears to be the sole author.
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  It's challenging to assess the contribution according to whether it constitutes "three months of work for an indivual". There was a two-month period of work in the beginning, but I can't tell whether it was a full time effort. There also appears to be a recent burst of activity in September, presumably to prepare the package for review.
  • The commit history is nearly two years old, but most activity happened in the first two months or very recently.
  • There are 184 commits. Over the course of three months, that's an average of three commits per day, which may or may not be full-time equivalent.
  • There is one author.
  • There are approximately 3000 lines total, the result of 5000 additions and 2000 deletions. 800+ lines are implementation and 900+ lines of tests, and these lines are reasonably compact (not padded with tons of whitespace).
  • It doesn't look like the work has already been cited.
  • I think it is likely to be used in scientific work. Whether it is cited depends on the disposition of the users to cite open source software!
  • The package is "feature-complete" and useable by others, not "half-baked" or "thin"
    Based on discussions with the author, I'm not sure whether this meets JOSS's "three months of work for an individual" for this individual to the letter, but I don't think it would make sense to reject the work on this basis. Had the same work - or work of lower quality - been completed by someone with less expertise over a greater period of time, the letter of this criterion would have been satisfied, but I don't think the work would be any more worthy of publication.
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
  I do not see any claims about original data.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
  I do not see any claims about original results.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.
  I do not see any claims about real original human or animal research. The "extended example" does not appear to contain real animal data; it uses fictitious data to illustrate the analysis procedure.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
  Yes. I confirmed that pip install combine_pvalues_discrete works on Colab.
• Functionality: Have the functional claims of the software been confirmed?
  I've begun to confirm some of the claims indirectly. For instance, when the null hypothesis of each individual test is true, the distribution of combined p-values produced by the software appears uniform. Also, when combining p-values from continuous tests, the results produced using the software appear to match those produced by scipy.stats.combine_pvalues.
  Followup: I've verified some of the claims more directly. Using scipy.stats.monte_carlo_test, I wrote my own version of Fisher's approach to combining p-values. This is similar to what the author shows in Checking the result. So I haven't done a line-by-line review of all functionality, but the basic functionality is definitely there.
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
  • "The presented module combine_pvalues_discrete implements this approach in a:
    • fast, yes, it is much faster to use the population of p-values rather than simulate them from random data
    • thorough,
    • and tested manner
  • taking care of pitfalls such as correctly handling
    • complements,
    • sidedness, and
    • empirical 𝑝 values
  • as well as handling tedious and error-prone tasks..."
    I have confirmed that the software correctly counts p-values considering numerical tolerances (atol, rtol are important and often neglected) and computes the p-value from counts (see "Permutation P-values should never be zero").
  • "This approach is considerably faster than a permutation test starting at the level of individual datasets."
  • "Thanks to modern computing and NumPy, it is easy to make the number of samples very high and the result very accurate." (in the documentation)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  Yes, here.
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  Yes. The requirements are included in a requirements.txt file.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  Yes. The "extensive example" seems very practical, even if it is not truly real-world data.
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  Yes. I was able to explore features not covered in the tutorials using the rest of the documentation.
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  Yes, automated tests are written using the Pytest framework. I have not verified that the tests are sufficient, but it looks like reasonable care has been taken.
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
  I see guidelines for reporting issues, but I don't see guidelines for contributing or seeking support. All guidelines added here.

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  Yes, I think the description of the purpose is very clear for anyone familiar with null hypothesis significance testing, which is a rather non-specialist audience.
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
  Yes. The paper is clear that the problem is that there are not other options in the Python ecosystem for combining p-values that arise from discrete tests (i.e. tests with discrete null distributions).
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
  Somewhat. The most obvious choice for combining p-values in Python, scipy.stats.combine_pvalues, is only valid for p-values from continuous tests, and the paper cites this. It also cites the R poolr package. However, to more fully satisfy this criterion, the paper could mention other meta-analysis packages in Python (e.g. PythonMeta, PyMARE, statsmodels.stats.meta_analysis) and/or other languages (e.g. R's metap). Yes, it does so more completely now.
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  Yes. I have a few specific questions, but I don't have any substantial suggestions for most of it.
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
  References seem appropriate, but I need to go over them in more detail (e.g. syntax).

[Wrzlprmft]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Wrzlprmft]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[steppi]

Review checklist for @steppi

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/BPSB/combine-p-values-discrete?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@Wrzlprmft) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  I saw your response here, Review of JOSS submission BPSB/combine-p-values-discrete#1. I agree that the level of effort seems appropriate, even if the time spent was not quite 3 months. I don't think it would be unreasonable for this to have taken 3 months.
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
  I've created a simulation which shows the software works beautifully exactly as advertised, here https://gist.github.com/steppi/77357e4456d9f4b92af2c614ec665c99
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)? I think the package is well documented. I left issue Suggestions for improving documentation BPSB/combine-p-values-discrete#3 suggesting to add examples to the docstrings as suggested in the JOSS review guidelines

Good: All functions/methods are documented including example inputs and outputs
OK: Core API functionality is documented
Bad (not acceptable): API is undocumented

because I agree this is good practice. I also asked if lines could be wrapped in the docstrings to make them easier to read in a terminal. Right now there lines up to over 400 characters wide.

• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified? There is a thorough test suite and automated CI. I think it would be nice to have documentation for how to run the test suite and have submitted a relevant issue here Documentation for how to run the test suite BPSB/combine-p-values-discrete#2.
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[vissarion]

Hi, @Wrzlprmft could you please update us on your status in addressing issues raised by @mdhaber?

Hi, @steppi could you please update us on the progress of your review? I see there are some checkboxies unchecked. Is it because there are still in progress or there is some issue with them?

[Wrzlprmft]

Hi, @Wrzlprmft could you please update us on your status in addressing issues raised by @mdhaber?

In my understanding, there is nothing for me to address at the moment. I addressed @mdhaber’s last comments here (and the commits referenced therein). Since my last reply was mostly asserting his understandings, I didn’t expect a response and assumed no further clarification or changes were needed.

[mdhaber]

Yes, my comments were addressed, and I only planned to do a more detailed check of the reference section.

[steppi]

Hi, @steppi could you please update us on the progress of your review? I see there are some checkboxies unchecked. Is it because there are still in progress or there is some issue with them?

Sorry @vissarion, since agreeing to review this I ended up being busier than expected but things are settling down. I will be able to complete my review within two weeks. The unchecked boxes are for things that are still in progress.

[mdhaber]

I completed my checklist. References look good except for "metap" probably isn't intended to be capitalized in the rendered document.

[Wrzlprmft]

References look good except for "metap" probably isn't intended to be capitalized in the rendered document.

Fixed. While I disagree with lowercasing proper names, this is reproducing the lack of capitalisation of the cited source.

[vissarion]

@mdhaber do you recommend to accept this paper?

[vissarion]

Hi @steppi, do you have any news from this review?

[steppi]

Sorry for the delay. I’ll get it done this week.

[mdhaber]

@mdhaber do you recommend to accept this paper?

Yes

[steppi]

I've completed my review and recommend this paper for publication. I would like to see the updates I suggested made to the documentation but don't think this is a blocker.

[Wrzlprmft]

@Wrzlprmft it seems that both reviewers recommend acceptance!

So it does.

Thanks to the reviewers (@mdhaber and @steppi) for their diligent criticism and constructive comments.

When a submission is ready to be accepted, we ask that the authors issue a new tagged release of the software (if changed), and archive it (see this guide). Please do this and post the version number and archive DOI here.

• Version: 1.2.2
• DOI: 10.5281/zenodo.10684537

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Wrzlprmft]

@Wrzlprmft please edit the zenodo record so that the title, author names and affiliation are the same as in paper.

Hopefully done. I added the paper title as an alternative title. Please tell me if it should be the main title.

[vissarion]

@Wrzlprmft please change the main title of zenodo to "A Python module to combine 𝑝 values arising from1
discrete tests." i.e. as in JOSS paper so that the DOI looks correct and has the same title as in your paper.

https://zenodo.org/records/10684537

[Wrzlprmft]

@Wrzlprmft please change the main title of zenodo to "A Python module to combine 𝑝 values arising from1 discrete tests." i.e. as in JOSS paper so that the DOI looks correct and has the same title as in your paper.

https://zenodo.org/records/10684537

Done (except for the stray “1” which I presume to be an error).

[vissarion]

@editorialbot set 10.5281/zenodo.8338798 as archive

[editorialbot]

Done! archive is now 10.5281/zenodo.8338798

[vissarion]

@editorialbot set 1.2.2 as version

[editorialbot]

Done! version is now 1.2.2

[vissarion]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK
- 10.2202/1544-6115.1585 is OK
- 10.5281/zenodo.4266738 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[vissarion]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vissarion]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK
- 10.2202/1544-6115.1585 is OK
- 10.5281/zenodo.4266738 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#5038, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[arfon]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Ansmann
  given-names: Gerrit
  orcid: "https://orcid.org/0000-0002-5472-7067"
doi: 10.5281/zenodo.8338798
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Ansmann
    given-names: Gerrit
    orcid: "https://orcid.org/0000-0002-5472-7067"
  date-published: 2024-02-29
  doi: 10.21105/joss.06096
  issn: 2475-9066
  issue: 94
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 6096
  title: A Python module to combine p values arising from discrete
    tests.
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.06096"
  volume: 9
title: A Python module to combine $p$ values arising from discrete
  tests.

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.06096 joss-papers#5071
1. Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.06096
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[Wrzlprmft]

🎉 Everything looks fine to me. Thanks to everybody involved.

However, I cannot close this issue (or find the editorial technical team to contact about this) as instructed. It seems like @vissarion or @arfon has to do this.

[arfon]

@steppi, @mdhaber – many thanks for your reviews here and to @vissarion for editing this submission! JOSS relies upon the volunteer effort of people like you and we simply wouldn't be able to do this without you ✨

@Wrzlprmft – your paper is now accepted and published in JOSS ⚡🚀💥

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.06096/status.svg)](https://doi.org/10.21105/joss.06096)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.06096">
  <img src="https://joss.theoj.org/papers/10.21105/joss.06096/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.06096/status.svg
   :target: https://doi.org/10.21105/joss.06096

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss