[PRE REVIEW]: Fast Resampling and Monte Carlo Methods in Python

[editorialbot]

Submitting author: @mdhaber (Matt Haberland)
Repository: https://github.com/mdhaber/scipy
Branch with paper.md (empty if default branch): joss_resampling
Version: 1.11.0.dev0+1225.ca8d481
Editor: @jbytecode
Reviewers: @coatless, @SaranjeetKaur
Managing EiC: George K. Thiruvathukal

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/59d971c467460f42be7168b87a1dfbd4"><img src="https://joss.theoj.org/papers/59d971c467460f42be7168b87a1dfbd4/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/59d971c467460f42be7168b87a1dfbd4/status.svg)](https://joss.theoj.org/papers/59d971c467460f42be7168b87a1dfbd4)

Author instructions

Thanks for submitting your paper to JOSS @mdhaber. Currently, there isn't a JOSS editor assigned to your paper.

@mdhaber if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=5.20 s (488.6 files/s, 165092.8 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Python                          960          80209         144387         221351
C                               313          13419          33524          77926
Fortran 77                      409           3798          71804          73352
reStructuredText                403           9312           6466          29912
C/C++ Header                    135           3937           7263          18535
Cython                          124           5953           9618          17108
C++                              30           1680           2011          10682
Meson                            88            397            192           4878
JSON                              5             15              0           3303
YAML                             25            217            293           1723
TeX                               5            141            161           1289
diff                              2             44            572            731
INI                               4            221              0            507
Pascal                            3            115              0            466
Bourne Shell                      9             59             89            264
Markdown                          9             64              0            201
SVG                               1              4              0            133
make                              4             40             30            111
TOML                              1             19             46            110
CSS                               1             31             20            106
MATLAB                            5             42             45             94
R                                 1              5             12             67
Bourne Again Shell                2             14             26             48
HTML                              2              5              0             21
Dockerfile                        1              5             31             16
Unity-Prefab                      1              0              0              2
--------------------------------------------------------------------------------
SUM:                           2543         119746         276590         462936
--------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.2307/2331554 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1214/aoms/1177729437 is OK
- 1544-6115.1585 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Wordcount for paper.md is 866

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[mdhaber]

if you have any suggestions for potential reviewers then please mention them here in this thread

Starting from the bottom of the list, as recommended.

Based on listing "Computational Statistics" as a domain and "Python" as a preferred language:
turgeonmaxime, matt-graham, coatless, davismcc, kose-y

Based on listing "Statistics" as a domain and "Python" as a preferred language:
yangli-stat, SaranjeetKaur, JohannesBuchner, ketozhang, zhenmuw.edu

[gkthiruvathukal]

@editorialbot invite @jbytecode as editor

[editorialbot]

Invitation to edit this submission sent!

[jbytecode]

@editorialbot assign me as editor

@gkthiruvathukal thank you!

[editorialbot]

Assigned! @jbytecode is now the editor

[jbytecode]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jbytecode]

Dear @mdhaber

I am the handling editor of your submission. May I ask some questions and could you please clarify the issues listed below?

• The repo is forked from the scipy/scipy repo and the forked one seems to be quite young.
• It seems you are one of the prominent contributors of the scipy main repo.
• In what extend you forked the repo with a few new contributions?
• Does the forked repo improve the previous one?
• The manuscript is written in a way that the scipy is only for resampling methods. Why? Is SciPy created for resampling?
• What is the point of creating a fork of a repo you are already a prominent contributor?
• Why haven't you submit a scipy manuscript with other contributors?

Thank you in advance.

[gkthiruvathukal]

@editorialbot query-scope

[editorialbot]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

[gkthiruvathukal]

@editorialbot commands

[editorialbot]

Hello @gkthiruvathukal, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Add to this issue's reviewers list
@editorialbot add @username as reviewer

# Remove from this issue's reviewers list
@editorialbot remove @username from reviewers

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Assign a user as the editor of this submission
@editorialbot assign @username as editor

# Remove the editor assigned to this submission
@editorialbot remove editor

# Remind an author, a reviewer or the editor to return to a review after a 
# certain period of time (supported units days and weeks)
@editorialbot remind @reviewer in 2 weeks

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for version
@editorialbot set v1.0.0 as version

# Set a value for archive
@editorialbot set 10.21105/zenodo.12345 as archive

# Set a value for branch
@editorialbot set joss-paper as branch

# Set a value for repository
@editorialbot set https://github.com/organization/repo as repository

# Mention the EiCs for the correct track
@editorialbot ping track-eic

# Reject paper
@editorialbot reject

# Withdraw paper
@editorialbot withdraw

# Invite an editor to edit a submission (sending them an email)
@editorialbot invite @(.*) as editor

# Generates the pdf paper
@editorialbot generate pdf

# Recommends the submission for acceptance
@editorialbot recommend-accept

# Accept and publish the paper
@editorialbot accept

# Update data on an accepted/published paper
@editorialbot reaccept

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Flag submission with questionable scope
@editorialbot query scope

# Get a link to the complete list of reviewers
@editorialbot list reviewers

# Open the review issue
@editorialbot start review

[gkthiruvathukal]

@editorialbot query scope

[editorialbot]

Submission flagged for editorial review.

[gkthiruvathukal]

@jbytecode While awaiting your answers, we're going to initiate a query scope here.

[mdhaber]

Thank you for your questions, @gkthiruvathukal. To begin, perhaps I should include the "Additional comments" I entered with my submission:

The URL submitted here is to a branch of my fork of the SciPy library. Since the contributions are parts of this much larger library, the following links may be helpful when assessing the magnitude of the contribution:

• Monte Carlo Test PRs: https://github.com/scipy/scipy/pulls?> q=is%3Apr+label%3Ascipy.stats+is%3Amerged+author%3Amdhaber+monte_carlo_test
• Permutation Test PRs: https://github.com/scipy/scipy/pulls?q=is%3Apr+label%3Ascipy.stats+is%3Amerged+author%3Amdhaber+permutation_test
• Bootstrap PRs: https://github.com/scipy/scipy/pulls?q=is%3Apr+label%3Ascipy.stats+is%3Amerged+author%3Amdhaber+bootstrap
• Implementation Code: https://github.com/scipy/scipy/blob/main/scipy/stats/_resampling.py
• Test Suite: https://github.com/scipy/scipy/blob/main/scipy/stats/tests/test_resampling.py

Since there are several questions about my fork of SciPy, it may also help to clarify that this manuscript is not about my fork of SciPy or another a new repository. The manuscript claims the enhancements listed above to the main SciPy repository.

This is based on Should I write my own software or contribute to an existing package?.

---

Responses to individual questions:

The repo is forked from the scipy/scipy repo and the forked one seems to be quite young.
It seems you are one of the prominent contributors of the scipy main repo.
In what extend you forked the repo with a few new contributions?

The features claimed in the JOSS manuscript have been under development for about two years, starting with scipy/scipy#13371.

The joss_resampling branch in my fork of the SciPy repository is identical to SciPy 1.10.0, released last week, except for the addition of the JOSS manuscript. I took this approach based on "Submitting a paper to JOSS".

The joss_resampling branch is temporary; it is only for the submission of this manuscript.

Does the forked repo improve the previous one?
What is the point of creating a fork of a repo you are already a prominent contributor?

I use my fork of SciPy, mdhaber/scipy, to submit pull requests to the main repository, scipy/scipy. The manuscript is about features added by pull requests to the SciPy main repository. All of these have been merged, and the claimed features are available in SciPy 1.10.0; see e.g. scipy.stats.bootstrap, scipy.stats.permutation_test, and scipy.stats.monte_carlo_test.

The manuscript is written in a way that the scipy is only for resampling methods. Why? Is SciPy created for resampling?

SciPy is a very broad scientific computing package for Python. These resampling methods were added to the scipy.stats subpackage of SciPy. To avoid this confusion, I included statements like:

Please let me know how I can clarify this in the manuscript.

Why haven't you submit a scipy manuscript with other contributors?

I am the only author of the contributions claimed by the manuscript. In the Acknowledgements section of the manuscript, I thanked those who reviewed my pull requests. The other features of SciPy, which have other authors, are not claimed by the manuscript.

[gkthiruvathukal]

@mdhaber I am actually satisfied with your responses. I'm going to ask @arfon to chime in here, because I don't have experience as EIC or editor with this type of submission. I think for me the biggest question is whether this "fork" is likely to be contributed upstream, perhaps as a pull request. This is a concern I have for any substantial fork, not just your project. If not, is there some plan to package it so it "tracks" SciPy closely enough so it can remain useful as a standalone package (or plugin, if applicable). I am worried that few researchers would use it in its current form (even if it is awesome otherwise) if is not well-integrated within the ecosystem.

Thanks @jbytecode for having me chime in. I think it is mostly compliant with our criteria, so it is a matter of getting a second look from Arfon before we proceed to edit it.

[arfon]

Thanks for the input @mdhaber and ping @gkthiruvathukal and @jbytecode. So just to clarify, the contributions outlined in this submission have already made it 'upstream' back to scipy/scipy?

[mdhaber]

Yes, they have.

[arfon]

@gkthiruvathukal – I think this is OK to proceed. @gkthiruvathukal – One thing we're going to need to do here is make it easy for reviewers to understand what exactly we're asking them to review.

@mdhaber – your help here outlining the exact capabilities added to SciPy is appreciated :-)

[arfon]

@mdhaber we're currently managing a large backlog of submissions and the editor most appropriate for your area is already rather busy.

For now, we will need to waitlist this paper and process it as the queue reduces. Thanks for your patience!

[mdhaber]

No problem. Thanks!

[jbytecode]

@arfon, @gkthiruvathukal - thank you for joining the conversation and your help.

@mdhaber - thank you for your answers and the explaination.

@arfon - I think we can remove the waitlist tag, right? As the handling editor can I proceed with my editorial stuff?

[gkthiruvathukal]

@arfon I agree it is important for them to know what they are reviewing. I believe this is already covered in @mdhaber's response to my query above. I tagged it with the 🎉. Reviewers can see the items of focus beginning with the "I use my fork of SciPy" sentence, which lists the claimed features associated with this JOSS submission.

@jbytecode Assuming your reviewers are available, I think you can remove the waitlist tag.

[jbytecode]

👋👋👋 Dear @turgeonmaxime and @matt-graham 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled: "Fast Resampling and Monte Carlo Methods in Python". You can find more information at the top of this Github issue (#5047).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

[jbytecode]

👋👋👋 Dear @coatless and @davismcc 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled: "Fast Resampling and Monte Carlo Methods in Python". You can find more information at the top of this Github issue (#5047).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

[coatless]

Sure.

[jbytecode]

@editorialbot add @coatless as reviewer

@coatless - thank you for accepting our invitation!

[editorialbot]

@coatless added to the reviewers list!

[matt-graham]

Hi @jbytecode - apologies for the delay in replying. I'm currently reviewing a couple of other JOSS submissions so I won't be able to help this time as I don't think I will be able to get to this in a timely manner.

[jbytecode]

Dear @matt-graham,

Thank you for the reply. I hope we will work together on future projects.

Best wishes!

[jbytecode]

👋👋👋 Dear @kose-y 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled: "Fast Resampling and Monte Carlo Methods in Python". You can find more information at the top of this Github issue (#5047).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

We've already assigned the first reviewer. We need a second one. I hope you accept our invitation.

Thank you in advance!

[jbytecode]

👋👋👋 Dear @SaranjeetKaur 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled: "Fast Resampling and Monte Carlo Methods in Python". You can find more information at the top of this Github issue (#5047).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

We've already assigned the first reviewer. We need a second one. I hope you accept our invitation.

Thank you in advance!

[SaranjeetKaur]

That would be great @jbytecode! Happy to accept the invitation!

[jbytecode]

@editorialbot add @SaranjeetKaur as reviewer

@SaranjeetKaur - thank you for accepting our invitation.

Now we have two reviewers for this submission.

The review will start in a separate thread. I'll introduce the instructions there.

@coatless

[editorialbot]

@SaranjeetKaur added to the reviewers list!

[jbytecode]

@editorialbot start review

[editorialbot]

OK, I've started the review over in #5092.

[kose-y]

@jbytecode Sorry for the late reply. Unfortunately, I am unlikely to get to this in a timely manner. I hope we work together on a paper later.

[turgeonmaxime]

@jbytecode I'm happy to see you already have two reviewers and the pre-review has been closed. I'm not available to review at this moment, but looking forward to reviewing for JOSS again in the future.

[jbytecode]

@kose-y, @turgeonmaxime - Thank you for the response. I hope we will get together in future works.