Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: scikit-posthocs: Pairwise multiple comparison tests in Python #1169

Closed
whedon opened this Issue Jan 12, 2019 · 50 comments

Comments

Projects
None yet
6 participants
@whedon
Copy link
Collaborator

whedon commented Jan 12, 2019

Submitting author: @maximtrp (Maksim Terpilowski)
Repository: https://github.com/maximtrp/scikit-posthocs
Version: v0.5.1
Editor: @katyhuff
Reviewer: @mnarayan, @djmitche
Archive: 10.5281/zenodo.2639732

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/c10b7f55a5206105c66428bd0e40a167"><img src="http://joss.theoj.org/papers/c10b7f55a5206105c66428bd0e40a167/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/c10b7f55a5206105c66428bd0e40a167/status.svg)](http://joss.theoj.org/papers/c10b7f55a5206105c66428bd0e40a167)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mnarayan & @djmitche, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @katyhuff know.

Please try and complete your review in the next two weeks

Review checklist for @mnarayan

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: v0.5.1
  • Authorship: Has the submitting author (@maximtrp) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @djmitche

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: v0.5.1
  • Authorship: Has the submitting author (@maximtrp) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 12, 2019

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @mnarayan, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 12, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 12, 2019

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Jan 13, 2019

In

Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

I'm unfamiliar with what "should have a DOI" means. But I don't see any DOI's in the references. @mnarayan is that something you're familar with and could look at more deeply?

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Jan 13, 2019

I have created and linked issues for all of the un-checked boxes in my review above (except DOIs, as noted). I will watch for updates and check boxes as the issues are resolved.

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Jan 19, 2019

@djmitche Thank you for your thorough review!

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Jan 26, 2019

I'm not aware of any further changes yet -- @maximtrp please let me know if there's something I should look at to check off another box..

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Jan 26, 2019

@djmitche Sure, I'm still improving the code and docs. The reviews uncovered the problems I've been laying aside for some time. Now, I have to deal with them, and it's taking more time than expected. I'll let you know as soon as I finish.

@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Mar 7, 2019

@maximtrp : How is this going? I don't mean to pester, but wanted to make sure that you'd like this review to continue. One option, if you need substantial time, is to withdraw the submission (close this issue) and resubmit when you're at a new code version that will satisfy the key features that the review has uncovered.

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Mar 9, 2019

@katyhuff Sorry for a long delay. I will finish in a few days.

@mnarayan

This comment has been minimized.

Copy link
Collaborator

mnarayan commented Mar 23, 2019

@maximtrp I can see a lot of code improvements, you are almost there!
My only sticking points are related to clarifying the functionality and description of what this package offers. Even the PMCMRplus package that is the inspiration for yours is a bit opaque. This shouldn't take you too long however.

@mnarayan

This comment has been minimized.

Copy link
Collaborator

mnarayan commented Mar 31, 2019

The scikit-posthocs package implements several statistical testing procedures to conduct pairwise multiple comparisons after the global null in an ANOVA has been rejected. It fills a clear gap in the python ecosystem and has been created as an analog to the PMCMRplus package in R.

@maximtrp has addressed all my issues and the package implementation meets all the requirements for JOSS.

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Mar 31, 2019

@mnarayan there are a few more check-boxes in your section of the review.

@whedon commands

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Mar 31, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Mar 31, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Mar 31, 2019

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Mar 31, 2019

Looks good to me!

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Mar 31, 2019

@djmitche @mnarayan Thank you for reviews! Your comments were very helpful.

@djmitche

This comment has been minimized.

Copy link
Collaborator

djmitche commented Apr 13, 2019

@katyhuff what is the status here? I see this is still assigned to me - is there anything further required?

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Attempting to check references...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

PDF failed to compile for issue #1169 with the following error:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 16 0 16 0 0 400 0 --:--:-- --:--:-- --:--:-- 410
sh: 0: getcwd() failed: No such file or directory
pandoc: 10.21105.joss.01169.pdf: openBinaryFile: does not exist (No such file or directory)
Looks like we failed to compile the PDF

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019


OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None
@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

@whedon check references

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Attempting to check references...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019


OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None
@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

Thanks so much for the ping @djmitche for the ping! It's a hectic end of the semester and I missed the notification that the comments had been handled.

Thanks @mnarayan and @djmitche for your excellent reviews. The care you took and attention to detail that you supplied is what makes JOSS possible.

@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

Thank you @maximtrp for a strong submission and for engaging actively in the review process! I have looked over the paper and have conducted a high level review of the code itself. Everything looks ship-shape to me and we are nearly ready to move forward with acceptance. That said, if you have a moment, I would like to see all existing references contain a DOI where possible. Some of the citations you've chosen are a bit older, and so they don't have DOIs. This is reasonable, though Pandas and SciPy both have options for newer citations if you are so inclined to use those instead. This is optional, in my opinion, since you seem to have used the preferred citations for all packages (which happen to not contain DOIs).

At this point, it'll be good for you as well to to double check the paper, review any lingering details in your code/readme/etc., and then make an archive of the reviewed software in Zenodo/figshare/other service. Please be sure that your created DOI metadata (title, authors, etc.) matches this JOSS submission. Once that's complete, please update this thread with the DOI of the archive, and I'll move forward with accepting the submission! Until then, now is your moment for final touchups!

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Apr 14, 2019

@katyhuff I would like to keep these references as they are indeed the preferred ones. I have created a release of scikit-posthocs on Zenodo. Here is the DOI: 10.5281/zenodo.2639732. Everything else is okay. Thank you!

@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

@whedon set 10.5281/zenodo.2639732 as archive

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

OK. 10.5281/zenodo.2639732 is the archive.

@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

@whedon accept

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Attempting dry run of processing paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Check final proof 👉 openjournals/joss-papers#625

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#625, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019


OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None
@katyhuff

This comment has been minimized.

Copy link
Member

katyhuff commented Apr 14, 2019

Thanks @maximtrp for your submission, patience, and cooperation with the process!

Thank you again @mnarayan and @djmitche for your detailed reviews!

This paper should be ready to accept @openjournals/joss-eics - over to you! (Please note the discussion about DOIs, above.)

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Apr 14, 2019

@whedon set v0.5.1 as version

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

OK. v0.5.1 is the version.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Apr 14, 2019

@maximtrp — I noticed the version on your Zenodo archive didn't match what we had here, went to your repo, and see your version tag as v0.5.1 ... so I updated the version. Please confirm that is what we want.

@maximtrp

This comment has been minimized.

Copy link

maximtrp commented Apr 14, 2019

@labarba That's right. It should be v0.5.1. Thank you!

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Apr 14, 2019

@whedon accept deposit=true

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

Doing it live! Attempting automated processing of paper acceptance...

@whedon whedon added the accepted label Apr 14, 2019

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

  1. Check final PDF and Crossref metadata that was deposited 👉 openjournals/joss-papers#626
  2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01169
  3. If everything looks good, then close this review issue.
  4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Apr 14, 2019

Congratulations, @maximtrp, your JOSS paper is published! 🚀

Big thanks to our editor: @katyhuff, and the reviewers: @mnarayan, @djmitche — your contribution to JOSS is invaluable! 🙏

@labarba labarba closed this Apr 14, 2019

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Apr 14, 2019

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01169/status.svg)](https://doi.org/10.21105/joss.01169)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01169">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01169/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01169/status.svg
   :target: https://doi.org/10.21105/joss.01169

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.