Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: The Riffomonas Reproducible Research Tutorial Series #13

Closed
whedon opened this Issue Jun 11, 2018 · 43 comments

Comments

Projects
None yet
5 participants
@whedon
Copy link
Collaborator

whedon commented Jun 11, 2018

Submitting author: @pschloss (Patrick D Schloss)
Repository: https://github.com/riffomonas/reproducible_research
Version: v1.0.0
Editor: @tracykteal
Reviewer: @jhollist
Archive: 10.5281/zenodo.1404230

Status

status

Status badge code:

HTML: <a href="http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36"><img src="http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36/status.svg"></a>
Markdown: [![status](http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36/status.svg)](http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jhollist , please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://jose.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @tracykteal know.

Review checklist for @jhollist

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source for this learning module available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
  • Version: Does the release version given match the repository release (v1.0.0)?
  • Authorship: Has the submitting author (@pschloss) made substantial contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

  • A statement of need: Do the authors clearly state the need for this module and who the target audience is?
  • Installation instructions: Is there a clearly stated list of dependencies?
  • Example usage: Do the authors include examples of how to use the module?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Pedagogy / Instructional design

  • Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
  • Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
  • Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
  • Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
  • Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state the need for this module and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jun 11, 2018

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @jhollist it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jun 11, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jun 11, 2018

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jun 11, 2018

👋 @jhollist — here's where the review happens! Feel free to ask any questions, tagging the author or handling editor, as needed. You can also open issues on the repository of the submission.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jun 13, 2018

@tracykteal I'm hoping you can secure a second reviewer for this submission. Thanks!

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Jul 6, 2018

@labarba and @tracykteal, I have a couple of questions about the review checklist.

  1. I assume the "statement of need" in Documentation and JOSE Paper are the same?
  2. How should the Documentation be presented in the submission? I am still going through the review but am pretty sure all of the required things will be there. Didn't know if you were also looking for a more concise doc (i.e. a README) that had all of this in it.

Hope to finish this up today, but if not it'll be week after next (on vacation next week).

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

Actually, we've been having a conversation in another review that tends to leaving the Statement of Need in the paper only. That is, not requiring it in the documentation.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

It looks like the submitted paper reproduces verbatim the contents of the repository's README. Hmm...

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

I find it unsatisfying that the paper and the README have the same contents. These documents have different goals., in my opinion

About the paper, the Author Guide says:

The goal is that someone reading the JOSE paper has enough information to decide if they'd be interested in adoping the learnig module or software.

The paper is a form of "scholarly advertisement" of the learning module. Readers want to know who the target audience, the design behind the learning module, how it could be used by another instructor or by an independent learner, how it has already been used in the classroom or elsewhere, and of course a description of the contents.

The README is the entry point to the repository, and it might be too lengthy to include all of the above. It should have installation instructions, list of dependencies, maybe statements about the license and how to contribute, and include an index of the lessons, with links.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

Looking now at the Introduction tutorial, I find a statement that succinctly says what this is:

this is a series of tutorials on improving the reproducibility of data analysis for those doing microbial ecology research. [...] the data set that we're going to be working with [...] is from the human microbiome research.

I wish this to-the-point statement appeared at the top of the paper!

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Jul 6, 2018

Thanks for the guidance! I was leaning in the same direction on the README. Will include in my review as well.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

@pschloss : I would like to see the bullet lists on slides 5 & 6 of the Introduction lesson, just like that, on the paper. It gives the reader an immediate picture of the contents of the module!

Also, in the Statement of Need (in the paper), you give general statements about reproducibility, about the field of microbiome research, and so on, and only in the final sentence refer to this tutorial. What we really want in this section is a statement about how this tutorial satisfies a particular need, and perhaps how it differs (or is similar) to other tutorials on the subject.

From the Author Guide:

Authors make the case for their submission's contribution in the paper, under the heading "Statement of Need."


@tracykteal : I'm stepping in here, in this review that you're editing, because with the first handful of papers in JOSE we are in fact defining our genre.

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Jul 6, 2018

Didn't get finished on review today. Will finish up week of July 16th.

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Jul 6, 2018

Thanks for the feedback and my apologies for missing the desired distinction between the paper and the README. Part of the "problem" as you all have identified in the thread here is that it's early days for the journal and I didn't have a lot to go off of for examples.

Should I go ahead and fix things up now as the reviews come in or wait until @jhollist (and @tracykteal?) has had a chance to go through it?

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jul 6, 2018

If you're going to re-structure the writing in the paper and README, it's probably a good idea to start now, so when the reviewer and managing editor come back to it, they can comment on your updated version.

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Jul 9, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jul 9, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jul 9, 2018

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Jul 9, 2018

FWIW, I've updated the README and paper.md files to reflect @labarba's comments

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Jul 25, 2018

All, done with my review (better late than never and apologies for taking so long). I have updated the checklist above and am providing a more detailed review in the target repo: riffomonas/reproducible_research#12

Any questions about the review or if you need me to place items here as opposed to the Riffomonas repo, let me know.

Thanks for asking me to do the review.

@jhollist jhollist referenced this issue Jul 25, 2018

Closed

jhollist JOSE Review Issue #12

10 of 10 tasks complete
@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Jul 26, 2018

Thanks so much @jhollist for doing this thorough review and the more detailed review notes. Just reading through your comments so I can come back and provide some recommendations.

JOSE reviews are discussions rather than a yes/no type of decision. Since we are just developing the pedagogy / instructional design section, that in particular is more of a discussion about suggestions. @jhollist made some nice suggestions, but @pschloss you don't necessarily have to implement all of the recommendations. It would be great to continue discussion on that issue that Jeff started.

One major point Jeff mentioned was that some of the sections had potentially a bit too much content. Given that the slide decks have accompanying video, I can see that it would be difficult to make that change however.

One point for the checklist is the Version. @pschloss it looks like there hasn't been a release yet. Before publication you can do a release and add the version.

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Jul 26, 2018

Was happy to do this review, @tracykteal and happy to be asked so early in the life of JOSE. I have some follow up on breaking the tutorials up but will put that over on riffomonas/reproducible_research#12

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Aug 15, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 15, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 15, 2018

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Aug 15, 2018

Thanks again for the review @jhollist and @tracykteal - I have addressed the comments over at riffomonas/reproducible_research as separate issues. I'm happy to follow up on any of the comments if you have further suggestions or comments.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 18, 2018

@jhollist and @tracykteal — Is this submission ready for acceptance?

@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Aug 20, 2018

Issues have been resolved. @pschloss will create version 1.0 and then we'll get it in for acceptance.

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Aug 20, 2018

I concur! The issues have been resolved. I look forward to recommending this material and think it will be a great first pub for JOSE!

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Aug 24, 2018

Everything should be all set - let me know what else you need me to do at this point.

Thanks!

@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Aug 27, 2018

You're all set! We'll get it published!

@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Aug 27, 2018

Thank you so much for your submission. We're very excited to have this in JOSE.

@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Aug 27, 2018

Sorry, one more thing. Can you create an archive (on Zenodo, figshare, or other) and post the archive DOI here. Then we link to that DOI for the publication.

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Aug 27, 2018

Wonderful - here you go:
https://doi.org/10.5281/zenodo.1404230

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 27, 2018

@whedon set 10.5281/zenodo.1404230 as archive

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 27, 2018

OK. 10.5281/zenodo.1404230 is the archive.

@labarba labarba added the accepted label Aug 27, 2018

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 27, 2018

@jhollist Not all review-checklist items are checked, but you commented above that you are satisfied with the revision. Can you complete the checklist?

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Aug 29, 2018

On vacation now with very flaky connection! Will finish this up on my return this Saturday.

@tracykteal

This comment has been minimized.

Copy link

tracykteal commented Aug 29, 2018

@jhollist had conveyed in the project repository that those items have been addressed, so I updated the checklist. @labarba this should be officially accepted and published. Thanks!

@labarba labarba closed this Aug 30, 2018

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 30, 2018

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://jose.theoj.org/papers/10.21105/jose.00013/status.svg)](https://doi.org/10.21105/jose.00013)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/jose.00013">
  <img src="https://jose.theoj.org/papers/10.21105/jose.00013/status.svg" alt="DOI badge" >
</a>

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Education is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 30, 2018

Yay! @pschloss —Your JOSE paper is now published and you can add the lovely green badge to your repo!

Tremendous thanks to @jhollist for reviewing and @tracykteal for handling the submission. Your volunteer efforts are helping us trail-blaze in new-wave publishing 🎉

@pschloss

This comment has been minimized.

Copy link
Collaborator

pschloss commented Aug 30, 2018

Wonderful - thanks everyone!

@jhollist

This comment has been minimized.

Copy link
Collaborator

jhollist commented Sep 3, 2018

Congratulations! And thanks to @tracykteal for taking care of the checklist for me.

Enjoyed reviewing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.