[REVIEW]: MNE-BIDS: Organizing neurophysiological data into the BIDS format and facilitating their analysis

[whedon]

Submitting author: @sappelhoff (Stefan Appelhoff)
Repository: https://github.com/mne-tools/mne-bids
Version: 0.3.1
Editor: @arokem
Reviewer: @jdkent, @TomDonoghue
Archive: 10.5281/zenodo.3580273

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108"><img src="https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jdkent & @TomDonoghue, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arokem know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @jdkent

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@sappelhoff) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @TomDonoghue

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@sappelhoff) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @jdkent, @TomDonoghue it looks like you're currently assigned to review this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Attempting to check references...

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

OK DOIs

- 10.1038/sdata.2016.44 is OK
- 10.3389/fnins.2013.00267 is OK
- 10.1038/sdata.2018.110 is OK
- 10.1038/s41597-019-0104-8 is OK
- 10.1038/s41597-019-0105-7 is OK
- 10.3389/fnins.2018.00530 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👉 Check article proof 📄 👈

[TomDonoghue]

I'm going to put comments on the JOSS paper itself here:

Content Notes

Broadly, the paper looks good, though I feel like it presumes a fairly high level of familiarity / knowledge of the BIDS and MNE worlds, and so I'm not so sure that the paper is as self-sufficient as one might want to introduce the key ideas to a relatively naive reader. I've added a few notes on things that came to mind, and possible suggestions, for doing so.

Examples:

• In the 1st paragraph, it doesn't indicate how BIDS stores the actual data, and only really refers to metadata. Though full description of BIDS is in other papers, I feel like a quick note along the lines that BIDS is an specification of organizational structure (not a file format) in which data files are stored in existing BIDS-approved file formats might be useful.
• In the 2nd paragraph, what seems to be missing is a note / introduction to what MNE python is. It sort of skips past noting, for example, that MNE is a package for processing electrophysiology data, making the integration of standardized data structures and analysis pipelines a useful thing.
• Somewhat relatedly, the paper doesn't explicitly introduce that MNE-BIDS is a Python package (though this is implicit in the mention of Python objects). I would suggest perhaps more explicitly introducing (perhaps in relation to MNE), that MNE-BIDS is a package in the Python ecosystem.
• In the third paragraph, I feel a bit like it jumps into mechanics while somewhat missing description of the goal of doing so. Potentially there could be an introductory sentence to this paragraph, perhaps along the lines of "The key functionality of the MNE-BIDS package is to take a file structure containing an existing dataset of EEG, MEG or iEEG data, and convert this file structure and metadata organization into a BIDS compatible format."
• Also, in the third paragraph, what does it mean to be BIDS compatible? This concept is presumed, but if one isn't already quite familiar with the notion of BIDS, I'm not sure this makes sense (this also relates back to para 1 not fully introducing how BIDS is structured). I think this means if file formats are not already a BIDS approved format then this tool can convert them, and if so I feel like this could be explained slightly more.

Typo Notes

• 2nd paragraph, second to last line: after word 'preprocessing', space before comma

---

The content notes are meant as possible suggestions, though not reflecting any changes I'd consider strictly necessary.

[TomDonoghue]

Okay, I have done my main review. In my review, I looked through the code and repository, installed the software, and ran some of the examples. Everything looks good to me, and I think this is a really nice tool, and well developed and useful.

In terms of comments on the repository and code, I have made issues on the main repository, with some notes / ideas on general project thoughts (mne-tools/mne-bids#305) and some little comments on style and consistency in the documentation (mne-tools/mne-bids#304). Comments on the paper are above.

Nothing came up that I consider fundamentally important in a way that I think this project could not be approved for publication in JOSS based on the posted guidelines, so all the comments are offered basically as any possible and optional suggestions that came up as I worked through the project.

I left a couple things unchecked, that are point most related to the main comments I brought up in paper comments and issues on the repository itself - I'm happy to check them off once the authors check through and say if they plan to change anything, so we can check back through those things if so.

[arokem]

Thanks for the thorough review, @TomDonoghue!

@jdkent : have you had a chance to take a look?

[jdkent]

Yes, I had a chance to take a look.
MNE-BIDS is a useful package for converting MEG, EEG, and (experimentally) iEEG datasets into BIDS format and interfacing with those BIDS organized datasets.

I have made issues pertaining to the following sections:

Functionality

• commandline app not working as intended: [joss] command-line app mne_bids "command not found" mne-tools/mne-bids#296 (addressed)
• similarity/differences from pybids: [joss,refactor] similarity to pybids mne-tools/mne-bids#297 (addressed)

Documentation

• fix binder examples: [docs,joss] binder examples do not have mne_bids installed mne-tools/mne-bids#295 (closed)
• add code of conduct: [joss] add code of conduct mne-tools/mne-bids#315 (closed)
• add issue templates: [joss] add issue_template mne-tools/mne-bids#316 (closed)
• ensure correct python version: [joss] list supported python versions in pypi mne-tools/mne-bids#314 (closed)

Software Paper

• suggestings for writing style: [joss] Paper Review mne-tools/mne-bids#318 (closed)

MNE-BIDS looks like a great addition to JOSS! Great work to the authors! I left a few comments to improve the documentation and make interactions with the project better. I also offered some suggested edits to the paper to promote clarity/conciseness. As soon as the above issues are addressed I'll check off the final marks in the review above.

[arokem]

Fantastic! Thank you both for the detailed and valuable reviews.

@sappelhoff: please let me know if you have any questions and/or ping me here when you have had a chance to address the comments.

[sappelhoff]

Dear @jdkent and @TomDonoghue we believe that all your valuable feedback has now been addressed. We took extra care to cross link from each of your raised issues to the PRs solving them, so the progress should be thoroughly documented on GitHub.

Thanks again for your reviews - this has really improved our current status with the package!

[sappelhoff]

and thanks @arokem for organizing the review so far! :-)

[TomDonoghue]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[TomDonoghue]

Thank you @sappelhoff for all the detailed responses and clear linking between comments and PRs - I've been able to follow along clearly with all the updates to all the points, and everything looks great to me!

I think this is a great package in great shape! I have ticked of my remaining review points, and from my end I absolutely think this project / paper is good to be published in JOSS.

[jdkent]

Thanks @sappelhoff! All the issues I raised were followed up on and all of them have reached a satisfactory conclusion. I have happily checked off all remaining tick marks.

Another 👍 from me for publishing this project in JOSS.

[arokem]

Yep - looks great. I added a small suggestion to language in mne-tools/mne-bids#333.

Once that's merged or closed, could you please:

• Make a tagged release of your software, and list the version tag of the archived version here.
• Archive the reviewed software in Zenodo or a similar service
• Check the Zenodo deposit has the correct metadata, this includes the title (should match the paper title) and author list (make sure the list is correct and people who only made a small fix are not on it); you may also add the authors' ORCID.
• Please list the Zenodo DOI of the archived version here.

I can then move forward with accepting the submission.

[sappelhoff]

Hi @arokem we made a release and archived the software with Zenodo:

The metadata is aligned with the paper (including title and authors ... orcids are added as well).

Thanks also for your language suggestion, which has been incorporated 👍

[arokem]

@whedon set archive as 10.5281/zenodo.3580273

[arokem]

@whedon set version as 0.3.1

[arokem]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Set the software version at the top of the issue e.g.
@whedon set v1.0.1 as version

# Open the review issue
@whedon start review

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Remind an author or reviewer to return to a review after a
# certain period of time (supported units days and weeks)
@whedon remind @reviewer in 2 weeks

# Ask Whedon to accept the paper and deposit with Crossref
@whedon accept

# Ask Whedon to check the references for missing DOIs
@whedon check references

[arokem]

@whedon set 10.5281/zenodo.3580273 as archive

[whedon]

OK. 10.5281/zenodo.3580273 is the archive.

[arokem]

@whedon set 0.3.1 as version

[whedon]

OK. 0.3.1 is the version.

[arokem]

DSLs are the worst

[arokem]

@openjournals/joss-eics : I believe this paper is now ready for your approval.

[arfon]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.1038/sdata.2016.44 is OK
- 10.3389/fnins.2013.00267 is OK
- 10.1038/sdata.2018.110 is OK
- 10.1038/s41597-019-0104-8 is OK
- 10.1038/s41597-019-0105-7 is OK
- 10.3389/fnins.2018.00530 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#1180

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1180, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[arfon]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.01896 joss-papers#1181
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01896
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[arfon]

@jdkent, @TomDonoghue - many thanks for your reviews here and to @arokem for editing this submission ✨

@sappelhoff - your paper is now accepted into JOSS ⚡🚀💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01896/status.svg)](https://doi.org/10.21105/joss.01896)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01896">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01896/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01896/status.svg
   :target: https://doi.org/10.21105/joss.01896

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[sappelhoff]

Thanks a lot @arfon @arokem and @TomDonoghue @jdkent for the reviews.

This was a great process :-)