[REVIEW]: StateMint: A Set of Tools for Determining Symbolic Dynamic System Models Using Linear Graph Methods

[whedon]

Submitting author: @CameronDevine (Cameron Devine)
Repository: https://github.com/CameronDevine/StateMint
Version: 2.0.0
Editor: @moorepants
Reviewer: @chrisdembia, @gboeing
Archive: Pending

Status

Status badge code:

HTML: <a href="http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7"><img src="http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7/status.svg"></a>
Markdown: [![status](http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7/status.svg)](http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@chrisdembia & @gboeing, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://jose.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @moorepants know.

Review checklist for @chrisdembia

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (2.0.0)?
• Authorship: Has the submitting author (@CameronDevine) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?)
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @gboeing

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (2.0.0)?
• Authorship: Has the submitting author (@CameronDevine) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?)
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @chrisdembia, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[labarba]

👋 @chrisdembia, @gboeing — Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed!

[labarba]

@CameronDevine — please bear in mind the comments that @chrisdembia posted on the Pre-Review issue: #38 (comment)

@chrisdembia — feel free to re-post those comments here, if you wish.

[moorepants]

Yes, @chrisdembia please repost those comments here, so we keep things in one place.

[moorepants]

@CameronDevine, this question is not part of the review, just a curiosity. I teach a system dynamics course as well, but I teach the students to use Bond Graphs. I am not familiar with the book you recommend, but the language in your course notes certainly parallels or is the same as the course I teach. The only thing I'm not familiar with is the linear graph representation. Would you mind commenting briefly on any advantages/disadvantages this might have over Bond Graphs? Feel free to email me at moorepants@gmail.com if you don't want to clutter the review.

[chrisdembia]

Please address the following questions in the way you find most appropriate.

1. For what size problems do you expect the tools to function well? It might be good to document the problem size you expect. For example, it seems that the web interface might not properly display results for problems with hundreds of equations and variables.

2. What are some concrete intended use case? Will the software be used in a class at your university?

3. Are there other similar tools out there?

4. It seems your submission is missing some items listed in the Reviewer Guidelines:

a. "The online repository of the software or learning module needs to contain guidelines for potential contributors who may want to: submit changes, make improvements or report issues."
b. "Pedagogical soundness...The authors should briefly explain their design in the JOSE paper."
c. There should be clear guidelines for third-parties wishing to: Contribute to the software/module, Report issues or problems with the software/module, Seek support.

I have submitted a PR to your StateMint with additional comments and suggested changes.

Web interface

The web interface has a beautiful design, and is, overall, easy to follow.

1. What does the SAVE button do? I entered a name but it wasn't obvious what happened after that. I eventually discovered that the "OPEN EXISTING" button showed the systems I had saved, but this was not obvious and I am still confused about where these files actually reside. What happens if I close the browser window? Do these saved files disappear?
2. The DOWNLOAD button provides an RND file. What is this file format? I haven't heard of it before and I didn't find any description of it.
3. If I enter incorrect input and hit "= CALCULATE", I get the flashing hexagons forever. If I scroll down, I see "We're sorry. The function has encountered an error". Ideally, this message would be shown immediately so that the user does not keep waiting unnecessarily.
4. It's unclear if the formatting of the "Results" pane will scale well to many equations.
5. Where do you document the "Upload a photo" functionality of the web interface? It's unclear what this does or what the photo should be.
6. For each example, consider a brief description of what system is being modeled.

WEB interface tutorial
The figures in the tutorial are well-designed. The choice of example is good. However, the second half of the tutorial is confusing and is not sufficiently clear for someone with a traditional system dynamics background and is not familiar with linear graph theory.

1. The tutorial would benefit from making clear which parts of the tutorial describe what a user must do to prepare input versus describing what the StateMint software does. It seems that the user must manually derive elemental, continuity, and compatibility equations, and the software uses these equations to provide state and output equations. This could be made clearer.
2. There are numbers on the figures in the Normal Tree section (e.g., 1, 2, 3, 4), but you do not describe the meaning of these numbers.
3. The writing would benefit from the use of the active voice instead of the passive voice.

Python

It's atypical to put all the code in init.py. Please create more descriptive Python files.

Example.ipynb

1. The first paragraph seems incorrect (there are 3 methods for using StateMint, not only 2).
2. Out[5] shows $(t)$ on its own. Placing $t$ in parentheses like this seems odd; is there an error?

Mathematica

I installed a trial of Mathematica and ran the example without errors.

Please state the version of Mathematica with which you tested StateMint.

[moorepants]

@CameronDevine, you are welcome to start work on addressing @chrisdembia's review if you have not yet started.

@gboeing, we have not heard from you in over a week. Can you update us on the status of your review?

[gboeing]

@moorepants I'll be able to complete this by early next week.

[moorepants]

Sounds great!

[moorepants]

@gboeing Can you give us a status update?

[gboeing]

Overall, the software looks useful and the web interface is very attractively designed. The paper is pretty clear and well-written. Regarding revisions/issues: rather than duplicating them all here, I'll start by saying that @chrisdembia's review covered most of the points I wanted to make. In addition, I have opened two issues in the software's repo CameronDevine/StateMint#9 and CameronDevine/StateMint#10.

[moorepants]

Fantastic! @chrisdembia and @gboeing, thank you both for you careful and thoughtful reviews.

@CameronDevine The reviewers' comments and requests seem appropriate. The ball is now in your court to address the reviews. Please do so and keep us up-to-date here on your progress or with any questions/comments you have.