[REVIEW]: StateMint: A Set of Tools for Determining Symbolic Dynamic System Models Using Linear Graph Methods

[whedon]

Submitting author: @CameronDevine (Cameron Devine)
Repository: https://github.com/CameronDevine/StateMint
Version: 2.1.0
Editor: @moorepants
Reviewer: @chrisdembia, @gboeing
Archive: 10.5281/zenodo.2633330

Status

Status badge code:

HTML: <a href="http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7"><img src="http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7/status.svg"></a>
Markdown: [![status](http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7/status.svg)](http://jose.theoj.org/papers/7caec95b5db5c18d8a14cbc42fef7bb7)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@chrisdembia & @gboeing, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://jose.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @moorepants know.

Review checklist for @chrisdembia

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: 2.1.0
• Authorship: Has the submitting author (@CameronDevine) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?)
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @gboeing

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: 2.1.0
• Authorship: Has the submitting author (@CameronDevine) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?)
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @chrisdembia, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[labarba]

👋 @chrisdembia, @gboeing — Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed!

[labarba]

@CameronDevine — please bear in mind the comments that @chrisdembia posted on the Pre-Review issue: #38 (comment)

@chrisdembia — feel free to re-post those comments here, if you wish.

[moorepants]

Yes, @chrisdembia please repost those comments here, so we keep things in one place.

[moorepants]

@CameronDevine, this question is not part of the review, just a curiosity. I teach a system dynamics course as well, but I teach the students to use Bond Graphs. I am not familiar with the book you recommend, but the language in your course notes certainly parallels or is the same as the course I teach. The only thing I'm not familiar with is the linear graph representation. Would you mind commenting briefly on any advantages/disadvantages this might have over Bond Graphs? Feel free to email me at moorepants@gmail.com if you don't want to clutter the review.

[chrisdembia]

Please address the following questions in the way you find most appropriate.

1. For what size problems do you expect the tools to function well? It might be good to document the problem size you expect. For example, it seems that the web interface might not properly display results for problems with hundreds of equations and variables.

2. What are some concrete intended use case? Will the software be used in a class at your university?

3. Are there other similar tools out there?

4. It seems your submission is missing some items listed in the Reviewer Guidelines:

a. "The online repository of the software or learning module needs to contain guidelines for potential contributors who may want to: submit changes, make improvements or report issues."
b. "Pedagogical soundness...The authors should briefly explain their design in the JOSE paper."
c. There should be clear guidelines for third-parties wishing to: Contribute to the software/module, Report issues or problems with the software/module, Seek support.

I have submitted a PR to your StateMint with additional comments and suggested changes.

Web interface

The web interface has a beautiful design, and is, overall, easy to follow.

1. What does the SAVE button do? I entered a name but it wasn't obvious what happened after that. I eventually discovered that the "OPEN EXISTING" button showed the systems I had saved, but this was not obvious and I am still confused about where these files actually reside. What happens if I close the browser window? Do these saved files disappear?
2. The DOWNLOAD button provides an RND file. What is this file format? I haven't heard of it before and I didn't find any description of it.
3. If I enter incorrect input and hit "= CALCULATE", I get the flashing hexagons forever. If I scroll down, I see "We're sorry. The function has encountered an error". Ideally, this message would be shown immediately so that the user does not keep waiting unnecessarily.
4. It's unclear if the formatting of the "Results" pane will scale well to many equations.
5. Where do you document the "Upload a photo" functionality of the web interface? It's unclear what this does or what the photo should be.
6. For each example, consider a brief description of what system is being modeled.

WEB interface tutorial
The figures in the tutorial are well-designed. The choice of example is good. However, the second half of the tutorial is confusing and is not sufficiently clear for someone with a traditional system dynamics background and is not familiar with linear graph theory.

1. The tutorial would benefit from making clear which parts of the tutorial describe what a user must do to prepare input versus describing what the StateMint software does. It seems that the user must manually derive elemental, continuity, and compatibility equations, and the software uses these equations to provide state and output equations. This could be made clearer.
2. There are numbers on the figures in the Normal Tree section (e.g., 1, 2, 3, 4), but you do not describe the meaning of these numbers.
3. The writing would benefit from the use of the active voice instead of the passive voice.

Python

It's atypical to put all the code in init.py. Please create more descriptive Python files.

Example.ipynb

1. The first paragraph seems incorrect (there are 3 methods for using StateMint, not only 2).
2. Out[5] shows $(t)$ on its own. Placing $t$ in parentheses like this seems odd; is there an error?

Mathematica

I installed a trial of Mathematica and ran the example without errors.

Please state the version of Mathematica with which you tested StateMint.

[moorepants]

@CameronDevine, you are welcome to start work on addressing @chrisdembia's review if you have not yet started.

@gboeing, we have not heard from you in over a week. Can you update us on the status of your review?

[gboeing]

@moorepants I'll be able to complete this by early next week.

[moorepants]

Sounds great!

[moorepants]

@gboeing Can you give us a status update?

[gboeing]

Overall, the software looks useful and the web interface is very attractively designed. The paper is pretty clear and well-written. Regarding revisions/issues: rather than duplicating them all here, I'll start by saying that @chrisdembia's review covered most of the points I wanted to make. In addition, I have opened two issues in the software's repo CameronDevine/StateMint#9 and CameronDevine/StateMint#10.

[moorepants]

Fantastic! @chrisdembia and @gboeing, thank you both for you careful and thoughtful reviews.

@CameronDevine The reviewers' comments and requests seem appropriate. The ball is now in your court to address the reviews. Please do so and keep us up-to-date here on your progress or with any questions/comments you have.

[CameronDevine]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[CameronDevine]

I have addressed the issues in @chrisdembia's pull request, but am waiting to close it until I get confirmation I have sufficiently answered his questions. I have a few more items to address in @chrisdembia's comment on this issue. After that I will work on @gboeing's issues.

[CameronDevine]

@chrisdembia Below are my responses to your comments.

1. For what size problems do you expect the tools to function well? It might be good to document the problem size you expect. For example, it seems that the web interface might not properly display results for problems with hundreds of equations and variables.

The question about the web interface is addressed below. Doing a couple tests I found that solving for the state equation of N independent RC filters is O(N). However, solving N cascaded RC filters was O(e^N^2). So far I have not encountered a problem that was not able to be solved in a reasonable amount of time, except those which I fabricated to test the performance.

2. What are some concrete intended use case? Will the software be used in a class at your university?

Prof. Rico Picone at St. Martin's University and Prof. Joseph Garbini at The University of Washington are both planning to use this tool in their system dynamics and control systems classes. This tool will allow students to more easily complete homework problems because they will not have to perform as much algebra.

3. Are there other similar tools out there?

I believe there are similar tools available for use with Bond Graphs, but I do not know of any for use with Linear Graphs.

4. It seems your submission is missing some items listed in the Reviewer Guidelines:

a. "The online repository of the software or learning module needs to contain guidelines for potential contributors who may want to: submit changes, make improvements or report issues."

This has been added.

b. "Pedagogical soundness...The authors should briefly explain their design in the JOSE paper."

Would the addition to the paper of an explanation of the algorithm used to find the differential equations be enough to satisfy this requirement?

c. There should be clear guidelines for third-parties wishing to: Contribute to the software/module, Report issues or problems with the software/module, Seek support.

This has been added.

Web interface

1. What does the SAVE button do? I entered a name but it wasn't obvious what happened after that. I eventually discovered that the "OPEN EXISTING" button showed the systems I had saved, but this was not obvious and I am still confused about where these files actually reside. What happens if I close the browser window? Do these saved files disappear?

These files are saved in your browser as cookies. They are set to expire after 10 years, but they may be deleted before that happens. I understand how this could be confusing so I added a note about this in the save dialog.

2. The DOWNLOAD button provides an RND file. What is this file format? I haven't heard of it before and I didn't find any description of it.

This file is actually a JPEG image with the system data added as metadata. The RND name was created back when we were calling this tool by its initial name, State Model RnD. I would welcome any suggestions on what file extension to use, and where to document this.

3. If I enter incorrect input and hit "= CALCULATE", I get the flashing hexagons forever. If I scroll down, I see "We're sorry. The function has encountered an error". Ideally, this message would be shown immediately so that the user does not keep waiting unnecessarily.

This has been fixed.

4. It's unclear if the formatting of the "Results" pane will scale well to many equations.

A system of 15 independent RC filters can be solved using this link. This shows how the page can be scrolled horizontally to view the entire equation.

5. Where do you document the "Upload a photo" functionality of the web interface? It's unclear what this does or what the photo should be.

I noticed a bug related to this, which is now fixed. This image is displayed for reference while entering equations on the "Enter System Equations" pane.

6. For each example, consider a brief description of what system is being modeled.

This has been added.

WEB interface tutorial

1. The tutorial would benefit from making clear which parts of the tutorial describe what a user must do to prepare input versus describing what the StateMint software does. It seems that the user must manually derive elemental, continuity, and compatibility equations, and the software uses these equations to provide state and output equations. This could be made clearer.

The tutorial has been updated.

2. There are numbers on the figures in the Normal Tree section (e.g., 1, 2, 3, 4), but you do not describe the meaning of these numbers.

This has been added.

3. The writing would benefit from the use of the active voice instead of the passive voice.

I will be working on this over the next couple of days.

Python

It's atypical to put all the code in __init__.py. Please create more descriptive Python files.

The Python library has been moved to a more descriptive filename.

Example.ipynb

1. The first paragraph seems incorrect (there are 3 methods for using StateMint, not only 2).

You are correct. This has been changed.

2. Out[5] shows $(t)$ on its own. Placing $t$ in parentheses like this seems odd; is there an error?

That is really strange. Running the Jupyter notebook on my machine this issue is not present. I have contacted GitHub about this issue.

Mathematica

Please state the version of Mathematica with which you tested StateMint.

This has been added to the Mathematica folder readme.

[moorepants]

Would the addition to the paper of an explanation of the algorithm used to find the differential equations be enough to satisfy this requirement?

The pedagogical soundness is asking for you to justify why your software/lesson is good for teaching and learning. Backing this with some educational research evidence is ideal.

[CameronDevine]

Thank you for the clarification @moorepants.

[CameronDevine]

@moorepants and @chrisdembia, the following statement is already included in the paper.

Finally, the set of equations can be reduced to a system of first-order differential equations
in state and input variables and their time-derivatives, alone. It is in these last two steps,
especially the very last, that a student, manually reducing the set of equations, will often make some minor mistake. This is typically of a “book keeping” variety that, if it
teaches the student anything, it is not system dynamics. Instead, the student can be
easily discouraged and confused about where they have made their mistake. Fortunately,
the software tools presented here will automate the algebraic reduction. These will allow
students to focus on understanding the process of dynamic system modeling.

Prof. Rico Picone and Prof. Joseph Garbini both have many years of experience teaching system dynamics using linear graph methods. They both agree that this is the area where most students struggle. We could add more to the paper to explain this.

We also could distribute a survey to Prof. Rico Picone's current students, possibly including some of the questions below.

1. At which step in the state-space modeling algorithm do you most frequently make a mistake? [list steps of algorithm]
2. If there was an effective tool that automated step X [the thing StateMint does], would you find that it helps your learning process? If yes, why? If no, why not?
3. Consider the tool at statemint.stmartin.edu. Try an example. Do you think this is an effective tool for the automation of step X?

Which of these do you think would best show the pedagogical soundness of this tool?

[moorepants]

@chrisdembia @gboeing Can you both comment on @CameronDevine's last post? I'll defer to the reviewers on deciding how to handle the pedagogical soundness requirement. I can help provide facts and clarifications.

[gboeing]

In my opinion, a survey is not necessary but it would be nice. Certainly more explanation regarding your comment ("Prof. Rico Picone and Prof. Joseph Garbini both have many years of experience teaching system dynamics using linear graph methods. They both agree that this is the area where most students struggle. We could add more to the paper to explain this.") would be helpful to elucidate too.

[labarba]

Hi everybody!
We don't require formal evaluation of the learning module, but we expect the author to describe their instructional design and the pedagogy behind it.

I see both reviewers have some items of their review lists unchecked. Are you working on addressing some of the reviewer comments, @CameronDevine?

[moorepants]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

PDF failed to compile for issue #44 with the following error:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 16 0 16 0 0 49 0 --:--:-- --:--:-- --:--:-- 49
pandoc: paper.md: openBinaryFile: does not exist (No such file or directory)
Looks like we failed to compile the Crossref XML

[whedon]

OK DOIs

- 10.7717/peerj-cs.103 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[moorepants]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

PDF failed to compile for issue #44 with the following error:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 16 0 16 0 0 43 0 --:--:-- --:--:-- --:--:-- 43
sh: 0: getcwd() failed: No such file or directory
pandoc: 10.21105.jose.00044.pdf: openBinaryFile: does not exist (No such file or directory)
Looks like we failed to compile the PDF

[whedon]

OK DOIs

- 10.7717/peerj-cs.103 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[moorepants]

@labarba Am I missing something here? Not quite sure why this is giving an error.

[labarba]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[labarba]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.7717/peerj-cs.103 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/jose-papers#29

If the paper PDF and Crossref deposit XML look good in openjournals/jose-papers#29, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[moorepants]

@whedon accept deposit=true

[whedon]

I'm sorry @moorepants, I'm afraid I can't do that. That's something only editor-in-chiefs are allowed to do.

[labarba]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSE! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 openjournals/jose-papers#30
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/jose.00044
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[labarba]

Congratulations, @CameronDevine, your JOSE paper is now published!

Big thanks to the editor: @moorepants, and the reviewers: @chrisdembia, @gboeing — your contribution is so important 🙏

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://jose.theoj.org/papers/10.21105/jose.00044/status.svg)](https://doi.org/10.21105/jose.00044)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/jose.00044">
  <img src="https://jose.theoj.org/papers/10.21105/jose.00044/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://jose.theoj.org/papers/10.21105/jose.00044/status.svg
   :target: https://doi.org/10.21105/jose.00044

This is how it will look in your documentation:

We need your help!

Journal of Open Source Education is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://jose.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-jose

[labarba]

https://twitter.com/JOSE_TheOJ/status/1115582355140378624

[moorepants]

Congrats @CameronDevine and thank you @chrisdembia and @gboeing!

[CameronDevine]

Thanks @labarba, @moorepants, @chrisdembia, and @gboeing for all your comments. I believe the review process has made some substantial improvements to the software.

[CameronDevine]

@moorepants, You asked about the benefits of Linear Graphs early in the review process. I forgot about your question until I was reviewing some notes today.

The main benefit of Linear Graphs is the resulting diagram is almost identical to an electrical circuit diagram. Since students often learn to read electrical circuit diagrams in physics, we can leverage this existing knowledge.

If you have any other questions, feel free to email me, camdev@uw.edu.