Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[REVIEW]: OrNet - a Python Toolkit to Model the Diffuse Structure of Organelles as Social Networks #1983
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @akeshavan know.
Review checklist for @serine
Conflict of interest
Code of Conduct
Review checklist for @vc1492a
Conflict of interest
Code of Conduct
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews
To fix this do the following two things:
For a list of things I can do to help you, just type:
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
Overview: This was an interesting piece of software to review as its use case it outside of my typical area, but makes use of many Python software packages I use in my work and side projects. I appreciate the authors commitment to providing docstring throughout the code base which allowed me to dive in more deeply and provide some suggestions for further improvement.
Below are my comments on particular sections, of which most (if not all) are relatively minor. When the authors strictly meet the requirements laid out in the review checklist, I have marked these items as complete. For those where the requirement is not strictly met, I have left the checklist entry empty. Either way, I have provided some comments below with actions that should be taken (in the case of unchecked entries) and actions that could be taken to improve the work further but that aren't strictly required for submission to JOSS (in the case of checked entries).
Repository URL: While the software is available at the repository URL, the URL listed in this issue as part of the review textually does not match the destination (https://github.com/quinngroup/ornet-JOSS vs https://github.com/quinngroup/ornet). This should be noted and corrected prior to publication to JOSS by the editor.
Installation and Installation Instructions: While the installation proceeds without issue according to the instructions provided in
pip install -r requirements.txt
It is recommended that the authors include a
In checking the agreement between the dependencies in
Lastly, it would be beneficial for the authors to consider using semantic versioning if not doing so already, consider releasing their software on PyPi for easy installation using a command like
Functionality, Functionality Documentation, and Example Usage: Running the samples took quite a bit of time on the machine in which I was using to review the software (greater than 11 hours, almost all in video processing), which is energy-efficient and not designed for large workloads (e.g. 1.1 Ghz Intel Core m3). It may be beneficial for the authors to note the expected duration of running OrNet on the sample videos and masks or provide suggested hardware requirements.
The authors did include sample (example) files for understanding how to use the functionality of OrNet and included the necessary commands needed to use the software (e.g. functionality documentation). However, it would be good to explicitly note in the documentation (e.g. in
Automated Tests: After ensuring
If the authors are looking to have others contribute to the project more easily in the future, they may want to consider setting up a continuous integration pipeline that will automatically run tests and report the test results (e.g. using Travis CI) and unit test coverage (e.g. using Coveralls) with any pushes to
State of the Field: The paper does a great job referencing how the software has been built on top of already-existing and well-utilized Python libraries such as scikit-learn or matplotlib. In addition, the paper makes reference to a C software called ITK-SNAP to generate masks for analysis by OrNet. However, the software paper does not explicitly state whether OrNet represents a new capability in the field or whether it ought to be compared with any existing software packages. The authors may want to consider being more explicit in the wording of the paper in this area - if its a new capability, explicitly make the claim. Otherwise, compare OrNet to any existing software packages that allow researchers to perform a similar task. This perhaps could be as simple as stating that previously researchers were required to build their own scripts for this task manually and on their own.
References: The two references provided are appropriately cited and relevant to this work and the software paper that is included in this submission, and are in the proper citation syntax. However, I feel that additional references would make the supplied software paper much stronger, especially for readers / reviewers that are not in the authors' field of study. For example, the first paragraph states "Fluorescent tagging has shed light on mitochondria's spatial orientation...", but does not provide a reference that indicates the first occurrence of this or any seminal works. Additionally, while it may seem odd at first, the Python language was originally presented at a conference and can thus be referenced (I have supplied this reference in
Community Guidelines: The authors have a section dedicated in the readme for users of the software to request changes to the software, report any issues during use, and ask any questions (to be performed by opening issues within the Github repository). However, the section does not outline specifically how users and/or developers may contribute to the software for future use. It may benefit the authors / developers to include more specific contribution instructions, such as asking users to first fork the repository and then create a new branch with a name that describes the issue or new feature before submitting a pull request. This may result in some additional organization and encourage others to contribute to the software.
Other: Running the samples takes a very long time on hardware that isn't optimized for video processing, like the hardware I used in my review (just over 11 hours to normalize the input video). There are perhaps a few areas in the code base that could take advantage of multi-processing or parallelism to speed up computation, which would be helpful in a practical sense, and I highly suggest introducing progress bars or other console output that will help inform the user of how much progress has been made when using OrNet. I've performed some code formatting and cleanup that is consistent with PEP8 formatting as part of the aforementioned pull request. Other small tweaks which could be made: the gamma parameter is not included in the docstring for the function
I very much look forward to seeing this software mature, and it was fun to review!
@vc1492a Thank you so much for the detailed review of our software! We really appreciate the effort you made into making that pull request. As of now, that request has been merged, and we are working on incorporating all of the other changes you pointed out. Also, we have improved the median normalization code so it no longer takes hours but minutes, so if you ever get free time and want to play around with it again it should be a much better experience. We will definitely update everyone once all of those changes have been made.
In the current form software doesn't install. It is missing:
Work in progress as I haven't been able to get results yet..
There doesn't appears to be any claims regarding software performance, so this one is ok
From reading a README.md in the repo it is hard to get an idea what the tool does. In general I'm still not clear if the tool is for some pre-processing or it is meant to yield some biologically meaningful / interpretable results.
One thing that concerns me is a statement in the paper.md that this is a "prototype" software and there some light hints that a new software might come to exists? I'll try to provide more comments in the next session Sortware paper but one of the main purposes of this review is for you to be able to say that you have a functioning software/tool that people in the world can use and we (JOSS reviews) have tested it out and thumbed up. So if you are planing to abundant this tool in the near future, the effort seems wasteful.
for linux (ubuntu at least) this additional package is required
It is probably good idea list all python packages under "dependencies" section
There is an example folder, however little description given about the video files. It would be nice if there was more explanation about the video files and what the tool will do to the them in terms of processing and the expected results. In addition author seems to assume that the user will know how to do the masking and generate
I'm confused about stdout/stderr from the tests. I think those
It would be nice to add detection of the existing output folder or simply warn users that results will be overwritten
There probably need some mention of the resources required for tool to run. i.e 1 cpu or several. Also RAM footprint. I've noticed that on mdivi sample, ornet was taking just over 5Gb. I'm guessing RAM usage will go up with the file (video) size. What is your "typical" amount of raw data and processed? I guess folks in the field should know, however I still feel that this information will be useful. Also mention runtime.. seems like it is taking a long time.
Does the video format matters? you should probably state file format dependency either way
This is being rather picky, but those spaces in file names are rather annoying for command line tool? Your sample videos have brackets and spaces :).
When you are working with image data, do you get a "movie" file e.g avi, from the instrument or you get bunch of images that you then convert into a "movie" file, which you then use with your tool?
I think python version is too stick? The tool seems to work on python version 3.5 and 3.6.7.. It doesn't really matter for the review process, but you sort of limiting amount of people that are potentially could use your tool
Do you need this print statement src/extract_cells.py:52: print(number_of_segments) ?
It gives a number on a console without much context.. If this is a meaningful message you should probably prefix a short message i.e
or something similar, or just comment out for debugging purposes later
I think from usability point of view it would be great to change
Hi @serine. Thank you for the detailed review as well! Many of the issues you have pointed out are actively being fixed as we speak, and we will update everyone once the last fixes are applied. As of now, I believe the last things necessary will be clearing up ambiguity in our documentation and paper, and improving our test suite. A major point needing clarification is that this software will not be abandoned in the near future. We apologize for the confusion caused by the prototype statement. It was intended to indicate that we already have a usable package and we plan on constantly improving many areas of it, but we see how it was misleading.
One interesting issue you discovered was a missing python3-tk package. After receiving feedback from @vc1492a, we've setup continuous integration using travis ci and has been able to successfully test our code against an Ubuntu environment for different python versions using our install instructions. Whenever you find free time, could you please run our latest install instructions without the python3-tk package and let us know if the problem is now resolved.
Once again, we thank everyone for their feedback, because this review feedback has been amazing in terms of improving our work. We are working hard on pushing these changes quickly, so the review process can continue smoothly.
@Marcdh3 yep, so
using python 3.6.7 fixed that issue.
However you should provide more explicit installation steps. This is what I had to do
Yep as you've mentioned some work on the docs and you should be all good.