[REVIEW]: Open Source Optical Coherence Tomography Software

[whedon]

Submitting author: @spectralcode (Miroslav Zabic)
Repository: https://github.com/spectralcode/OCTproZ
Version: v1.0.0
Editor: @arfon
Reviewers: @jdavidli, @brandondube
Archive: 10.5281/zenodo.4148992

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/539ea5d7842ff0a7607a4a405ea69730"><img src="https://joss.theoj.org/papers/539ea5d7842ff0a7607a4a405ea69730/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/539ea5d7842ff0a7607a4a405ea69730/status.svg)](https://joss.theoj.org/papers/539ea5d7842ff0a7607a4a405ea69730)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@phtomlins & @jdavidli, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arfon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @brandondube

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@spectralcode) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @jdavidli

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@spectralcode) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @phtomlins, @jdavidli it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary:

OK DOIs

- 10.1364/boe.3.003067 is OK
- 10.1364/oe.18.011772 is OK
- 10.1117/1.3548153 is OK
- 10.1117/1.JBO.17.10.100502 is OK
- 10.1109/fccm.2011.27 is OK
- 10.1117/1.JBO.18.2.026002 is OK
- 10.1364/OE.18.024395 is OK
- 10.1364/BOE.5.002963 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👉 Check article proof 📄 👈

[arfon]

@phtomlins, @jdavidli - please carry out your review in this issue by updating the checklist above and giving feedback in this issue. The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

Any questions/concerns please let me know.

[arfon]

@whedon add @brandondube as reviewer

[whedon]

OK, @brandondube is now a reviewer

[brandondube]

👋 @spectralcode good news at the start - the 1.2 release opens on my desktop with no issue. I did not easily find an example file in the repository or referenced in the documentation. Is there one available to demo the software with?

[spectralcode]

Hi @brandondube thank you very much for reviewing!
That's great that the software opens without issues. To get a link to a test data set, please have a look at the "Download and Installation" section in the readme.md. The same link can also be found in the quick start guide of the user manual.

[brandondube]

Thanks for pointing out the demo data, I missed it the first time. Here are my comments:

I want to discuss performance at the outset, since it is a major focus of this work. My computer is as follows:

• i7-9700k
• 32GB of RAM
• GTX 2080
• W10 pro

If I disable all processing and display, then the performance yield is:

• 4 volumes per second
• 64 buffers per second
• 1020 B-scans per second
• 5.2e7 A-scans per second
• 26MB buffer
• 1.7GB/s data throughput

A 2080 is about equivalent to a 1080 Ti, if I remember correctly. So this seems in-line with your lab system, modulo the OS which can make a large difference. All my performance numbers are in-line, except the number of volumes per second. Why? I have no graphical displays up and no processing done.

W.r.t the paper and the code..

Paper

Last line of section 5 should be "being processed."

In Python JOSS submissions, it is customary to provide citations for numpy, etc. You do not have one for Qt or Cuda. if there are equivalents, please cite them. Perhaps if there are no academic citations, citing a product page or similar.

Do you have permission to use the EU and regional development fund logos?

I did not see a comparison to other competing technologies or justification for C++/Qt as the technologies chosen. This is not required, but is usually nice to see. For example, implementation with python and CuPy probably would have a much lower barrier to entry for scientists.

Software / Docs

✔️ sample extensions provided

The manual feels a bit light. Your paper has a major focus on user extensibility and open source contributions from others. I feel that not having a developer guide or similar is a strong impediment to success in that area.

It is unclear to me when taking a glance at the code how the internal dependencies work. E.g., processing.h/cpp can see much of the application and (I think) nearly all of the data. I think this goes along with a need for a developer guide.

The example dataset could be more boldly pointed out (I missed it the first time).

I also did not find a test suite.

The remaining check boxes on my review list are related to these points - I will wait for your feedback.

Cheers

[spectralcode]

@brandondube thank you very much for your feedback!
Performance:
Thank you for testing the performance and providing this information!
Have you used the same data dimensions and the same batch size as stated in the paper? (1024 samples per raw A-scan, 512 A-scans per B-scan, 256 B-scans per volume and batch size: 256 B-scans per buffer)

Paper:

Last line of section 5 should be "being processed."

→ Fixed!

You do not have one for Qt or Cuda. [...] please cite them.

→ Done! I couldn't find any other JOSS paper citing Qt or CUDA so I am not sure if the way i cited it is the common way to do it. Please let me know if you would have done it differently.

Do you have permission to use the EU and regional development fund logos?

→ Thank you very much for pointing this out. I am even obliged to use the logo. In addition, the logo must be at least as large as the largest other logo. So maybe I have to enlarge it as the JOSS logo is quite big. I will talk with the administration regarding the logo size and maybe there is a way out not to use the logos at all as it seems quite unusual to me.

For example, implementation with python and CuPy probably would have a much lower barrier to entry for scientists.

→ Yes, I think you are right about that! There is an idea for the long run to extend the plugin system such that it enables scripting with Python, but no evaluations have been made yet in this regard. At the start of the project it did not cross my mind to use Python for a full blown desktop application that has specific requirements on memory management, processing speed and should be able to control all kinds of hardware. I knew all of these requirements could be met by C++ and CUDA, so I went for it.

Software / Docs:

I feel that not having a developer guide or similar is a strong impediment to success in that area.

→ This is a great comment. Thank you for pointing this out! I feel the same way, so here is the developer guide.

The example dataset could be more boldly pointed out (I missed it the first time).

→ Done! It now has its own section.

I also did not find a test suite.

→ There is none. I hoped that providing test data and a detailed step by step guide how to use it would be sufficient. See the JOSS review criteria for Tests.

[spectralcode]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[spectralcode]

Just a small update: I received feedback regarding the logo and it should stay the way it is.

[brandondube]

I do not remember the precise settings I used for the sample data configuration 😅 - been too many days. I used the settings listed in the documentation where it is referenced. If those are different to the settings in the paper, then the perf may differ.

No sweat on the logos - if you are required to use them by your funding agencies, that is quite a strong "permission to use." We would not want to be in a situation where JOSS receives a C&D or similar on your paper over a logo; glad it's sorted.

The developer guide looks solid.

Regarding tests - I had in mind something like a GUI automation script that can verify the software correctly performs some action. Your software meets the standard for JOSS, but there is nothing wrong with exceeding the standard :)

LGTM

-Brandon

[arfon]

@whedon add @jrasakanthan as reviewer

[arfon]

@spectralcode - thanks for the updates.

@jdavidli - with these latest changes, are you ready to check off the remaining items in your reviewer checklist above?

[jdavidli]

@spectralcode thanks for making those changes!
@arfon yes this concludes my review.

[arfon]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[arfon]

@spectralcode - could you please merge this PR which implements a few small fixes to your paper: spectralcode/OCTproZ#5

After this could you make a new release of this software that includes the changes that have resulted from this review. Then, please make an archive of the software in Zenodo/figshare/other service and update this thread with the DOI of the archive? For the Zenodo/figshare archive, please make sure that:

• The title of the archive is the same as the JOSS paper title
• That the authors of the archive are the same as the JOSS paper authors

I can then move forward with accepting the submission.

[spectralcode]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[spectralcode]

@arfon Thank you for the fixes. I merged your pull request!
Here is the DOI of the Zenodo archive: https://doi.org/10.5281/zenodo.4148992

The version number of the most recent release is v1.2.1

Please note: I have made some minor changes to the paper. One change that is noticable is figure 2, which I redesigned to make the text in the figure easier to read. I hope this is fine for everybody, please let me know if this is not the case.

[arfon]

@whedon set 10.5281/zenodo.4148992 as archive

[whedon]

OK. 10.5281/zenodo.4148992 is the archive.

[arfon]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1364/boe.3.003067 is OK
- 10.1364/oe.18.011772 is OK
- 10.1117/1.3548153 is OK
- 10.1117/1.JBO.17.10.100502 is OK
- 10.1109/fccm.2011.27 is OK
- 10.1117/1.JBO.18.2.026002 is OK
- 10.1364/OE.18.024395 is OK
- 10.1364/BOE.5.002963 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1876

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1876, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[arfon]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02580 joss-papers#1877
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02580
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[arfon]

@jdavidli, @brandondube - many thanks for your reviews here ✨

@spectralcode - your paper is now accepted into JOSS ⚡🚀💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02580/status.svg)](https://doi.org/10.21105/joss.02580)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02580">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02580/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02580/status.svg
   :target: https://doi.org/10.21105/joss.02580

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[spectralcode]

@brandondube , @jdavidli many thanks for your reviews and to @kthyng and @arfon for editing this submission!