[REVIEW]: DiffeRT2d: A Differentiable Ray Tracing Python Framework for Radio Propagation

[editorialbot]

Submitting author: @jeertmans (Jérome Eertmans)
Repository: https://github.com/jeertmans/DiffeRT2d
Branch with paper.md (empty if default branch):
Version: v0.3.2
Editor: @danielskatz
Reviewers: @idoby, @roth-jakob
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0"><img src="https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0/status.svg)](https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@idoby & @roth-jakob, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @idoby

📝 Checklist for @roth-jakob

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.07 s (1152.6 files/s, 182987.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          31           1346           1631           4323
Markdown                        18            443              0           1197
Jupyter Notebook                 1              0           1341            572
YAML                             8             43             10            256
TeX                              2             18              0            185
TOML                             1             23              5            170
JSON                             1              0              0            154
SVG                              1              0              0             54
DOS Batch                        1              8              1             26
reStructuredText                 9             11             33             20
make                             1              6              7             14
CSS                              1              1              0              9
-------------------------------------------------------------------------------
SUM:                            75           1899           3028           6980
-------------------------------------------------------------------------------

Commit count by author:

   277	Jérome Eertmans
    36	pre-commit-ci[bot]
     9	dependabot[bot]
     2	ImgBotApp

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.23919/EuCAP60739.2024.10501570 is OK
- 10.1109/TAP.2014.2323961 is OK
- 10.1109/ACCESS.2015.2453991 is OK
- 10.23919/EuCAP57121.2023.10132934 is OK
- 10.1371/journal.pone.0260060 is OK

MISSING DOIs

- No DOI given, and none found for title: Adam: A Method for Stochastic Optimization
- No DOI given, and none found for title: The DeepMind JAX Ecosystem
- No DOI given, and none found for title: JAX: composable transformations of Python+NumPy pr...
- No DOI given, and none found for title: jaxtyping: type annotations and runtime checking f...
- No DOI given, and none found for title: Equinox: neural networks in JAX via callable PyTre...
- No DOI given, and none found for title: Advanced Radio Channel Simulator
- 10.1109/gcwkshps58843.2023.10465179 may be a valid DOI for title: Sionna RT: Differentiable Ray Tracing for Radio Pr...
- No DOI given, and none found for title: TensorFlow: Large-Scale Machine Learning on Hetero...

INVALID DOIs

- None

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1243

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[danielskatz]

@idoby and @roth-jakob - Thanks for agreeing to review this submission.
This is the review thread for the paper. All of our communications will happen here from now on.

As you can see above, you each should use the command @editorialbot generate my checklist to create your review checklist. @editorialbot commands need to be the first thing in a new comment.

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

As you both know, the JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#6915 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if either of you require some more time. We can also use editorialbot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

[idoby]

Review checklist for @idoby

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/jeertmans/DiffeRT2d?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@jeertmans) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[roth-jakob]

Review checklist for @roth-jakob

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/jeertmans/DiffeRT2d?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@jeertmans) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[idoby]

Hey @jeertmans, thanks for submitting your package to JOSS! 📦
It seems very cool!

A few comments about the paper:

1. Lines 50-51: "such as the number of multipath components and angle of arrival."
2. Line 68: Please spell out the acronym MPT for people who don't know what it means.
3. Line 69: "...we can easily accomodate these surfaces." Referring to objects here can be confusing since the next sentence refers to software objects rather than physical simulation objects.
4. Figure 2: "it is evident that..." It isn't evident. Please assume the reader is not well versed in the specific subdiscipline, explain in more detail what is being shown in the figure. Point out what the RIS is and where it is depicted in the image. I also advise to plot a specific path to illustrate clearly the 45-degree angle you're referring to. re: the code, I suggest to provide the code sample as a complete, runnable script with the package and to link externally to the relevant Python file. As it stands, the code is (admittedly) incomplete and not runnable because it omits the actual drawing of the figure. While understandable, this takes away substantial value from a potential user. Consider linking to a stable repository (e.g., Zenodo) as well as GitHub once the paper is accepted and your software archive is created. Besides this minor detail, I think the code illustrates the basic usage of the package well. As a final point, if you do decide to link to the complete example as an external file, consider pointing this out in the text and not as a footnote. If you do not want to include a link, it's enough to state that the example is available as a sample in the package repository.
5. Figure 3: Likewise, I suggest providing full runnable code for this example as well, though there is indeed no need to include the code in the paper itself.
6. "Machine Learning" section: It is unclear what "this COST meeting" means and why it is relevant to the paper. I suggest rewriting this part and including a representative figure from the linked page. I also strongly suggest linking to a stable URL and not your personal website.

Other than these minor comments, good work! I will take a look at the software itself soon.

[roth-jakob]

Hi, thanks for your effort in writing the paper! Just a follow-up on the 4. above comment. I fully agree with that. Giving a little bit more explanation for readers not familiar with the subdiscipline could make the paper much more accessible.

[jeertmans]

Hey @jeertmans, thanks for submitting your package to JOSS! 📦 It seems very cool!

Hello @idoby, thanks for your first review comments, which I try to address in jeertmans/DiffeRT2d#69.

Let me add a few comments here:

A few comments about the paper:

1. Lines 50-51: "such as the number of multipath components and angle of arrival."
2. Line 68: Please spell out the acronym MPT for people who don't know what it means.
3. Line 69: "...we can easily accomodate these surfaces." Referring to objects here can be confusing since the next sentence refers to software objects rather than physical simulation objects.

Thanks for catching! I have rephrased those sentences.

4. Figure 2: "it is evident that..." It isn't evident. Please assume the reader is not well versed in the specific subdiscipline, explain in more detail what is being shown in the figure. Point out what the RIS is and where it is depicted in the image. I also advise to plot a specific path to illustrate clearly the 45-degree angle you're referring to. re: the code, I suggest to provide the code sample as a complete, runnable script with the package and to link externally to the relevant Python file. As it stands, the code is (admittedly) incomplete and not runnable because it omits the actual drawing of the figure. While understandable, this takes away substantial value from a potential user. Consider linking to a stable repository (e.g., Zenodo) as well as GitHub once the paper is accepted and your software archive is created. Besides this minor detail, I think the code illustrates the basic usage of the package well. As a final point, if you do decide to link to the complete example as an external file, consider pointing this out in the text and not as a footnote. If you do not want to include a link, it's enough to state that the example is available as a sample in the package repository.

I think I have addressed this issue by (1) putting the complete code and (2) rephrasing the caption.

5. Figure 3: Likewise, I suggest providing full runnable code for this example as well, though there is indeed no need to include the code in the paper itself.

Done.

6. "Machine Learning" section: It is unclear what "this COST meeting" means and why it is relevant to the paper. I suggest rewriting this part and including a representative figure from the linked page. I also strongly suggest linking to a stable URL and not your personal website.

I hope this is now clearer (I forgot to update that part before submitting, this is my bad).

I always hesitate between putting links from Zenodo, GitHub, or else. The motivation to use a link to my website was that, I am more likely to keep my eertmans.be up for a long time, than keeping the absolute path to the rendered notebook the same. On my website, I just set up a permalink redirect that I can update anytime I want. The permalink is also checked by tests, so hopefully I will always point to a valid url.
Zenodo (or web archives) can be nice to have a permalink, but then you cannot update the code (e.g., to fix typos etc.).

Other than these minor comments, good work! I will take a look at the software itself soon.

Looking forward to the software review! :D

[jeertmans]

Hi, thanks for your effort in writing the paper! Just a follow-up on the 4. above comment. I fully agree with that. Giving a little bit more explanation for readers not familiar with the subdiscipline could make the paper much more accessible.

Hello! I have (I think) addressed this comment in jeertmans/DiffeRT2d#69. Can you review it and tell me if it needs more details?

[idoby]

The issue with the links is that Zenodo etc are meant to be truly permanent in the sense that they are meant to be archived forever. This is why JOSS requests that upon acceptance, you archive your software with one of these services for posterity. The JOSS guidelines do not seem to address the issue of including external links in the paper. @danielskatz what do you think?

The same applies to the link you added to https://dial.uclouvain.be/pr/boreal/object/boreal:288635
I'm not sure what this is exactly. Is it a publication? If so, it should be cited properly. Is it a preprint? If so, it should be placed on a preprint server and cited properly as a preprint paper.

Other than these points, the paper seems good and my comments have been addressed as far as I'm concerned.
Will take a look at the software soon.

[danielskatz]

Personally, I'm ok with content on website being linked to, but I would prefer that an archive.org copy of it was linked, as this is more "permanent" if possible. e.g., rather than referencing https://www.ncsa.illinois.edu, one could reference https://web.archive.org/web/20240620232147/https://www.ncsa.illinois.edu/ for the version as of a few days ago.

For the link above, using the hdl (http://hdl.handle.net/2078/288635) would be better than using the direct link, even though it points to the same thing. If https://dial.uclouvain.be is an institutional repository, which I think it is, then this is enough. If not, then doing what @idoby suggests would be good, putting it on a "permanent" archive like Zenodo, arXiv, ...

[jeertmans]

Indeed, https://dial.uclouvain.be/ is an institutional repository, and should be a permalink that will always be valid. This is a preliminary work, so not really published, but I transformed the URL-reference to a citation that points to the "preprint", available on our inst. website.

For the other internet links, I will convert them later (today or this week) to web archives :-)

[jeertmans]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[roth-jakob]

Thanks for fixing the links and adding the full code for the figure in jeertmans/DiffeRT2d#69. I was able to reproduce the figure. I will have a look at the software in the next few days.

[idoby]

Looks like the new paper proof did not pick up the changes

[jeertmans]

Looks like the new paper proof did not pick up the changes

Yes, I forgot to first merge the PR ^^'. I'll fix that soon!

[jeertmans]

@editorialbot commands

[editorialbot]

Hello @jeertmans, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Run checks and provide information on the repository and the paper file
@editorialbot check repository

# Check the references of the paper for missing DOIs
@editorialbot check references

# Generates the pdf paper
@editorialbot generate pdf

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Get a link to the complete list of reviewers
@editorialbot list reviewers

[jeertmans]

The software itself looks good to me, pretty cool work. I'm not a huge fan of a fully object-oriented style with abstract classes but that's really a matter of preference/design choices. I have no comments on the software other than good job, it seems well crafted and easy to read. 👍

Thanks!

I did find some typos in the documentation, I will open a PR to fix them soon.

I have reviewed and merge all PRs, thanks for your help!

Other than that, l think the doc intro page deserves a little more exposition wrt what this tool is and how it's different than other similar tools (what JOSS would call a "statement of need"). I suggest to take some prose from the paper and graduate the footnote on the intro page into normal text. However, I leave this up to the author and will not block the review over this.

Good remark, I will wait for the second reviewer (and the editor?) to give their feedback first, then I think I might improve a bit the documentation according to your suggestions.

Other than that, I will re-generate a paper with the latest changes.

[jeertmans]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jeertmans]

My TODO list:

• Improve the main index.md documentation page, as suggested by @idoby
• Create a new Zenodo release to point to the latest version (and update in README.md)
• Change URLs in paper to webarchive links (or Zenodo)
• Change package version in JOSS paper (to include latest changes)

I put that here to keep track of important changes to make, and I won't have access to a computer from July 2nd to July 16th, so this doesn't get lost :-)

[roth-jakob]

@jeertmans, nice software package! I had a closer look at it. I have only one minor suggestion, along the lines of @idoby's comment. I personally like it if there are a few sentences at the top of the README explaining what the package is doing and for whom it is targeted. All this information is in the paper, and you could consider summarizing it at the top of the README.

[danielskatz]

👋 @idoby & @roth-jakob - You both have checked off all your items - but I also see comments about things to do from you and @jeertmans. So I'm a little uncertain about if you think this is done, or if we need to wait for @jeertmans to do some more things.

[idoby]

@danielskatz Jerome still wants to improve the documentation page, but as far as I'm concerned the review is done and is up to @jeertmans

[danielskatz]

👋 @jeertmans

My TODO list:

• Improve the main index.md documentation page, as suggested by @idoby
• Create a new Zenodo release to point to the latest version (and update in README.md)
• Change URLs in paper to webarchive links (or Zenodo)
• Change package version in JOSS paper (to include latest changes)

I put that here to keep track of important changes to make, and I won't have access to a computer from July 2nd to July 16th, so this doesn't get lost :-)

At the end of the process, once the review is complete, I will proofread the paper, and perhaps suggest changes, then ask you to make an archival deposit (e.g., in Zenodo) and tell me that DOI and the version associated with it, so you might want to wait on the 2nd and 4th items above until then.

[roth-jakob]

Yes, for me, it's the same. I proposed to @jeertmans to expand the readme a bit, but otherwise, I'm done with the review.

[danielskatz]

@jeertmans - Also, I think we can complete this before you go away on July 2nd if you want, since we are quite close.

[jeertmans]

Sure @jeertmans, I will do that in the next few hours :-)

[jeertmans]

@danielskatz done, the latest version is v0.3.3 (https://github.com/jeertmans/DiffeRT2d/releases/tag/v0.3.3). I'll update the Zenodo after you have proofread the paper, as you suggested :-)

[danielskatz]

My institution's summer picnic is starting soon, but I'll do this after it and some meetings, in the 4-8 hours...

[danielskatz]

@editorialbot generate pdf

[danielskatz]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.23919/EuCAP60739.2024.10501570 is OK
- 10.1109/TAP.2014.2323961 is OK
- 10.1109/ACCESS.2015.2453991 is OK
- 10.23919/EuCAP57121.2023.10132934 is OK
- 10.1371/journal.pone.0260060 is OK

MISSING DOIs

- No DOI given, and none found for title: Adam: A Method for Stochastic Optimization
- No DOI given, and none found for title: The DeepMind JAX Ecosystem
- No DOI given, and none found for title: JAX: composable transformations of Python+NumPy pr...
- No DOI given, and none found for title: jaxtyping: type annotations and runtime checking f...
- No DOI given, and none found for title: Equinox: neural networks in JAX via callable PyTre...
- No DOI given, and none found for title: Learning to Sample Ray Paths for Faster Point-to-P...
- No DOI given, and none found for title: Advanced Radio Channel Simulator
- 10.1109/gcwkshps58843.2023.10465179 may be a valid DOI for title: Sionna RT: Differentiable Ray Tracing for Radio Pr...
- No DOI given, and none found for title: TensorFlow: Large-Scale Machine Learning on Hetero...

INVALID DOIs

- None

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[danielskatz]

@jeertmans - I've suggested some changes in jeertmans/DiffeRT2d#78. Please merge this or let me know what you disagree with.

[jeertmans]

Thanks, @danielskatz! It looked great, and I also remove one footnote that was no longer needed (as I now include the full code snippet).

[jeertmans]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈