[REVIEW]: APGG - A Modular C++ Framework for Asymmetric Public Goods Games

[editorialbot]

Submitting author: @jhstaudacher (Jochen Staudacher)
Repository: https://github.com/APGG-Lab/APGG
Branch with paper.md (empty if default branch):
Version: v1.1.2
Editor: @Nikoleta-v3
Reviewers: @ieyjzhou, @mstimberg
Archive: 10.5281/zenodo.8334926

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/4558a91d3a77910764ab49cd2db1a84f"><img src="https://joss.theoj.org/papers/4558a91d3a77910764ab49cd2db1a84f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/4558a91d3a77910764ab49cd2db1a84f/status.svg)](https://joss.theoj.org/papers/4558a91d3a77910764ab49cd2db1a84f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ieyjzhou & @mstimberg, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @Nikoleta-v3 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @mstimberg

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.06 s (1156.7 files/s, 68280.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                             29            417             95           1506
C/C++ Header                    30            157             11            644
Python                           5            155             28            437
XML                              2              0              0            337
Markdown                         2             42              0            131
TeX                              1              1              0            126
make                             1             18              2             25
-------------------------------------------------------------------------------
SUM:                            70            790            136           3206
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1162/evco.1996.4.2.113 is OK
- 10.1126/science.162.3859.1243 is OK
- 10.1007/s00355-012-0658-2 is OK
- 10.1038/s41598-020-79731-y is OK
- 10.1016/j.plrev.2016.08.015 is OK
- 10.7551/ecal_a_016 is OK
- 10.1038/415137a is OK
- 10.1088/1478-3975/12/4/046005 is OK
- 10.1007/s00265-006-0305-y is OK
- 10.1038/ismej.2017.69 is OK
- 10.13140/RG.2.2.27100.72322/2 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Wordcount for paper.md is 2049

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Nikoleta-v3]

👋🏼 @jhstaudacher, @ieyjzhou, @mstimberg this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist
as the top of a new comment in this thread.

These checklists contain the JOSS requirements ✅ As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention #4944 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@Nikoleta-v3) if you have any questions/concerns. 😄 🙋🏻

[Nikoleta-v3]

@jhstaudacher, you might recall that sylvaticus , even though they could not review the submission, left some useful suggestion/comments on #4711 (comment). I am not sure if you have addressed these. Just in case I am pasting them here:

Just a few comments/suggestions to the author package jhstaudacher if I can:

• add a couple of sentences after the logo on what your program does;
• create a proper documentation (there is no link to the "our wiki" page referred in the README);
• linked to the previous point, the code I saw has no comments. In C++ you have tools like Doxygen to help you structure the documentation that you write close to the code as "comment" and automatically extract the API (perhaps even other ones better nowadays). Use them :-) ;
• you shouldn't need to register to download an open source software, check the "release" method;
• on a quick look on the repository I haven't see any unit test. They are really useful, and some people even write first the test and then the implementation !

[jhstaudacher]

@Nikoleta-v3 Thank you very much for opening a review thread for our submission.

[jhstaudacher]

@ieyjzhou and @mstimberg Thank both very much for your readiness to review our software APGG and our paper.

[jhstaudacher]

@Nikoleta-v3 Thank you very much for reminding us of the precious suggestions and comments on our software APGG that sylvaticus provided three weeks ago. We will address these issues in the next release of our software APGG once the reviews are completed. Right now, I feel it is preferable not to change the code while the reviewers are assessing the software. Thus we can avoid confusion and misunderstanding. We will create a new release of APGG -- and revise our paper for JOSS -- addressing all suggestions from the reviewers (including the comments from sylvaticus) after we have the reviews. Thank you very much again for all your efforts.

[Nikoleta-v3]

Sounds good to me 😄

@ieyjzhou & @mstimberg, please remember to create your checklist by commenting the following:

@editorialbot generate my checklist

[mstimberg]

Review checklist for @mstimberg

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/APGG-Lab/APGG?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@jhstaudacher) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[mstimberg]

As a reminder: the topic is outside my field of expertise, so I cannot really judge, say, the appropriateness of the methods or the relevance to the field. I will therefore (mostly) constrain my comments on the technical aspects.

I was able to successfully compile and run the software (on Linux). IMO, the C++ code is of good quality and rather well-structured and readable. I agree with the comments by sylvaticus that it would be good to use some API documentation tool and/or unit testing, but from my point of view this is not strictly necessary for the paper/software acceptance into JOSS. However, I feel that there needs to be better step-by-step documentation for how to verify the correct working of the software, e.g. by reproducing Figure 4 of the manuscript. I've opened this and a few other issues directly over in the APGG repository (APGG-Lab/APGG#2, APGG-Lab/APGG#3, APGG-Lab/APGG#4, APGG-Lab/APGG#5, APGG-Lab/APGG#6).

I also have a few comments on the manuscript, I'll directly include them in this comment here:

• As I mentioned earlier, I am not from the field, but I do know the (in)famous "Tragedy of the commons" essay by Hardin. I am sure the authors are aware that the essay is highly problematic, to say the least. As e.g. Matto Mildenberger noted, "[Hardin] was a racist, eugenicist, nativist and Islamaphobe—plus his argument was wrong". I don't have a clear idea of what the best approach here would be, but I strongly feel that citing this paper/author without comment is not appropriate nowadays.
• Figure 4 is claimed to replicate Figure 6 from Hintze & Adami (2015), but I have trouble mapping the results between the two figures. Also, shouldn't the "percentile of agents" (rather "percentage"?) for "cooperators" and "punishers" add up to 100? The authors state that "For each datapoint along the axis of the synergy factor 10 replicate experiments
  were run", but do I understand correctly that the figure only shows the results from a single simulation? Wouldn't it make sense to use e.g. error bars or intervals to use all the information available? Finally, either the manuscript or the documentation should give clear instructions how to reproduce the figure.

Minor issues/comments:

• In l.29/30, the references seem to be for the MABE framework. Is this just meant as an example of a modular framework? Or APGG is related to MABE in some way?
• the citations in paper.md should use e.g. [@mcginty2013public,@hintze2020inclusive] instead of [@mcginty2013public],[@hintze2020inclusive] to correctly display multiple citations. Also, try reformatting the reference in l. 130.
• Figure 4: I wouldn't claim that a single successful replication proves that "APGG does work as intended"
• l. 103: I think it should be made clearer that itIsNotReallyAGroupLevelPayOffCalculator (what a name!), is an example for an extension, and not part of APGG (in case I understood that correctly, of course).

[ieyjzhou]

I tested the code on a Windows 10 system by using Visual Studio 2022. The code was successfully compiled and run. Even though the code is well-designed, it still takes a long time to read it. More document needs to be given for helping the reader understand the code. The maximum population should be explained in their paper. This will help the readers for setting their experiments. Usually, the software can not support a very large population. In the flowchart of figure 3, it seems that there is no end up. The authors need to add the stopping conditions in the flowchart.

[Nikoleta-v3]

Hey @jhstaudacher 👋🏻 did you have a chance to look over the reviewers' comments? 😄

[jhstaudacher]

Hey @Nikoleta-v3, thank you very much for your message from last Friday, December 16.
Yes, I forwarded the very helpful review by Marcel Stimberg (mstimberg) from Friday, December 16, 17:34 (CET), as well as the very helpful comments by Yanjie Zhou (ieyjzhou) from Friday, December 16, 18:09 (CET), to my coauthors and we already discussed the feedback we received. We will be very happy to revise our software APGG and our paper submission accordingly. Still, we are not certain whether the first round of reviews has actually been completed or whether there will be some kind of formal "decision letter" from you as our editor.
I am sorry if this question appears obtuse, but for my coauthors and me it is our first submission to JOSS. Thank you very much in advance for your reply and your efforts in editing our submission to JOSS.

[Nikoleta-v3]

Hey @jhstaudacher 👋🏻

I am sorry if this question appears obtuse, but for my coauthors and me it is our first submission to JOSS.

Not a problem at all! 😊

In general you won’t receive any formal decision letter from me or from JOSS. Everything (reviews, acceptance, rejection) will happen on this issue.

The format that JOSS has for the reviews is that reviewers write their comments here, or open issues on the project’s repository, and the authors address them. I won’t intervene, except if I believe an intervention is needed. The reason for this format is to keep the discussion between you and the reviewers ongoing. This way you can have a real conversation with them.

Regarding mstimberg's review: From their checklist (#4944 (comment)) there are only a few points that have not been ticked. They have raised their problems with these points by opening issues on https://github.com/APGG-Lab/APGG/issues.
I would now start addressing them. Ideally for each issue you can have a different commit or even a pull request if the issue requires a lot of work. If you have any questions you can reply on the issue for clarification. Once you are done addressing the issue, comment on the issue that you are done and ideally also include a link to the commit/pull request.

Regarding ieyjzhou's review: I can see that they have also raised some issues. I would address them and comment here how you resolved them (again with links to commits/pull requests). @ieyjzhou has to generate their checklist.

Once both reviewers have completed their checklist we can move forward. Please let me know if everything makes sense or if you have any further questions!

[Nikoleta-v3]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1162/evco.1996.4.2.113 is OK
- 10.1126/science.162.3859.1243 is OK
- 10.1007/s00355-012-0658-2 is OK
- 10.1038/s41598-020-79731-y is OK
- 10.1016/j.plrev.2016.08.015 is OK
- 10.7551/ecal_a_016 is OK
- 10.1038/415137a is OK
- 10.1088/1478-3975/12/4/046005 is OK
- 10.1007/s00265-006-0305-y is OK
- 10.1038/ismej.2017.69 is OK
- 10.13140/RG.2.2.27100.72322/2 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/csism-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4558, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[danielskatz]

Hi - I'm the track editor for this submission, and I'll proofread this and get back to you with next steps soon.

[danielskatz]

@jhstaudacher - see APGG-Lab/APGG#8 for some minor changes. Please merge this, or let me know what you disagree with, then we can proceed.

[jhstaudacher]

@danielskatz I just merged your changes. They are all fine. Thank you very much for all your efforts on our paper.

[danielskatz]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1162/evco.1996.4.2.113 is OK
- 10.1126/science.162.3859.1243 is OK
- 10.1007/s00355-012-0658-2 is OK
- 10.1038/s41598-020-79731-y is OK
- 10.1016/j.plrev.2016.08.015 is OK
- 10.7551/ecal_a_016 is OK
- 10.1038/415137a is OK
- 10.1088/1478-3975/12/4/046005 is OK
- 10.1007/s00265-006-0305-y is OK
- 10.1038/ismej.2017.69 is OK
- 10.13140/RG.2.2.27100.72322/2 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/csism-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4561, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[danielskatz]

@jhstaudacher - sorry, I missed one small change, as indicated in APGG-Lab/APGG#9 - after this we should be ready for acceptance and publication

[danielskatz]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1162/evco.1996.4.2.113 is OK
- 10.1126/science.162.3859.1243 is OK
- 10.1007/s00355-012-0658-2 is OK
- 10.1038/s41598-020-79731-y is OK
- 10.1016/j.plrev.2016.08.015 is OK
- 10.7551/ecal_a_016 is OK
- 10.1038/415137a is OK
- 10.1088/1478-3975/12/4/046005 is OK
- 10.1007/s00265-006-0305-y is OK
- 10.1038/ismej.2017.69 is OK
- 10.13140/RG.2.2.27100.72322/2 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/csism-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4562, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[danielskatz]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Rosenthal
  given-names: Mirko
- family-names: Richter
  given-names: David J.
  orcid: "https://orcid.org/0000-0001-5413-6710"
- family-names: Hübner
  given-names: Falk
- family-names: Staudacher
  given-names: Jochen
  orcid: "https://orcid.org/0000-0002-0619-4606"
- family-names: Hintze
  given-names: Arend
  orcid: "https://orcid.org/0000-0002-4872-1961"
doi: 10.5281/zenodo.8334926
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Rosenthal
    given-names: Mirko
  - family-names: Richter
    given-names: David J.
    orcid: "https://orcid.org/0000-0001-5413-6710"
  - family-names: Hübner
    given-names: Falk
  - family-names: Staudacher
    given-names: Jochen
    orcid: "https://orcid.org/0000-0002-0619-4606"
  - family-names: Hintze
    given-names: Arend
    orcid: "https://orcid.org/0000-0002-4872-1961"
  date-published: 2023-09-13
  doi: 10.21105/joss.04944
  issn: 2475-9066
  issue: 89
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 4944
  title: APGG - A Modular C++ Framework for Asymmetric Public Goods
    Games
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.04944"
  volume: 8
title: APGG - A Modular C++ Framework for Asymmetric Public Goods Games

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.04944 joss-papers#4563
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.04944
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[danielskatz]

The PDF isn't loading for me, so I'll hold off on closing this issue until it does.

[jhstaudacher]

@danielskatz Thank you very much for your last small change and all your efforts on our paper. This is just to confirm that I can not yet download the PDF of our paper as well. Thank you very much for holding off on closing this issue until the PDF becomes available and the problem is resolved.

[jhstaudacher]

@danielskatz Just to say that the PDF of our paper now downloads properly. As far as I can see, everything is fine (-- but you are the expert). Thank you very much again for all your support and efforts.

[danielskatz]

Congratulations to @jhstaudacher (Jochen Staudacher) and co-authors on your publication!!

And thanks to @ieyjzhou and @mstimberg for reviewing, and to @Nikoleta-v3 for editing!
JOSS depends on volunteers and we couldn't do this without you

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.04944/status.svg)](https://doi.org/10.21105/joss.04944)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.04944">
  <img src="https://joss.theoj.org/papers/10.21105/joss.04944/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.04944/status.svg
   :target: https://doi.org/10.21105/joss.04944

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss