-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Software Paper Review: Suggestions for Clarity and Completeness #115
Comments
Thank you. I will make these changes as soon as possible. |
Further to @Naeemkh comments above, some additional comments and suggestions on the updated version of the paper in #116 SummaryThe summary section needs to be made more suitable for a diverse and non-specialist audience with some context about problem. What do you mean specifically mean by 'Online ranking' and what specifically is a rank in this context? I would add a couple of sentences just to give some introduction to the problem being considered for readers who may have no prior knowledge of online gaming, for example
Ideally avoid short forms like 1v1 which will not necessarily be clear to all readers. There are several vague, unreferenced or otherwise backed up claims that should either be better justified or made more specific:
Statement of need'MMO' needs defining on first usage. 'Similar to TrueSkill' - 'OpenSkill offers a pure Python implementation of their models' - from a brief read through, it seems the main contribution of Weng and Lin (2011) is a methodology / algorithm for approximating the Bayesian updates to player skill estimates given outcome data for a series of previously proposed probabilistic models for ranked data. I think it would be clearer therefore to say something like 'OpenSkill offers a pure Python implementation of their Bayesian approximation method for probabilistic models of ranked data'. or 'approximate Bayesian inference algorithm for estimating the parameters of probabilistic models of ranked data'. 'designed for asymmetric multi-faction multiplayer games' - I would say defining what is meant by asymmetric and multi-faction here would be helpful. 'However OpenSkill boasts several advantages over proprietary models like TrueSkill' - 'TrueSkill model' is not entirely clear here, as TrueSkill framework combines both a specific probabilistic model and an expectation propagation based approximate Bayesian inference algorithm for updating the model parameters. Likewise OpenSkill is an implementation of a specific algorithm for estimating the parameters of several different probabilistic models. Ideally you should make clear whether a claim is with regards to the models used, algorithm or specific implementation.
BenchmarksThis section should explicitly state what TrueSkill implementation is being compared to (from the code I believe this is Python 'Using a dataset of overwatch matches and player info' - 'overwatch' should be capitalized to make clear it is a proper noun and either a citation to a URL explaining what it is included or a footnote / inline comment giving some context. 'info' should be written in full as 'information' 'Using a dataset of Overwatch [citation to a URL or document explaining what Overwatch is] and player information' 'predicts the same number of matches as TrueSkill' - I think something like 'gives a similar predictive performance to TrueSkill' would be better here. Comparison to related packagesYou currently do not directly mention any other commonly used software packages for online player rating estimation, or state how OpenSkill.py compares to them in the main paper (there is brief mention of openskill.js in Acknowledgements in updated paper). Given the acknowledgement in the project README that 'this project is originally based off the openskill.js package', this should also be made clear in the paper, along with some overview of the relative advantages of the implementation here (even if this is just being available in Python as opposed to JavaScript). Similarly some of the implementations in other languages references in the README could be mentioned in the paper, as could the Python ReferencesTwo instances of 'bayesian' need to be capitalised by adding braces |
Please consider the following in drafting the software manuscript:
This issue is related to this submission: openjournals/joss-reviews#5901
The text was updated successfully, but these errors were encountered: