[PRE REVIEW]: CiteLang: Modeling the Research Software Ecosystem

[editorialbot]

Submitting author: @vsoch (Vanessa Sochat)
Repository: https://github.com/vsoch/citelang
Branch with paper.md (empty if default branch):
Version: 0.0.27
Editor: @faroit
Reviewers: @gflofst, @rmmilewi
Managing EiC: Kevin M. Moerman

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5"><img src="https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5/status.svg)](https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5)

Author instructions

Thanks for submitting your paper to JOSS @vsoch. Currently, there isn't an JOSS editor assigned to your paper.

@vsoch if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.09 s (1299.8 files/s, 120382.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          61           1156           1166           4207
Markdown                        19            137              0            608
HTML                             4            106              0            532
YAML                            12             79             16            464
reStructuredText                 8            511            532            411
SVG                              2              1             21            239
make                             1             28              6            143
TeX                              1             19              0            123
Bourne Shell                     4             29             20            105
CSS                              2             17              8             82
XML                              1              0              0             30
JSON                             1              0              0             22
Dockerfile                       1              3              1             14
-------------------------------------------------------------------------------
SUM:                           117           2086           1770           6980
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1398

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/jors.by is OK
- 10.1109/MS.2020.2973362 is OK
- 10.48550/ARXIV.1905.08674 is OK
- 10.1109/MIC.2014.88 is OK
- 10.1109/CHASE.2019.00039 is OK
- 10.7717/peerj-cs.86 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1016/S0950-5849(02)00049-6 is INVALID because of 'https://doi.org/' prefix

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vsoch]

@editorialbot commands

[editorialbot]

Hello @vsoch, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Generates the pdf paper
@editorialbot generate pdf

# Get a link to the complete list of reviewers
@editorialbot list reviewers

[vsoch]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/jors.by is OK
- 10.1016/S0950-5849(02)00049-6 is OK
- 10.1109/MS.2020.2973362 is OK
- 10.48550/ARXIV.1905.08674 is OK
- 10.1109/MIC.2014.88 is OK
- 10.1109/CHASE.2019.00039 is OK
- 10.7717/peerj-cs.86 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[Kevin-Mattheus-Moerman]

@arfon do you have the bandwidth to handle this one as editor?

[Kevin-Mattheus-Moerman]

@editorialbot invite @faroit as editor

[editorialbot]

Invitation to edit this submission sent!

[Kevin-Mattheus-Moerman]

@faroit can you handle this work as editor?

[Kevin-Mattheus-Moerman]

@editorialbot invite @faroit as editor

[editorialbot]

Invitation to edit this submission sent!

[Kevin-Mattheus-Moerman]

@faroit 👋

[faroit]

@Kevin-Mattheus-Moerman hi, sorry for not replying and delaying the process. I was on vacation and forgot to set my status. Let me have a look at the paper today and then come back to this issue.

[Kevin-Mattheus-Moerman]

@faroit any news, can you help edit this? It would be great if you could, even if it is outside your domain a bit.

[faroit]

@Kevin-Mattheus-Moerman I spent some time getting into this now and - yes, while its clearly outside my core domain – I find the contribution interesting and relevant. Happy to accept

[Kevin-Mattheus-Moerman]

@editorialbot assign @faroit as editor

[editorialbot]

Assigned! @faroit is now the editor

[faroit]

👋 Hi @vsoch, thanks again for your submission to JOSS and sorry for the delay. I will be serving as the editor for this now. We'll use this issue to identify reviewers and resolve technical issues.

This tool implements a broad (and alternative) vision of how research software can be cited outside classical DOIs. I would like to diversify the reviewers in order to accommodate this vision but also assess the implementation details at the same time. E.g. at least one very experienced senior researcher that can comment on the broader vision of citelang. But also one experienced python expert.

I think the paper is already in a good shape. Looking at the repository however, one thing I clearly miss is more polished examples of typical outputs from popular repositories in order to better understand the vision of citelang and how it is recommended to be used by research software engineers.

I feel that potential reviewers would like to quickly assess what citelang is addressing for their citation/research software workflow. Therefore I would suggest - before we find potential reviewers - to update the repository to accommodate:

• more examples outputs, ideally in a separate html docs gallery.
• more detailed "getting started" guide that describes the vision and how citelang addresses the issues raised in the JOSS paper. Ideally an example workflow. I would also think that part of your blog post can be used as material.
• clarification of why the software is subtitled "markdown syntax for citing software" - what is the relation to markdown?

@vsoch let me know if you think this sounds appropriate to you

[vsoch]

hey @faroit totally no worries about the timing - it's an amazing effort that y'all do here to do this in your free time (and it's still pretty fast imho).

I think the paper is already in a good shape. Looking at the repository however, one thing I clearly miss is more polished examples of typical outputs from popular repositories in order to better understand the vision of citelang and how it is recommended to be used by research software engineers.

I'm happy to add this, but did you not see the docs? https://vsoch.github.io/citelang/getting_started/user-guide.html. CiteLang is a fairly diverse tool and there is no "here is the one use case" or "this is exactly how you are supposed to us it" for it. It's main purpose is that it's an API for getting data about contributions, turning that into a graph structure, and the intention (and why it is research software) is that you can use these graphs to understand an ecosystem as I've done with the rsepedia-analysis. That entire thing is derived from CiteLang data, and akin to how a researcher would use numpy in their project to do something higher level, I used citelang. The render/graph/badge provided with the software are extra bells and whistles, or how I chose to use these graphs (and provide fun tools for the interested user). So let's chat about what you think is missing - granted that you saw that documentation that has links to:

• badges that show the entire credit tree (including deps). That is shown in the README and in the docs here for multiple variations.
• summarizing contributions, described here
• interactive badges and credit trees linked from docs
• The functionality that originally derived the name - a "markdown syntax for citations" there is a pre and post rendering, where citelang reads in the pre, parses the citation and generates those same credit trees, and makes a nice table. That is also done automated and present in the citelang repository.

And the content of the paper is somewhat represented here and at the top of here.

I feel that potential reviewers would like to quickly assess what citelang is addressing for their citation/research software workflow. Therefore I would suggest - before we find potential reviewers - to update the repository to accommodate:

I don't think I'm saying this is yet a part of any research software citation workflow? I think right now our workflow is "make bibtex or a DOI with Zenodo and copy paste into a bibliography" and CiteLang is saying "this could be different." CiteLang is intended to be provided to empower people to model the space and think more about these ideas. To go back to the numpy example, numpy itself isn't explicitly stated in some workflow for a domain of analysis, but it's an underlying library that forms components that turn out to be. Let me know if that makes sense.

more examples outputs, ideally in a separate html docs gallery.

So I'm happy to add what you think is missing, but I don't see that any of the functionality that is provided is missing. Could you tell me what you think is missing? Do you want some sort of a "quick summary" bullet list of things that link to the same examples? Specifically what output are you looking for? I thought the requirement for JoSS was robust documentation, which I think I have with every example that the software can do, so I apologize if I missed an extra requirement (and I'd like to ask for clarity or a link to what you are looking for).

more detailed "getting started" guide that describes the vision and how citelang addresses the issues raised in the JOSS paper. Ideally an example workflow. I would also think that part of your blog post can be used as material.

Would this also be addressed by this summary listing? I think I present the tool as one idea for how we can influence the future of credit for software, but I don't make claims that there is some single command here that you run and "that's the answer." CiteLang has a lot of cool functionality and commands, but those are more examples of what I've built using these APIs and graphs. I'm hoping other researchers interested in software credit might do similar and make other cool ideas.

clarification of why the software is subtitled "markdown syntax for citing software" - what is the relation to markdown?

This is the syntax shown in the links above that is shown with render.

And there are GitHub actions for all of those things so if someone just wants to "generate a cool badge or a software credit graph from a paper they have written in markdown" that's probably the most direct usage, along with summarizing contributions. E.g., the singularity project is using CIteLang to be able to say thank you to contributions for each release, here is an example I was just checking with one of their developers:

Thank you for your comments!

[faroit]

@vsoch thanks for your detailed explanations.

I'm happy to add this, but did you not see the docs? https://vsoch.github.io/citelang/getting_started/user-guide.html. CiteLang is a fairly diverse tool and there is no "here is the one use case" or "this is exactly how you are supposed to us it" for it.

Thanks for the clarification. In the submitted paper you state:

"On a cultural level, CiteLang is the first tool of its kind to suggest an alternative way to value software beyond the traditional academic practice of publication. It suggests a model where RSEng are not required to generate separate DOIs (digital object identifiers) associated with papers..."

That is why, when looking only at the repo, I was a bit overwhelmed by the possibilities of citelang - in a very positive way :-) As I am not doing research on research software engineering, I was directly looking for an (opinionated), best-practice guide of how to use citelang in a research software package. I understand now that this wasn't the software's intention and you believe that this wouldn't be helpful for users – That's okay for me then and we can still come back to this later during the review process if reviewers ask for it.

So I'm happy to add what you think is missing, but I don't see that any of the functionality that is provided is missing. Could you tell me what you think is missing? Do you want some sort of a "quick summary" bullet list of things that link to the same examples? Specifically what output are you looking for?

Sorry, i wasn't clear. The example outputs are extensive enough. Then, a very quick summary at the top of #usage would be indeed helpful. E.g. what would a research software engineer typically use Credit, Graph and Badges for and how would one usually present them in a paper or a website -> example links.

I thought the requirement for JoSS was robust documentation, which I think I have with every example that the software can do, so I apologize if I missed an extra requirement (and I'd like to ask for clarity or a link to what you are looking for).

From what I see, the documentation meets all requirements!

This is the syntax shown in the links above that is shown with render.

👍 Maybe a bit of this could be added to the readme.md?

[faroit]

Proceeding with the review... @vsoch, I have one or two potential reviewers in mind but I am also interested in your input on potential reviewers, so any suggestions you have are also welcome.

For this, please check the JOSS COI policy and only mention them with a space between the @ and their handle (so you would refer to me as "@ faroit").

[vsoch]

Quick reference to supplement my description above - here is how the Singularity project is using it - https://sylabs.io/2022/05/whats-in-a-singularityce-release/ and https://twitter.com/Singularity_CE/status/1528785462210469889

[faroit]

here is a random search for "reproducible".
@ gflofst @ daissi @ fabianomenegidio.

@vsoch I did some research and contacted some people working in this area. I would also suggest: @ sdruskat @ hainesr @ yochannah - do you see any conflicts of interests with respect to the JOSS COI policy?

[vsoch]

I haven't worked with any of them - the only connection is that I interviewed the second on RSE Stories a few years ago: https://us-rse.org/rse-stories/2020/robert-haines/.

[faroit]

👋 @rmmilewi @gflofst @sdruskat @hainesr @yochannah - would any of you be willing to review this submission for JOSS? The JOSS review process takes place on GitHub and focuses on the software and a short paper. We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The software under review is CiteLang.

This issue is a "pre-review" issue in which reviewers are assigned. Once sufficient reviewers are recruited we will open a dedicated review issue where the review will take place.

[hainesr]

I could probably review this - my reticence is that I don't count Python in the set of ecosystems that I am confident in. I am on leave next week and then very busy in June, but so long as "sometime in June" is an acceptable timeframe, then I can do this.

[gflofst]

I can and would like to do this.

I know someone perfect we should add though. Reed Milewicz (rmilewi@sandia.gov). This is exactly his area and he is likely to generate citations as well.

[faroit]

@editorialbot add @gflofst as reviewer

[editorialbot]

@gflofst added to the reviewers list!

[faroit]

@gflofst thanks – great to have you on board here.

@vsoch do you see a COI with respect to @ rmmilewi?

[vsoch]

I know him from USRSE (he gives great talks!) but we’ve never worked together, so no I don’t think there is one.

[gflofst]

@vsoch I don't know all of his COI, but I believe this is safe.

[sdruskat]

I'm very interested in reviewing this, as this is extremely relevant for my own research. Would like to get a few days to clarify a potential COI before I confirm if that's okay?

[faroit]

@hainesr my reticence is that I don't count Python in the set of ecosystems that I am confident in.

I had a look at the code base and I think a deep knowledge of the python packaging system is likely to be useful. Thanks for your quick reply, though!

[sdruskat]

@faroit, sorry, I will have to pass on this opportunity to review.

[rmmilewi]

Sure! I'd be happy to do so!

…

On Wed, Jun 1, 2022, 11:34 PM Fabian-Robert Stöter ***@***.***> wrote: 👋 @rmmilewi <https://github.com/rmmilewi> @gflofst <https://github.com/gflofst> @sdruskat <https://github.com/sdruskat> @hainesr <https://github.com/hainesr> @yochannah <https://github.com/yochannah> - would any of you be willing to review this submission for JOSS? The JOSS review process takes place on GitHub and focuses on the software and a short paper. We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html The software under review is CiteLang <https://github.com/vsoch/citelang>. This issue is a "pre-review" issue in which reviewers are assigned. Once sufficient reviewers are recruited we will open a dedicated review issue where the review will take place. — Reply to this email directly, view it on GitHub <#4352 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD3OEVMFYCILQJTUKDSKVI3VNBBU7ANCNFSM5UOTXLMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[faroit]

@editorialbot add @rmmilewi as reviewer

[editorialbot]

@rmmilewi added to the reviewers list!

[faroit]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[hainesr]

I had a look at the code base and I think a deep knowledge of the python packaging system is likely to be useful. Thanks for your quick reply, though!

I thought as much! No worries

[vsoch]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vsoch]

I'm not sure if the bot here is using a cache? I fixed the URLs at the bottom to address vsoch/citelang#41 (see preview attached from Whedon preview service) but it doesn't appear to be reflected above.
whedon-preview-service.pdf

[faroit]

@vsoch thanks for the quick response. For me the proof looks fine now

[faroit]

@editorialbot start review

[editorialbot]

OK, I've started the review over in #4458.