[x] [Topic 1 Day 2] Recognising RSEs by making software visible

[supernord]

Come share ideas for making your software more visible! This discussion aims to highlight how visibility can help Research Software Engineers (RSEs) gain recognition for software, and to workshop potential next steps that the community could take to make software findable, citable, and cited.

[supernord]

Sharing some resources here

• National Agenda for Research Software: National Agenda for Research Software au-research/Visible-Research-Software#23
• How do researchers find research software?: Short term action: Understand: How do researchers find research software? au-research/Visible-Research-Software#32
• Five recommendations for fair software: https://fair-software.eu/
• FAIR4RS: https://doi.org/10.1038/s41597-022-01710-x
• Visible research software interest group: https://ardc.edu.au/article/launching-the-visible-research-software-interest-group/
• FAIR workflows: https://doi.org/10.5281/zenodo.7783772
• Finding workflows in the Galaxy universe: integrating a global ecosystem using workflow registries: https://doi.org/10.5281/zenodo.8157531
• WORKSHOP: Make your bioinformatics workflows findable and citable: https://doi.org/10.5281/zenodo.7787487

[tom-h]

There are lots ways to break this down. It would be interesting to hear about preferences for discussion. Here are some ideas:

(1) what do researcher actually want? The ARDC have a report looking at "how researchers find software" which gives hints about how to reach researchers who might use your software.

(2) Publishing best practice. What works for drawing attention.

(3) Software Citation. A favourite for discussion, but something kind of out of the control of this group.

(4) Dan Katz' notion of "transitive credit" is another angle -- surfacing credit due to dependencies.

(5) Workflows or formal workflow capture.

(6) Policies laying out norms? This is tricky, because there are lots of touch points: journal policies, institutional policies, funder policies, and scholarly society policies all impact on norms or expectations for making software visible.

In some work I've been doing with Karthik Ram and James Howison, they made several recommendations for suitable infrastructure, but two specific ones are:

(7) the adoption of SBOMs for research software, and

(8) the production of a "Software Infrascope" using a methodology similar to the way the French Open Science monitor which measures the increase of openly available code arising from research over time by monitoring open access articles looking for mentions of software use (and creation and sharing as well).

[supernord]

Link above didn't work, adding it again here: https//doi.org/10.5281/zenodo.7340033

[tom-h]

Thanks... I fixed the link

[Aleem2]

Software < -- RSE <-- Recognition
So what kind of software outputs should be recognised?
Or is it we treat software (say script) = to data

[supernord]

Software covers everything from code snippets and scripts through to software packages and computational workflows

[tom-h]

The ARDC uses a three layer model, expressed in the "National Agenda for Research Software". What "recognition" means depends on which of three you are considering.

What we call "analysis code and other related forms of code" documents the research process. The code is the documentation of the analytical process. What is being recognised with this code is transparent ways of working, and doing research with a high degree of integrity.

What we call "prototype tools" capture new ideas methods and models for use by other in research. What is being recognised in this category is novelty and innovation: those new ideas, new ways of working, methodological innovations, and novel modelling of phenomena.

What we call "Research Software Infrastructure" are those forms of software which are used and valued by researchers in the conduct of research. What is being recognised is that research depends upon this software to do it's work.

This breakdown is independent to the way the code is expressed. For instance, computational workflows could be any of these three categories.

[cynthiahqy]

Researchers working on "Glue" components and methods more broadly are often structurally disadvantaged.

[tom-h]

I wish I had the reference handy, but I still believe the most effective thing a researcher writing glue code can do is to clearly and prominently ask to be cited in return for using the code. It doesn't guarantee it will be -- nothing does -- but the vague quote I remember in the forgotten paper I read is that it jumps from a 20% to 60% likelihood if you do so.

The other half of the equation is the researchers themselves agreeing upon norms of citation within their field. If you have any chance to be involved in such a discussion, then this is I believe the most realistic avenue for change in practice. Such discussions are active and fruitful in computationally dependent fields like astronomy, bioinformatics or earth sciences. It's early days in most other fields though.

[Aleem2]

I was wondering if there are any journals and conferences to which RSE's can publish their work and get recognition?

[supernord]

https://www.software.ac.uk/which-journals-should-i-publish-my-software

[Aleem2]

These days ML/AI is taking over most of the traditional computational techniques. Would it be right to say that it is more about the data itself with labels and less of AI methods and pipelines being use as we progress? Hence it is about data visibility we should also be talking about?

[tom-h]

I'd say "taking over" is a bit of an overstatement. I think there's a lot of experimentation, and critique, but I think we're still very far from it being the norm. Research still leans on a very broad array of computational techniques.

[tom-h]

And for what it's worth, I think data visibility is also worth talking about. It's not an either or thing in my mind.

[pdart]

Summary of discussion (thank you to all contributors):

The first three comments above (Topic outline by Johan, Resources by Johan, and Topic breakdown by Tom) provide context for some of the notes here.

We discussed:

• Katz’s concept of transitive credit; often just the code that captures a key research process gets credited, but code is usually depends on other software which doesn’t get credit.
• Contributing to open source and other systems; we can collect statistic such as downloads to support contribution, esp. where subsequent publications show usage.
• Dependencies are a big issue for (computational) reproducibility – meta data around binder compatible repositories - can specify in a technical sense but how do we reference and make visible.
• Licencing with dependencies – software bill of materials of materials (SBOM) and other licencing approaches.
• ARDC has report coming out – paper on how software is cited in research – raises the question of the adoption of SBOMs.
• Sometime very liberal licencing is a barrier to reuse, for example, in trying to develop a toolbox for MathWorks, can’t reuse from very liberally licenced publicly available toolbox.
• Need to consider the purpose for which the software is being developed and how this purpose changes over time (e.g., from local focus to national/global, from internal to public engagement or commercialisation) potentially broadening the target audience and creating opportunities for increasing visibility, but also potentially additional obligations in maintenance, quality, ethics, volume and duration of data storage, and so on.
• How do we solve the problem of what to cite for research in relation to software? The culture can vary widely for citation in general across different fields/domains. Researchers are looking for guidance about what to cite; we should work with scholarly societies to provide such guidance. Method makers (a class of software producers, often those with CS, Maths, or similar backgrounds) often produce software with broad applicability, however the culture of citation focusses on domain peers rather than recognising more generic contributions. Integrity is part of acknowledging more generic contributions but focus on this also varies across different fields of science.
• Provenance for data products can be a big problem, for example, a project producing terabytes of data based on converting telemetry on tape to imagery on HPC; it is easier to capture the environment with containerisation, however minor software version changes can result in differences such as with floating point results.
• FAIR Data, including Tidyverse, tidy data, making Jupyter notebooks fair (ref. to poster).
• How to write up the different types/parts of research software: process capturing vs domain specific platforms vs other elements and so on, and how to raise the profile of the different elements. Developers of glue software can be at a disadvantage, despite this underpinning much research.

[tom-h]

The ARDC has two reports coming out before the end of the year. The first is by Karthik Ram and James Howison making suggestions for suitable infrastructure developments or activities to drive greater software visibility. This is due by October. The second is a report by Nick Weber and Eva Maxfield Brown will estimate the likelihood of software outcomes arising from funding in Australia. It is close to a rerun of their preprint doing much the same for the US. This will probably come out November/December.