A thought experiment. There must be somehwere administrators can go to find out how and why they should credit a researchers digital contributions, right? We'll replace this with that place when we find it.
A semi-currated list for administrators documenting how/why scientific researchers should get credit for their digital contributions.
Best: Fork the repository, make a pull request. Good: Submit an issue.
Must meet each of the following:
- Citable
- Accessible beyond the individual researcher
- Published, public repository
- Published, private repository [use by invitation, sign-in]
And be in one of these categories:
- Data compilations (e.g. databases, controlled vocabularies, ontologies, spreadsheets, digital image sets, shapefiles)
- Software (e.g. code libraries, desktop or mobile applications, scripts)
- Digital software services (e.g. application programming interfaces (APIs), software as a service layer, wrappers to computational resources)
- Technical documents (e.g. standards like ontologies, controlled vocabularies, schemas, software manuals, video tutorials, standard operating procedures)
- Technical illustrations (e.g. animations communicating scientific concepts, cover illustrations for journals)
- Hardware systems administration (e.g. managing a multi-user computational cluster, managing research-supporting clouds)
- Software development administration (e.g. defining programmers tasks, merging code, reviewing code, communicating with developers support channels)
- Technical software administration (e.g. administration of relational database, administrating research support wikis, administrating computing cluster software, administering scientific applications on the web or networks)
- Data manipulation (e.g. data extraction, data merging, data cleanup, data validation, data packaging, instantiating databases)
- Data archiving (e.g. properly registering datasets, annotating datasets with citable identifiers and appropriate standards, validating data in archives and archive derivatives)
- Informatics Education (e.g. teaching researchers software carpentry principles, database management, scripted data manipulation, principles of data archiving, principles of data standards)
Metrics are dependant on the type of product being evaluated. It is critical to note that products in these categories have a diverse range of impacts, some of which can be far beyond the original intent. The following are variously applicable, and not intended to represent a comprehensive set. They should not be evaluated without a deeper understanding of what they represent. For example a research stating that they made 10,000 changes to a code base may not be as important as a researcher who made 10 changes (i.e. they simply might have a different workflow, be not as effective, etc.). Nevertheless when taken collectively each item here is important to evaluation.
- Number of times asset has been downloaded (e.g. downloading a software library)
- Number of times a digital asset has been referenced in a publically accessible format (e.g. references can be DOIs, but also things like URIs from ontologies or controlled vocabularies)
- Number of times asset has been accessed (e.g. page hits)
- Number of times an asset has been independently contributed to (e.g. outside researchers voluntarily contributing code to open source code libraries, this metric is particularly important)
- Number of dependencies (e.g. if a software library is produced, adopted by other external software developers, and required, then it is particularly impactful; determinable from sites like Github
- Number of open source libraries pushed to repositories
- Number of containerized applications pushed to repositories (containerized applications are applications that can be independently deployed to the cloud, their maintanence and packaging is an important technical contribution)
- Number of researchers commits to a code base
- Number of code bases committed to (e.g. a research committing small amounts to many applications or code bases will have a very big impact; one possible metric is the number of accepted “pull requests”, a standard mechanism for adding code to someone else’s project)
- Number of lines of code written (note this says nothing of quality of code)
- User time spent on digital application
- Number of data changes by OTHER users to digital resources managed by researcher (e.g. if I make a tool that 1000 researchers use to alter data this is immensely impactful, much more so than, say, a paper that gets cited 20 times)
- Number of users using a tool
- Number of logins of users using a tool
- Frequency of users returning to a tool
- Lines of code documented, lines of documentation written
- Hours of digital tutorials produced
- Number of project stars (e.g. on Github a star indicates someone else has more than a passing interest in a project)
- … many others
- Number of external contributors “managed” (open source software development is highly impactful if external contributors are involved, this takes a lot of time to manage, but is highly rewarding)
- Number of code reviews done
- Lines of data extracted, manipulated, etc.
- Number of datasets manipulated
- Number of machines managed
- Number of databases managed
- Size of databases managed
- Longevity of databases managed (seems abstract, but is increasingly important)
- Types of software used to manage machine (e.g. address concept of technical competence)
- Types of software managed in support of science (e.g. managing relational databases is much more demanding than desktop applications installed once and forgotten)
- Number of researchers helped with digital problems (side by side)
- Number of issues closed (application, database, code support)
- Number of responses to ticketed issues (times digital support provided)
- … many others