Skip to content

FAIRness assessment for software

Jump to bottom Edit New page
ljgarcia edited this page Dec 14, 2018 · 9 revisions

Table of contents

FAIRness assessment for software

In this project we want to figure it out how the FAIR principles apply to software, i.e., applications and tools, and how the FAIR metrics should be interpreted for this case. We want to find answers to the following questions:

  • Question 1. What is the scope we want to cover? Could/Should all sort of software be FAIR? We could consider here research software, programming languages, web services, commercial software, etc.
  • Question 2. What FAIR principles can be directly adopted?
  • Question 3. What FAIR principles should be adapted and how?
  • Question 4. Anything on the principles that should be dismissed?
  • Question 5. What should be added to the principles for the software case?

Background

The following sections reflect discussions previous to the Matsue BioHackathon as well as during the event.

Participants in discussions

Interested people

  • Erick Antezana
  • Salvador Capella
  • Jen Harrow (Elixir Hub)
  • Tomoe Nobusada
  • Núria Queralt

Additonal links

Goal and scope

The goal regarding FAIR principles for software is to make source code and/or executables easy to find and reuse on the long term. In this sense, FAIR principles for software apply to any possibl software. Metadata, however, could possibly change from, for instance, commercial to research software. By now, we will limit our scope to research software accross multiples sciences, using the Life Sciences domain as an initial example.

Whether metadata changes (significantly or not) from science to science requires a further evaluation out of the scope of this initial assessment.

The idea behind software FAIRness relies on common well practices, particularly those regarding versioning and releases as well as making software citable.

Community role on FAIR principles

FAIR principles rely on community agreements; therefore, a community should be created and define what FAIR principles apply to it and what stadards are required. The FAIR evaluation tool (under developement during this BioHackathon), will allow communities to register, select those principles that apply to them, and point to their standards validation services.

The FAIR evaluation tool will assess resources against those principles selected by a community. In this sense, resources will not be universally FAIR but FAIR according to a registered community.

Some thoughts on metadata and data

Based on discussions, some FAIR elements have been already identified for the software case

  • Metadata persistent identifier by submitting a release to a registry, e.g., Zenodo or FigShare
  • software identifier (probably a URL)
  • title
  • short name
  • description
  • version
  • release date
  • creator
  • maintainer
  • publisher
  • license and/or terms of use
  • URL to zip file containing software release
  • machine readable descriptions of input(s), parameter(s), and output(s) following stadards agreed by community
  • documentation including functionality
  • how to cite your software
  • following best practices
  • transparent contribution, governance and communication processes

Alignment to FAIR principles

Findability

F1. (Meta)data are assigned a globally unique and persistent identifier

  • Metadata Because of A2 "metadata should be accessible even when the data is no longer availabe", you should register your software, i.e., data in this case, somewhere else. There are some possibilities here, for instance:

    • Use a DOI provider so you automatically get metadata via content negotiation; you will not be able to decide which metadata is provided though, your metadata identifier will be that DOI
    • Use Zenodo to link to a GitHub repository for a particular release, it will automatically add some metadata as schem.org markup on their web pages, you have no control on the metadata provided. It will also assign a DOI to your software so you will also get that metadata, still your metadata identifier will be that DOI
    • Register your software in a specialized registry agreed by your community so the metadata will be that one also agreed on, your metadata identifier will be provided by the chosen registry
    • Provide metadata together with your data together
    • Check out the terms of use of the repository to understand their use and commitment to the DOI system

  • Data

    • Use a URL; as far as that domain is alive, it will be unique and "persistence". This URL sould point wither to your source code or to a dowload page
    • Register (so far for free) a PURL for your organization/team/group/your software and create an entry for the software you are describing (commonly used for vocabularies rather than software or other digital resources though)

  • Metadata and data

    • Provide a web page explaining your policy regading globality, unicity and persistence for your metadata and data identifiers. If you are using an external place for your metadata, you should link to their policy, for your data identifier you will have to provide your data
    • FAIr evaluator might ask you to provide a link to a page explaining the schema behind your identifiers (so you show you are aware of it and its implications)
    • FAIR evaluator should provide the properties to be used to link to those policy pages

F2. Data are described with rich metadata

  • Minimum set of metadata will correspond by that one asked for the registry/DOI provider you are using
  • Communtiy needs to define what rich metadata should be included
  • Community needs to provide a validator/evaluator against their metadata specification/guidelines

F3. Metadata clearly and explicitly include the identifier of the data they describe

  • Community will decide the property required to include the data identifier, it should be a link to a particular release where versioning have been used

F4. (Meta)data are registered or indexed in a searchable resource

  • Metadata

    • If you used a DOI provider or a registry, use their search capabilities. Make sure you can use the identifier, e.g., DOI, to find your metadata
    • If you provide your metadata together with your data, a search engine could fulfill this requirement but the FAIR evaluator might not be able to check it due to search engine restrictions

  • Data

    • See considerations above for metadata

Accessible

A1. (Meta)data are retrievable by their identifier using a standardised communications protocol

  • FAIR evaluator should define the properties to indicate what the protocol is and whether it is A1.1. or A1.2. for both metadata and data
  • If a community agreed registry is used, it should take care of comply to such property for the metadata case
  • If DOI is used, the community should at least try to reach DOI providers for them to include the metadata property required by the FAIR evaluator

A1.1 The protocol is open, free, and universally implementable

  • Metadata

    • Either DOI or registry URL, both of them can be retrieve via HTTP or HTTPS

  • Data

  • As URLs are in used here, retrievable via either HTTP or HTTPS
  • FTP possibly for downloads althought HTTP/HTTPS also works

A1.2 The protocol allows for an authentication and authorisation procedure, where necessary

  • FAIR evaluator should provide the properties to indicate authentication and authorization procedures
  • In this case, no validation regarding data will be done, only for metadata

Interoperability

I1. (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.

  • Metadata

    • It should be expressed following a knowledge representation language

  • Data

    • Not applicable for software (although it is possible to argue that programming langugaes are the equivalent here, all of them with their own syntax and semantics)

I2. (Meta)data use vocabularies that follow FAIR principles

  • Metadata

    • The chosen knowledge representation language should be FAIR

  • Data

    • Not applicable for software

I3. (Meta)data include qualified references to other (meta)data

  • Metadata

    • As a minimum, software should use well-known vocabularies to express therir input and output
    • Community will agree on what properties use to refer to other software or digital resources

  • Data

    • Not applicable for software

Reusability

R1. Meta(data) are richly described with a plurality of accurate and relevant attributes

R1.1. (Meta)data are released with a clear and accessible data usage license

  • Metadata

    • Make sure you understand the license that the metadata from the repository are released with

  • Data

    • Make sure you have a license for your software from day 1

  • Metadata and data

    • FAIR evaluator should provide the property to be used in this case for both metadata and data
    • If a community agreed registry is used, it should take care of comply to such property for the metadata case
    • If DOI is used, the community should at least try to reach DOI providers for them to include the metadata property required by the FAIR evaluator

R1.2. (Meta)data are associated with detailed provenance

  • FAIR evaluator should provide the property to be used in this case for both metadata and data
  • If a community agreed registry is used, it should take care of comply to such property for the metadata case
  • If DOI is used, the community should at least try to reach DOI providers for them to include the metadata property required by the FAIR evaluator

R1.3. (Meta)data meet domain-relevant community standards

  • See F2. Data are described with rich metadata
  • Please be aware that some of the xommunity standards regarding software are not necessarily related to FAIR prinicples or metadata. There are many guides and recommendations for effective release of software. This can include everything from use of code versioning systems, code review, unit testing, performance testing, usability testing, issue trackers, discussion forums, user feedback mechanisms, dependency management, software documentation, release notes, containerization (e.g. Docker), and rollback plans. Either seek out a production readiness checklist or work with your community to establish what needs to be done to prepare a public release

Assessment against metrics

Research software: BioKnowledge reviewer library

Metric Metadata Data Evaluator
FM-F1A: Identifier Uniqueness DOI 10.5281/zenodo.2235452 URL https://github.com/NuriaQueralt/graph-hypothesis-generation-lib/tree/v1.0 Passing
FM-F1B: Identifier persistence https://www.doi.org/doi_handbook/6_Policies.html#6.5 https://help.github.com/articles/github-terms-of-service/#o-disclaimer-of-warranties pending
FM-F2: Machine-readability of metadata https://zenodo.org/record/2235452 NA pending
FM-F3: Resource Identifier in Metadata https://zenodo.org/record/2235452 NA passing
FM-F4: Indexed in a searchable resource https://zenodo.org/search?page=1&size=20&q=zenodo.2235452 https://github.com/search?q=graph-hypothesis-generation-lib pending
FM-A1.1: Access Protocol HTTPS, open true, free true HTTPS, open true, free true pending
FM-A1.2: Access authorization false false pending
FM-A2: Metadata Longevity http://about.zenodo.org/policies/ NA pending
FM-I1: Use a Knowledge Representation Language http://schema.org/docs/jsonldcontext.json NA pending
FM-I2: Use FAIR Vocabularies http://schema.org/docs/jsonldcontext.json NA pending
FM-I3: Use Qualified References NA input: Edges to build the structured review, http://edamontology.org/data_2080; output, Structured review network, http://edamontology.org/data_2600 community
FM-R1.1: Accessible Usage License https://zenodo.org/record/2235452 https://github.com/NuriaQueralt/graph-hypothesis-generation-lib/blob/master/LICENSE pending
FM-R1.2: Detailed Provenance http://schema.org NA pending/community
FM-R1.3: Meets Community Standards ? ? community

Conclusions

The FAIR evaluator, also a project at BH18-Matsue, is a good starting point for FAIRness assessment. However, given the community factor inhent to FAIR principles, there is the need to

  • create community standards,
  • create community standars registries,
  • define community validators, and
  • make FAIR evaluator and community validators work together
Add a custom footer
Add a custom sidebar

Clone this wiki locally