Create mcro.md

[ProfTuan]

Resolves #1912

[jamesaoverton]

One of our automated tests failed: For your contact person, we require a GitHub username in the github field.

[jamesaoverton]

Thanks for adding the github field. I also shortened the title to just "Model Card Report Ontology" (to match all the other OBO projects). Is this OK with you?

There are some other problems, hopefully small, that I can't figure out. I'll leave that to @pfabry and @matentzn.

[ProfTuan]

@jamesaoverton that's fine. Thanks!

[deepakunni3]

It looks like the error in the test is due to a difference in the formatting (observed vs expected). It would be useful to run tox -e lint and commit the changes for mcro.md. This should fix the tests.

[cthoyt]

There are several parts in the dashboard check that are not passing: https://obofoundry.org/obo-nor.github.io/dashboard/mcro/dashboard.html

what's the policy on accepting ontologies that are not passing parts (besides the plurality of users check)?

[matentzn]

The definition of "passing" is to not have any fails, not to have only passes, so MCRO passes.

However, @ProfTuan,

Version IRI 'https://raw.githubusercontent.com/UTHealth-Ontology/MCRO/main/mcro.owl' is not in recommended format

• Is a really central error, even though the dashboard is a bit lenient about it. Can you fix the version IRI in your ontology please to adhere to the versioning principle?

[pfabry]

The definition of "passing" is to not have any fails, not to have only passes, so MCRO passes.

Indeed, the ontology passes the dashboard. However, a quick manual review shows several problems: for example there are two classes named entity: http://www.w3.org/ns/prov#Entity and http://purl.obolibrary.org/obo/BFO_0000001, the object property is about is duplicated because of different URIs: http://purl.obofoundry.org/obo/IAO_0000136 and http://purl.obolibrary.org/obo/IAO_0000136
Maybe this is not the place to discuss this and I could create an issue elsewhere but my understanding is that to add a mcro.md in OBOFoundry github means that this ontology is already accepted?

[matentzn]

@pfabry @cthoyt - Let's stick with the process.

1. Errors are reported on the ontologies issue tracker, not here.
2. This PR has nothing to do with the ontologies acceptance, this is in the past. If you see glaring issues, open an issue here on the issue tracker and request a certain QC check to become mandatory. We vote on it, add it to the Dashboard, and next time it will be caught.
3. The PR is purely about ensuring that the ontologies description is sufficient.

@ProfTuan I agree though with @cthoyt that your description could be a bit more detailed, given that the main consumers of the registry will not know what a model card report is. Is a rough approximation:

1. A single clear sentence in the description field
2. At least a paragraph describing the ontology and its intended uses in the main part in the YAML below. As @cthoyt suggests, you can draw from the examples in the acceptance discussion.

[ProfTuan]

@pfabry @cthoyt - Let's stick with the process.

1. Errors are reported on the ontologies issue tracker, not here.
2. This PR has nothing to do with the ontologies acceptance, this is in the past. If you see glaring issues, open an issue here on the issue tracker and request a certain QC check to become mandatory. We vote on it, add it to the Dashboard, and next time it will be caught.
3. The PR is purely about ensuring that the ontologies description is sufficient.

@ProfTuan I agree though with @cthoyt that your description could be a bit more detailed, given that the main consumers of the registry will not know what a model card report is. Is a rough approximation:

1. A single clear sentence in the description field
2. At least a paragraph describing the ontology and its intended uses in the main part in the YAML below. As @cthoyt suggests, you can draw from the examples in the acceptance discussion.

@matentzn @cthoyt I provided an update to .md file. However. I want to be sure, is there a description section for the .yml file? I've looked at various .yml files for obo and none of them have a description section.

[cthoyt]

It's true, we've had difficulty convincing new ontology requesters to write nice, detailed descriptions of what their ontology does, what was the motivation, etc. in the actual submission markdown, even given the fact that they already had to write it for the submission.

Here's an example of a well-written ontology description: https://obofoundry.org/ontology/mondo. In https://raw.githubusercontent.com/OBOFoundry/OBOFoundry.github.io/master/ontology/mondo.md you can clearly see where the detailed description is written. I highly encourage you to start adding the following text, which I have merely copy-pasted from your original issue:

Model card reports are documents detailing transparent metadata information relating to machine learning models. Similar to what we have with drug labels and nutritional labels, the goal of model cards are to communicate relevant information on all aspects of a machine learning model that have undergone any experimentation. However these important reports of the machine learning models are presented in static documents. This work encodes the structure of model card reports and align them to standard OBO Foundry ontologies to help formalize and enrich these documents. The end result is computable model of the model card that can be used to standardize reporting and be integrated in future software tooling (searching and indexing, etc.).

The project was started when we noticed that NIH had mentioned an interest in model card reports for AI-based machine learning models in their bridge2AI initiative. So while the ontology resource can be used for application purposes, it serves as way to standardize and formalize model card reporting documents. The comments about the performance metric subclasses are actually document sections not the actual metric. We could import STATO later and link the metric subsections (e.g. "accuracy section" > is about > stato:accuracy). Overall the ontology just represents report documents.

Use Case

I'm trying to understand what your use case of the ontology is - looking at the use of data properties, are you building an ontology that is used as a data model, perhaps for validation? If so, have you considered technologies like SHACL instead that might allow more complex shapes to be built for validation purposes?

I think looking at the data properites is misleading. We are not trying to validate data, as so much just to provide a template for model card reports using an ontology-based framework to formalize these types of documents for biomedical informatics research that utilize AI-based machine learning. I also feel that use case question might be a bit unfair. Without naming specific projects there are one or two ontologies that are currently on OBO which are clearly application ontologies.

We do have a paper that might clarify the scope and use case:

Amith, M.T., Cui, L., Zhi, D. et al. Toward a standard formal semantic representation of the model card report. BMC Bioinformatics 23 (Suppl 6), 281 (2022). https://doi.org/10.1186/s12859-022-04797-6

^ all of this text was already written. Please please please edit AND revise it so anyone who looks at the page for your ontology can better understand what it is, why it exists, and why it's part of obo (in a way that's nice english prose, not copy/pasted offhand comments)

[ProfTuan]

I'll revise it. I should mention that the examples and templates that was shown to me had very simple descriptions which is what I thought was expected.

[matentzn]

@ProfTuan just a quick ping so we can start listing your ontology on our site!

[matentzn]

Remaining action items:

@ProfTuan thanks for the edits. This is better, but

• it does not address @cthoyt comments in Create mcro.md #2368 (comment)
• your version IRI is still wrong: Create mcro.md #2368 (comment)

[matentzn]

@ProfTuan thanks for adding more content, please ping me with my github handle when both the open action items above are addressed. Thank you!

[ProfTuan]

@matentzn both are addressed to the best of my knowledge.

[deepakunni3]

@ProfTuan Thank you for working on addressing the concerns raised by @cthoyt and @matentzn.

Couple of things regarding the metadata provided:

• please provide a user for usage section. This is typically a URL for the resource that is utilizing MCRO
• please provide a url for the examples section that demonstrates the usage of one (or more) terms from MCRO

You can refer to https://raw.githubusercontent.com/OBOFoundry/OBOFoundry.github.io/master/ontology/mondo.md or https://raw.githubusercontent.com/OBOFoundry/OBOFoundry.github.io/master/ontology/uberon.md for how these fields are populated.

[cthoyt]

@deepakunni3 FYI my concerns are explicitly not yet addressed

[deepakunni3]

@cthoyt Got it. I have updated my previous comment accordingly

[matentzn]

My concern is addressed, I checked it off. @ProfTuan sorry being a bit insistent, but we need to get this right now, else it will never be fixed. Can you carefully re-read @cthoyt comments above and extend your explanation according to his suggestions?

[ProfTuan]

@matentzn I added what I thought was missing along what @deepakunni3 had mentioned. Let me know if there's any issues.

[cmungall]

I recommend sourcing the original publication my Mitchell et al https://arxiv.org/abs/1810.03993

…

On Fri, Jun 9, 2023 at 9:30 AM Charles Tapley Hoyt ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In ontology/mcro.md <#2368 (comment)> : > + title: 'Toward a standard formal semantic representation of the model card report' +repository: https://github.com/UTHealth-Ontology/MCRO +tracker: https://github.com/UTHealth-Ontology/MCRO/issues +usages: +- description: 'MCRO used for publishing model cards' + user: https://github.com/UTHealth-Ontology/MCRO-Software + examples: + - description: 'Demonstration of Java-based library utilizing MCRO to output RDF-based model card reports' + url: https://github.com/UTHealth-Ontology/MCRO-Software + publications: + - id: https://doi.org/10.1145/3543873.3587601 + title: 'Application of an ontology for model cards to generate computable artifacts for linking machine learning information from biomedical research' +activity_status: active +--- + +Model card reports are documents detailing transparent metadata information relating to machine learning models. Analogous with drug labels and nutritional labels, the goal of model cards are to communicate relevant information on all aspects of a machine learning model that have undergone any experimentation. The National Institutes of Health, through their Bridge2AI initiative, expressed interest in model cards as an artifact to communicate specifics and promote transparency for AI-based machine learning models in biomedical research. However, these reports are presented in static documents, and have the potential to impede any possible retrieval, analysis, and aggregation of these reports. Model card reports translated as RDF-based formats could help in this effort, including supplementing application tools to analyze AI-based machine learning models. I now understand that a model card is some kind of metadata about a machine learning model, but I still don't understand in any detail. Can you provide a detailed, concrete example of a model card (in the markdown)? Maybe an image would be helpful. @ProfTuan <https://github.com/ProfTuan> Do you understand the issue I am having? I am a person with no knowledge of this topic and I want to understand it, so I need to be taught in a way that's written to help me learn. Other people who are reading through OBO Foundry will probably also feel the same way. — Reply to this email directly, view it on GitHub <#2368 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOMCWXBA64RXOUCY3XLXKNFT7ANCNFSM6AAAAAAYCRYHHQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[cthoyt]

Hi @ProfTuan, I saw you made an update to the PR, but I am not sure if you saw my message at #2368 (comment). Here's what it said:

I now understand that a model card is some kind of metadata about a machine learning model, but I still don't understand in any detail. Can you provide a detailed, concrete example of a model card (in the markdown)? Maybe an image would be helpful.

@ProfTuan Do you understand the issue I am having? I am a person with no knowledge of this topic and I want to understand it, so I need to be taught in a way that's written to help me learn. Other people who are reading through OBO Foundry will probably also feel the same way.

Can you please comment on this? I think Chris's suggestion is good to include relevant references, but not sufficient to address my concerns. I would appreciate if you could give some feedback, because I'm not sure if you understand what I'm asking or not.

[handemcginty]

Hi @ProfTuan, I saw you made an update to the PR, but I am not sure if you saw my message at #2368 (comment). Here's what it said:

I now understand that a model card is some kind of metadata about a machine learning model, but I still don't understand in any detail. Can you provide a detailed, concrete example of a model card (in the markdown)? Maybe an image would be helpful.
@ProfTuan Do you understand the issue I am having? I am a person with no knowledge of this topic and I want to understand it, so I need to be taught in a way that's written to help me learn. Other people who are reading through OBO Foundry will probably also feel the same way.

Can you please comment on this? I think Chris's suggestion is good to include relevant references, but not sufficient to address my concerns. I would appreciate if you could give some feedback, because I'm not sure if you understand what I'm asking or not.

@cthoyt The paper @cmungall linked contains images of cards and details. Why wouldn't giving the paper as an example reference won't be sufficient?

[cthoyt]

@handemcginty I disagree that just linking to a paper is sufficient. Expecting someone to follow a link to read and understand paper is a pretty big burden for someone reading through OBO Foundry who wants to just understand what any given ontology is about. I admit that in many places, the topics are pretty common knowledge and asking for a detailed description wasn't necessary. However, here, this topic is outside of the typical scope. If the paper has nice examples, perhaps we could adapt them directly into the mcro.md file.

@ProfTuan it would be really important that you can let us know that you have read some of this feedback

[handemcginty]

@handemcginty I disagree. Expecting someone to follow a link to read and understand paper is a pretty big burden for someone reading through OBO Foundry who wants to just understand what any given ontology is about. I admit that in many places, the topics are pretty common knowledge and asking for a detailed description wasn't necessary. However, here, this topic is outside of the typical scope. If the paper has nice examples, perhaps we could adapt them directly into the mcro.md file.

@ProfTuan it would be really important that you can let us know that you have read some of this feedback

@cthoyt I don't understand what you disagree with. I was asking why you think it is not sufficient to have the paper for this ontology is not sufficient. I see similarly situated ontologies: https://obofoundry.org/ontology/ontoavida.html and https://obofoundry.org/ontology/kisao.html citing references for further information.

[ProfTuan]

@handemcginty I disagree. Expecting someone to follow a link to read and understand paper is a pretty big burden for someone reading through OBO Foundry who wants to just understand what any given ontology is about. I admit that in many places, the topics are pretty common knowledge and asking for a detailed description wasn't necessary. However, here, this topic is outside of the typical scope. If the paper has nice examples, perhaps we could adapt them directly into the mcro.md file.

@ProfTuan it would be really important that you can let us know that you have read some of this feedback

Actually I don't understand what the misunderstanding is, especially since we are referring to a short document (1-2 pages) that describe information about a thing. I did add some updates recently but what more is there to add, short of writing a pedantic description about a simple report?

[balhoff]

I think the description is useful and sufficiently detailed. My only suggestion would be to link to the arXiv paper instead of (or in addition to) the ACM paper, since the ACM paper is not open access and so unreadable by many people.