Should we allow the use of URI syntax in SSSOM tables?

[matentzn]

To be self contained, the tsv format metadata should provide a curie map - we will make this mandatory.

[wdduncan]

I am working with the Darwin Core group on the Darwin Core to MIxS mappings. See this repo:
https://github.com/tdwg/gbwg

In their SSSOM spreadsheet, they have been using full URIs. Why do we want prohibit this?

cc @cmungall @raissameyer @pbuttigieg

[balhoff]

Should full URIs be enclosed in angle brackets, to make CURIE vs. URI obvious?

[matentzn]

Hmmmm... I dont know. Can I ask why? In the original proposal the whole point was to zone in on a terse, curie based representation for the tables that can be handled by standard python libraries.. I am for now slightly against this, but I can be convinced otherwise.

[balhoff]

Personally I find CURIEs frequently an unnecessary complication, but I haven't been much involved here so take my opinion with a grain of salt. :-)

[matentzn]

I personally agree with that as well - the amount of times I wrote curie-iri converters is uncountable :D I just feel that this is too much of burden to standardisation.. SSSOM plus some standard curie maps to refer to would allow a very easy way to compare stuff, and not bring us back into the country of UMLS cui vs OBO purl vs SCTID etc.. Lets see what @cmungall says..

[wdduncan]

We can (of course) add a section to the SSSOM document that is used to define the URI to CURIE mappings.

[matentzn]

The main idea of restricting to CURIEs is, correct me if I am wrong @cmungall, that the whole design of SSSOM revolves around the idea that the files should be easy to use for regular bioinformaticians - and URIs are considered an ugly distraction. I am super torn about the overhead for the semweb/obo community to having to provide prefixes each time they emit an sssom file..

One compromise could be that we are more permissive in sssom parse, basically allowing TSV files to be passed in that have IRIs, but spit out the proper, clean prefixed SSSOM files using the linkml curie map. What do you think @cmungall ?

[graybeal]

Funny, I thought it should be easy to use for regular LOD folks, and CURIEs are an ugly distraction for them. In addition to (or because of) being non-rigorous, unless they (and all the systems they are ever used in) fully support prefix definitions to supply the rigor.

[matentzn]

Can you elaborate on "ugly" in this case? Is it too much effort to use compress the IRI into a prefix, or is it somehow a churn to decompress them? I kind of like CURIEs as they are well-defined (in conjunction with mandatory prefix maps of course) - I would, from a purely aesthetical standpoint say CURIEs are prettier, easier to read and lead to much smaller file sizes.

With bioregistry and curies module we can also easily convert back and forth between the two representations. My main worry of permitting the use of URI is that it will create churn in interpreting the SSSOM files and integrating them. But as I said below, I think we can decide to go in that direction, or at least start preparing a vote about it.

See also comments below;

[graybeal]

So speaking somewhat off-the-cuff here, the 'W3C-trained' semantic professional (who didn't come up through bio ranks) is used to a single simple system: you can use prefixes if you define them at the beginning of the file. Many tools including SSSOM do this, and it is intuitive and not very burdensome. So most semantic assets mostly use CURIEs anyway—it's way easier to proofread. For the non-W3C semantic assets, there's a little bit of hesitation about whether any given tool can be counted on to work the same way, but not a big deal.

OTOH, I have run into many applications and files and values that use CURIEs without following that principle—they don't have a prefix definition. Typically the CURIE can be deciphered confidently with a bit of thought; sometimes it can't (a local CURIE not in the registry; a wrong prefix; or a prefix that has multiple expansions). Now the document is a hassle to interpret and manage automatically.

For example, take the SSSOM plus some standard curie maps to refer to phrase you used. In that case an outside user won't have any idea that they should look up the prefixes in 'the standard curie maps', if they aren't encoded at the top of the file in the standard prefix-defining syntax. For another example, "in conjunction with mandatory prefix maps" could mean 'just look it up in bioregistry'. But that only solves the problem if your data stays within your bioregistry community—the rest of the LOD and semantic communities don't use bioregistry as the standard set of prefixes to rule them all.

If we want the information to be truly interoperable across all communities in a rigorous way, the best way to do that is if everyone uses a self-evident and self-contained mechanism that conforms to a well-known international standard. That's what I'd like SSSOM to do consistently, even if it makes local identifiers a bit harder to parse out and adds a prefix definition for every CURIE used.

Default namespaces could be a good solution to the 'local identifier' or 'omit the common CURIE', no? I don't love that style but hey, it meets the requirements (self-evident (kinda), self-contained, well-known standard).

[wdduncan]

FYI for the DWC-MIXS mapping we re removing quotes from the curies.

cf. DWC issue 68

[matentzn]

Does DWC-MIXS implement a sssom validate QC step to ensure the mapping sets are compliant?

[wdduncan]

I don't know. Perhaps @raissameyer @pbuttigieg would.

[matentzn]

as long as it is valid yaml its fine

[matentzn]

In the workshop, we need to decide whether we should officially permit the use of IRIs in SSSOM or not. My sense is that parse should allow inputs with IRIs, but the canonical form should be in CURIE form. But I am happy to be convinced otherwise.

[nichtich]

In fact there are three possible methods to provide identifiers to be used as URIs:

• full URI: most flexible but very verbose
• CURIEs: shorter but requires prefix-map to be configured
• local identifier: condense and human-redable (e.g. use space instead of %20) but require namespace to be configured and all identifiers must share same namespace

[matentzn]

Can you give an example of how the latter are used in practice? This seems like baking too much logic into the datamodel at first sight, but I can see how some use cases which do not deal with IRIs might benefit from something like this (I doubt though that core team will allow this, but who knows :P).

[nichtich]

We usually get a spreadsheet with mappings for a specific vocabulary and in many cases the vocabulary must first be converted to RDF as well. For instance at the moment we are working with TUM classification with classes such as CHE 075: Sonochemistry. A CSV file referencing this class would include local identifier CHE 075. Given the information that TUM classification has URI namespace https://www.ub.tum.de/en/tum-systematik/no/, this identifier is then converted to URI https://www.ub.tum.de/en/tum-systematik/no/CHE%20075. No CURIEs involved and readable identifiers in CSV. A more complete example with a mapping between TUM and Wikidata given in three forms:

https://www.ub.tum.de/en/tum-systematik/no/CHE%20075 | http://www.w3.org/2004/02/skos/core#exactMatch | http://www.wikidata.org/entity/Q2475074
tum:CHE%20075 | skos:exactMatch | wd:Q2475074 
CHE 075 | exactMatch | Q2475074

The first is basically NTriples, easy to process but difficult to write by humans. The third is easiest for humans but requires additional background knowledge to process. CURIE is somewhere in between.

[matentzn]

Exactly! This is why we promote curies for this case; how would you know that Q2475074 in table 1 is the same as Q2475074 in table two? implicit knowledge from the context of where you find the table?

[nichtich]

CURIE prefixes are implicit knowledge as well, but at least they their syntax can be checked without knowledge of actual namespaces. It's all a matter of convention with benefits and disadvantages for each solution. SSSOM seems to be used more in the territory of RDF where basic acceptance can be assumed, so CURIEs give a good compromise. As said above, we more have to deal with data that is not RDF in the first place. Therefore we provide ways that people can keep using their non-RDF identifiers as long as possible.

[matentzn]

CURIE is not quite as implicit though as a prefix declaration must be provided for every prefix used to construct a curie. Anyway, point taken! Love the way you think this through!

[wdduncan]

In the Darwin Core SSSOM spreadsheet there is a mix of IRIs and CURIEs. Since an IRI to CURIE mapping may not always be available (or known), allowing IRIs seems reasonable to me. Allowing "local identifiers" seems more fraught with difficulty.

[nichtich]

I just looked up CURIE specification and its reference to RFC 3987. As far as I understand, a HTTP(S) URI is already a valid CURIE if we assume default namespace prefixes:

#curie_map:
#  http: "http"
#  https: "https"

Most applications seem to apply additional restrictions on CURIE syntax, e.g. IRIs in SPARQL to prevent this.

[matentzn]

In theory perhaps, but not in spirit - this is not a useful curie at all. In any case, you would never be able to register it in any meaningful prefix registry such as https://bioregistry.io/; we should strive for a more harmonised use of prefixes as well.

[graybeal]

But I think bonus points for combining creativity and rigor!

[graybeal]

Love the example by @nichtich.

How about this? Allow IRIs but strongly encourage them to be represented as CURIEs with included prefixes in the SSSOM. (We do require the prefixes, right?) Or, OK, even require they be represented as CURIEs.

I mean, if the reason for CURIEs is to make it easy to read and build for the bio community (and many others), then let's just say that's how we want all the IRIs to look too. Where the mapping is being done to a lot of different ontologies, you'll be able to see the list of ontologies via the (now-required) prefixes. It means extra work for the IRI fans, but they're smart, they can handle it. I mean, we can handle it.

Except for those funky http and https prefixes, of course, they have to go. Even if they do make me laugh every time.

[nichtich]

I support the use of full URI anyway, so better don't ask me. The CURE specification gives guidelines what to do when both URI/IRI and CURIE are allowed:

Specifications for particular attribute values or other content MAY be written to allow either CURIEs or IRIs (or URIs, etc.). The specifications for such languages MUST provide rules for disambiguation in situations where the same string could be interpreted as either a CURIE or an IRI.

For SSSOM I'd suggest that a http or https IRI should be allowed where a CURIE is allowed, so either it starts with http:, or https: or it must be a CURIE and prefix http and https are not allowed to be redefined.

[matentzn]

I doubt you will get a lot of support for this - but by all means, give it a shot on the issue tracker. There are a few people out there that would like to allow full IRIs, but they violate the spirit of the TSV serialisation, which is to be as compact and human readable as possible (and full URIs are neither). Also, we are trying to standardise prefix use at least in our community, which means that if we allow full URL, we will end up with a lot of plumbing for non-RDF people that need to understand that UBERON:123 and http://purl.obolibrary.org/obo/UBERON_123 mean the same thing.

Lastly, if we allow CURIE and URI syntax, we need a special line of code (if URL, compress, else expand) in every program to harmonise the two, while now, we can use a standard toolkit (pip install curies) to translate back and forth.

[graybeal]

Well, I thought this was the issue tracker, and that there already is a lot of support in this ticket (4)—but maybe we 4 are the entire pool of believers. I think the argument has been adequately made that full URIs are more readable for many of us, and we have in many cases added required elements to SSSOM that make things less compact. So it's a judgment. A special line of code is sort of a red herring given assumptions about installing software libraries with pip—obviously that target is developers, who could presumably write a whole line of code.

With that said, if the answer is still no, let's just say the answer is still no. You created the ticket and I think it's been sufficiently discussed at this point.

[matentzn]

It is not the issue tracker to be technically correct, it's the discussion section so we wont make any decisions here :).

It is important to realise that there is no one person that can make a decision like this; if you (4) feel like this is the right way to go, you should drive it forward in an issue that includes a vote and the summary of all arguments given here! I am ok either way, I am only supportive of curies because @cthoyt and @cmungall have bullied me (lovingly) into making sure that we mandate the use of CURIE syntax in the TSV format, and their arguments convinced me; I do see though that your arguments are also convincing, and it will end up to be a bit of a battle of tastes in the end. I will await one of you 4 calling for a vote to settle the issue!

[cthoyt]

I will remain strongly opposed to any suggestion to allow IRIs in SSSOM. Writing code to decide if something is a CURIE or IRI is painful and error-prone. URIs are longer, include extraneous/duplicated information, and are ultimately harder to read. URIs are (despite best efforts) not standardized in many cases either on a syntactical level (e.g., URL encoding), on a protocol level (http/https), or just that there are different URI formats. This is all solved by providing a prefix map, ideally one grounded to an external standard (i.e., the Bioregistry), that lets you keep the "rigor" of URIs and have the domain knowledge/interpretation + other benefits of CURIEs.

[graybeal]

Won't be me calling for a vote. I don't have the capacity to personally argue, let alone demonstrate, the validity of my point of view, and it's clear that some are not open to convincing. But much more importantly, we can do all the prefixing that @cthoyt calls for and it will not be painful or error-prone, so I'll just manage it that way. Fair enough.

[matentzn]

Just to be clear, there are always people that do not want to be convinced, but they cannot decide bottom up standardisation efforts :) Otherwise, there would not be any community standards in this world! But I hear you @graybeal, you are a beacon of reason in this sea of emotions and opinions, and my steam in terms of pushing the standardisation envelope is also not what it was!

[gouttegd]

It seems like a decision has been taken, since sssom validate now enforces that any EntityReference in a SSSOM/TSV file MUST be in CURIE form only – and there’s a PR in the work to update the documentation accordingly.

So can we close that discussion, or at least remove the requires-decision label?

[matentzn]

I removed the "requires decision" label. I think once we have a solid documentation together, we will close this discussion here with a pointer to that doc.

Note that we should probably have a sentence covering #188 (comment)