Advertise score range used by a service in its manifest

[wetneb]

Services are free to use any numerical range for the scores they give to reconciliation candidates.

As a client, this makes it a bit hard to process those scores: we can only compare the candidates between each other and have no idea what the maximum or minimum score might be.

Services could advertise their min and max scores in their manifest (if any). Some might use unbounded scores on purpose and it could be nice to be able to indicate that too.

Alternatively, the specifications could mandate a particular score range.

Reconciliation features could also have similar metadata.

[thadguidry]

The specs should standardize the score range. I'd avoid negative numbers completely and standardize on [0..100] which is easy enough and wide enough for representing "a percentage of how closely the candidates match the query against this service "

[Abbe98]

I do not have much of an opinion on the range itself but I'm not fond of the idea to have all clients handle different ranges.

[tfmorris]

Something to consider here would be how this works for a service which is built on something like Solr/Lucene which doesn't have a fixed scoring range (and scores aren't comparable across queries). What type of normalization would the service do and how would the user interpret the resulting score?

[ChristianeKlaes]

Unifying score ranges between reconciliation services would definitely improve further processing of reconciliation results.

Additionally, recon services should clearly state in their manifest/documentation

• how similarity is computed (similarity measure, similarity features)
• which similarity score leads to automatic matching, if enabled

[fsteeg]

Additionally, recon services should clearly state in their manifest/documentation

• how similarity is computed (similarity measure, similarity features)
• which similarity score leads to automatic matching, if enabled

For this part see also the discussion in #128.

[fsteeg]

We discussed this in today's meeting:

There seems to be consensus that a standardized score range from 0 to 100 would be useful. To address the problem of service backends without fixed scoring ranges mentioned by @tfmorris in #147 (comment), this could be made optional. This would also retain compatibility with existing services.

So we could add an optional field in the service manifest, like "standardizedScore": true, and specify that this means the score will be between 0 and 100. This would allow for clients to check if the service does use a standard range and process or display candidates accordingly.