Incorrect clustering of records

[gbif-portal]

Cluster, an experimental feature

We encountered an interesting new clustering feature in one of the Dutch databases. Two occurrences are clustered, but probably because one occurrence from Germany lacks data on date & location. See https://www.gbif.org/occurrence/3128544409/cluster
Why are these specimens clustered, and where can I found more information about this feature?

---

User: See in registry
System: Chrome 99.0.4844 / Windows 10.0.0
Referer: https://www.gbif.org/occurrence/3128544409/cluster
Window size: width 1536 - height 722
API log
Site log
System health at time of feedback: OPERATIONAL

[MortenHofft]

The process is described here

https://www.gbif.org/news/4U1dz8LygQvqIywiRIRpAU/new-data-clustering-feature-aims-to-improve-data-quality-and-reveal-cross-dataset-connections

https://data-blog.gbif.org/post/clustering-occurrences/

join the discussion here https://discourse.gbif.org/t/identifying-potentially-related-records-how-does-the-gbif-data-clustring-feature-work-gbif-data-blog/3083

sent per email

[ManonGros]

In this case, it is because the catalogue number of the SMF record (746) overlaps with the "other catalogue numbers" of the NMR record and the lack of country, date and coordinates in the SMF record makes it non-conflicting.

@timrobertson100 is there a way to specify that two records shouldn't be clustered?
Should the algorithm take into account the collection and institution codes/identifiers when provided? Or would it be an issue to detect herbarium duplicates across institutions?

[timrobertson100]

Thanks @ManonGros

It's tricky with sparsely populated records like this.

There's no current way to identify a specific record pair to ignore, but we could tighten the rules a bit - e.g. making it such that you need a match for either a date or location and not non_conflicting in both. That would mean to remove this rule.

What do you think?

[ManonGros]

thanks @timrobertson100, yes it might be a good idea to removed the non-conflicting date and location rule. Now that I see this example, I think this rule doesn't enough give evidence to infer the cluster.
Let me know if/when you remove it (I will update the blogpost accordingly).

[timrobertson100]

This has now been run in production and the records no longer cluster. This shows on the detail pages for the occurrences now, but it will take a little time before they disappear from the "is in cluster" search option. I've initiated a recrawl/reprocess for the two datasets involved, and that needs to complete for the search option to disappear.

Thanks for offering to update the blogpost @ManonGros - can you please do that and then close this issue?
I'm not sure if we'd want to state within the text that it's changed today, but if so, we could refer to this release note?

[ManonGros]

Thanks Tim! I updated the blogpost (I originally put the link to this issue but I just changed it to point to the release note)

[abubelinha]

Hi. I wonder if you prefer us to re-use this issue title for other incorrect clustering of records we found, or you prefer separated posts for this. I am posting them here for now (please mention me if you open a new issue, so I can track it).

• Plant occurrence 180425491 is matched to an animal (api link).
  For some reason, cluster info says "same accepted species".

• Occurrence 2821298442, contains "isInCluster":true (in the api link)
  But for some reason, I see nothing in the related occurrences web and api pages.

If those are expected behaviours I would like to understand the reasons.

Thanks a lot
@abubelinha

[ManonGros]

@timrobertson100 could you take a look?
I cannot explain why the occurrence 180425491 is in this cluster.

For the other issue mentioned above, perhaps it is time to run an update?