support inaturalist taxon and occurrence dwca's

[jhpoelen]

@kueda suggested to use dwca instead of inaturalist paging api to link to inaturalist interaction data.

[jhpoelen]

I prepared how the DwC-A resource relations extension would link an observation with a taxon using a iNaturalist example at https://github.com/jhpoelen/inaturalist-dwca/ . I hope we end up using the newly accepted (?) http://rs.tdwg.org/dwc/terms/relationshipOfResourceID discussed in tdwg/dwc#186 , to make the interaction type mapping a little more robust and so that other projects can benefit from the work.

fyi @peterdesmet @baskaufs @qgroom @LienReyserhove

PS Is so happens that Field Museum folks are also thinking of capturing species interaction using DwC-A @magpiedin .

[kueda]

Ok, sounds like the DwC-A we prepared for you is working though, right? And thus no more action needed on our part? We can model these relationships in a different way if someone wants to provide an example of what that looks like, including actual files in the archive, what the controlled vocabulary is, etc., but that seems like future work pending acceptance of http://rs.tdwg.org/dwc/terms/relationshipOfResourceID

[jhpoelen]

I'd rather be closer to the Resource Relationships extensions than using the custom iNaturalist extension you proposed for many reasons (e.g., code maintenance, data integration quality, stimulating standards use/development). You might actually reduce the amount of DwC-A you generate by adding the resource relationship table to the DwC-A endpoint you registered with GBIF.

The term http://rs.tdwg.org/dwc/terms/relationshipOfResourceID has been accepted, just hasn't been added yet. Also, I imagine you can re-used your own controlled vocabulary in the form of the existing "observation_fields", so no need to worry about inventing some new vocabulary. DwC simply suggests to use a controlled vocabulary, but does not mandate a specific one.

The example https://github.com/jhpoelen/inaturalist-dwca/ provides a worked out example of what this would look like.

[kueda]

Doesn't https://github.com/jhpoelen/inaturalist-dwca/blob/master/resourcerelationship.csv imply that the observation http://www.inaturalist.org/observations/2309983 is eaten by the taxon http://www.inaturalist.org/taxa/133061? This is why I wanted an example, b/c what the resource is supposed to be wasn't clear to me from https://dwc.tdwg.org/terms/#resourcerelationship. I understand how the example you provided would meet your specific need, but it seems like a person familiar with the ResourceRelationship docs but unfamiliar with iNat might look at this and think it describes a situation where an otter ate an observation (stole a phone?), and not a situation where an otter ate an organism the observation depicts.

[jhpoelen]

In DwC, the resource relationships are between resources. And "Resources can be thought of as identifiable records or instances of classes and may include, but need not be limited to dwc:Occurrence, dwc:Organism, dwc:MaterialSample, dwc:Event, dwc:Location, dwc:GeologicalContext, dwc:Identification, or dwc:Taxon." So, the definition of a resource is pretty broad.

In the example, one resource, http://www.inaturalist.org/observations/2309983 , is the id of an occurrence that is defined in https://github.com/jhpoelen/inaturalist-dwca/blob/master/observations.csv as a human observation of Gorgonocephalus eucnemis . The other resource, http://www.inaturalist.org/taxa/133061 , the id of a taxon defined in https://github.com/jhpoelen/inaturalist-dwca/blob/master/taxa.csv with name Enhydra lutris kenyoni . Combined with the observation field id and label http://www.inaturalist.org/observation_fields/879 and "Eaten by" respectively, you'd get something along the lines of this Gorgonocephalus eucnemis was eaten by some Enhydra lutris kenyoni .

This approach is not specific to GloBI's needs. Other projects like uredinales-belgium-checklist https://trias-project.github.io/uredinales-belgium-checklist/ use a similar approach to link resources. This project involves some DwC contributors like @peterdesmet , so I figured this would be a good example to follow. Perhaps he can better address your concerns.

[qgroom]

I can't think of any negative implications of making this change and I can see many advantages to using DwC. It would also encourage DwC to be more supportive of interactions data.

[jhpoelen]

@qgroom thanks for sharing. I hope that iNaturalist can join other projects in pushing the standardization of sharing biotic interaction records forward.

[jhpoelen]

Hi @pleary !

Got your message, and I can see how paging through your API might not be the most efficient way to index observations that have interaction data annotated in observation fields.

Just wanted to get your thoughts on this unresolved issue from 2019 (#427) .

Looking forward to figuring this out together.
-jorrit

[pleary]

iNaturalist currently prepares https://www.inaturalist.org/observations/globi-observations-dwca.zip at the beginning of every month. It contains files similar to what you proposed in https://github.com/jhpoelen/inaturalist-dwca/, but in place of resourcerelationship.csv it contains observation_fields.csv with custom fields not defined in https://dwc.tdwg.org/terms/#resourcerelationship. It appears to me all the same data is there, just not in the ResourceRelationship format.

Would your preference be that we update https://www.inaturalist.org/observations/globi-observations-dwca.zip to replace observation_fields.csv with resourcerelationship.csv as in your example at https://github.com/jhpoelen/inaturalist-dwca/? Is that a change we could make now without being disruptive - i.e. can we remove observation_fields.csv in the next export without breaking anything on your end? In the mean time, how disruptive would it be for you if we disabled the observation_field_values endpoint?

Assuming some answers to the above, I'd propose these steps:

• we disable the observation_field_values endpoint
• we update globi-observations-dwca.zip to replace observation_fields.csv with resourcerelationship.csv in the next export
• Globi updates its sync process to get interactions data from globi-observations-dwca.zip using the resourcerelationship.csv format, and stops querying the observation_field_values API

Does that seem reasonable to you?

[jhpoelen]

@pleary thanks for taking the time to have a look at this open issue!

I was thinking - why not include the resource relations in to the DwC-A you expose to GBIF ? This way, you'll be able to get feedback from the GBIF folks and their nice validation tools. With this, you'd get two reviews for the price of one - GloBI reviews and GBIF reviews.

Any chance you can create a little dwc-a sample to play with? You'd also able to plug this dwc-a sample into the GBIF dwc-a validator? By starting small, we can focus in structure, then transition to scale.

Also, as far as transitioning goes - there's a versioning mechanism in place on my end, so you can just turn off the endpoint whenever you'd like.

Curious to hear your thoughts,

[jhpoelen]

Ideally, we'd first get some dwca integration setup, and then deprecate the endpoint. Like you, I like to reduce stress and development time as much as possible.

[jhpoelen]

Also, please note that I put effort into prototyping some dwca format some time ago. See https://github.com/jhpoelen/inaturalist-dwca . Is this what you had in mind?

[jhpoelen]

Hi @pleary!

I was just about to do some planning and I was wondering how you decided to move forward on implementing a more efficient way to share iNaturalist observation fields with GloBI (and other interested parties).

Hope all is well,
-jorrit

[jhpoelen]

hey @pleary - thanks for sharing your updated iNaturalist DwC-A that uses the Resource Relationships extension. Very neat to see that after the Field Museum (ping @magpiedin @rondlg ), you and @kueda are now also experimenting with this extension.

I am working on making sure that GloBI understand your resource relationships records. Looking pretty good as far as I can tell.

Two remaining questions that prevent me from completing the work,

1. in meta.xml, no taxon extension is defined, but taxa.csv is provided in the archive (see attached zip for truncated test resource at https://github.com/globalbioticinteractions/globalbioticinteractions/tree/312b87227db41da9f85400360aee1465b7001cad/eol-globi-data-sources/src/test/resources/org/globalbioticinteractions/dataset/inaturalist-resource-relationships). Without taxa.csv in meta.xml, the taxon information is "hidden" from the DwC universe. Was taxa.csv excluded from meta.xml by design?
2. in resource relationships table, a reference to occurrence ids starting with https appear (e.g., https://www.inaturalist.org/observations/131), however, in the observations table occurrence ids with plain http (not https) are seen (e.g., http://www.inaturalist.org/observations/131) . Is this intended?

[pleary]

What do you recommend for point 1 above? In this archive, observations/occurrence is the core and resourceRelationships is an extension. We have many observations of the same taxon, so we could make taxa an extension, but then we'd be repeating taxa information many times for their many observations. We also know that taxa IDs are the IDs referenced in resourceRelationships.relatedResourceID, but the star schema doesn't allow us to define that. Ideally we'd have a way to list taxa, once per taxon, and tell the consumer that these are the same taxa from observations.taxonID as well as resourceRelationships.relatedResourceID, but that's not possible with DwC-A.

We could fix the taxon duplication issue by changing the core to a Taxon core, but then observations/occurrence would be an extension and we couldn't include resourceRelationships since it would be an extension of an extension.

In short we can't with DwC-A define all relationships that would be useful with these data. What do you recommend we put in the meta.xml that would be most useful for you (ideally with repeating as little data as possible)?

[jhpoelen]

Thanks for taking the time to respond.

Yes, occurrence -> taxon relationships are a bit wonky in DwC-A.

And, a way to work with the model is to insert "proxy" occurrences. Then, link the proxy occurrence ids in the resource relations table.

This way of writing down the association makes sense to me, because in iNaturalist, these interaction observation fields link two observations: the primary observations and a secondary observations only annotated by their suspected classification.

Ideally, there'd be two observations for a single picture that describes an interaction . . . but for historic reasons, a single observation take the center stage, while other observations are tagged on in observation fields.

Would it be hard to insert the secondary observations (the object of the interaction) into the occurrence table somehow?

So, instead of:

primary occurence -[:interactsWith]-> secondary taxon

with

primary occurrence -[:classifiedAs] -> primary taxon

instead, we'd have a more explicit and, in my mind, intuitive description:

primary observation -[:interactsWith] -> secondary observation

where

primary observation -[:classifiedAs] -> primary taxon
secondary observation -[:classifiedAs] -> secondary taxon

Wasn't there some related discussion to link observations like this to make it easier to re-using existing identification verification workflows?

Curious to hear your thoughts.

[jhpoelen]

Note that I am not suggesting to change the existing iNaturalist UI or workflows, but simply to export the existing data into DwC-A in a way that better aligns with the current DwC models.

[pleary]

We could model the data this way for the export, but I don't like that it doesn't reflect how iNaturalist ever models these data with our current data model. Personally I'd prefer not to approach it that way.

If we were to use this approach, what values would you put in all the columns for the pseudo observation? We can't re-use the observation ID, so we'd have to construct a new one (that wouldn't be recognized anywhere in iNaturalist since technically it is all the same observation). We could probably reuse the observer info, dates, and location, but info about "identification" would be trickier - we maybe can use the creator and created_at on the observation_field_value. If you want us to try modeling things this way, it would be helpful to have an example with few real observations with the columns filled out, and a proposal for what IDs to use for the pseudo observations.

As for the additional taxa.csv file - in my opinion there is useful information in there for the data consumer, but it doesn't fit well within the DwC-A. That file contains information on all taxa in iNaturalist, even if they aren't directly associated with an observation or observation field. For example to recreate the iNaturalist taxonomy in its entirety we need to include ranks not represented by DwC, like section, subphylum, superclass, etc. The structure of the archive right now has occurrence as a core, and some basic taxon ranks are columns in that file. We can add taxa as an extension, but we'd need to include a coreID and then repeat taxa info for all observations that reference the same taxa. And then to include ranks that aren't associated with observations or resourceRelationships (lets say some species has a superclass as an parent [parentNameUsageID]), if we wanted to include them we'd need to include rows where the coreID is empty.

I see a few considerations for the taxa file - documenting it in meta.xml, making sure it contains all observation taxa as well as all taxa referenced by relatedResource, including other non-directly referenced taxa like ancestors needed to fully recreate the iNaturalist taxonomy, and ideally not repeating data.

One thing that is technically possible is to include taxa.csv as an extension, but leave the coreid index value empty. If we did that it would exist in the archive as an extension, with all columns mapped to term URIs, but not associated with anything as far as meta.xml is concerned. Consumers of the archive could be informed via documentation that the taxonID of that file maps to taxonID on occurrences, and also maps to relatedResourceID on resourceRelationships. From a data modeling perspective this is in my opinion closest to ideal - we can model the data just as we do internally, without needing to repeat anything, having the same taxa being referenced from the core and extension, and can include unreferenced ancestor taxa. We can't convey the relationships among the files in the meta.xml schema, so it doesn't play nicely with DwC-A. But I'd also argue that resourceRelationships as an extension already doesn't play nicely because there's no way to say what ID is used in resourceRelationship and it's up to the user to figure that out (currently it's a taxonID, but in your proposal it would be an observationID, and meta.xml doesn't allow us to declare that).

[jhpoelen]

Thanks for elaborating.

I agree that faking the observations by introducing proxy ids may be more confusing than informative.

Perhaps ditch DwC altogether and stick to the a format like that shown in:

https://github.com/globalbioticinteractions/template-dataset

here, a single table is used to represent an interaction claim. A similar representation is used to review iNaturalist interaction claims https://depot.globalbioticinteractions.org/reviews/globalbioticinteractions/inaturalist/ .

You can find some examples at: https://depot.globalbioticinteractions.org/reviews/globalbioticinteractions/inaturalist/indexed-interactions-sample.tsv .

Note that this sample already has the interaction types mapped. In providing the data from the iNaturalist side, I imagine that only iNaturalist speak is included: not RO terms, but the observation field type id/label.

The benefit of a single table is that you can stream it . . . .

Perhaps even simpler would be to generate a jsonl (see https://jsonlines.org) dump with same structure as produced by the "old" api. I can then simply fetch the jsonl dump every once in a while and you don't have to worry about the bot that retrieves page after page after page from the web api.

What do you say? Ditch DwC and go native? Or attempt to make DwC work?

[pleary]

Maybe my view is tainted a little by laziness, but I feel like what we have now is almost all the way there. Unless its much more work for you to support the archive I sent you a link to, I'd suggest we go with that with a few changes. If it is much more work for you, then it would be up to us to do the work to generate another format, like the single table view you linked to, that you can ingest automatically.

Here's a proposal for minimal number of changes from where the archive is now:

• we make sure URIs are presented consistently including protocol
• we document the taxa.csv file as an extension with a coreid with no index, which tells the users the term URIs associated with each column
• we add a statement to the abstract in the metadata.eml.xml that states there is a taxa.csv file included, but not directly associated with other files, informing users that taxonID is shared with observations.csv, and that all resourceRelationship.relatedResourceIDs can be mapped to a taxonID

It will still be a valid DwC-A. taxa.csv won't be connected in the traditional DwC-A way, but there is still some taxon information in the observations.csv, and users can get all the rest from taxa.csv if they want. If that seems acceptable to you, it would be far easier on our end to make these few changes than to change export formats entirely.

[jhpoelen]

hey @pleary thanks for sharing your notes, and apologies for the delay.

In my mind, your hybrid proposal implies that: iNaturalist produces separate two data products, one including observations with resource relationships pointing outside the scope of the DwC-A and the other including a version of the iNaturalist taxonomy.

Also, note that there's now at least two DwC-A products exposed to the world - one for GBIF and another for GloBI. I'd imagine that it'd take less maintenance to support only a single DwC-A product. And, the benefit would be that you'd have more eyes on the single product instead. I imagine that this increased focus would make improvement opportunities easier to spot.

So, to help reduce re-work and maintenance, I'd suggest to:

1. merge GBIF and GloBI DwC-A products
2. extend DwC-A with resource relationships
3. extend DwC-A with observations fields as measurementOrFact
4. use measurementOrFact as a bridge between the "core" (the observation in this case) and the observation field and specify the kind of annotation via the ResourceRelationship extension.

from "https://rs.gbif.org/extensions.html#http://rs.tdwg.org/dwc/terms/MeasurementOrFact" -

[...] Support for measurements or facts, allowing links to any type of Core. [...]

As iNaturalist is growing, I imagine that mobilizing the association iNaturalist data is key to translating various aspects captured knowledge into a broader science community. And, I hope we'll figure out a way to build a bridge that doesn't require too much maintenance. And, I hope that iNaturalist will continue to invest in working with other biodiversity infrastructure to help exercise our collective skill to manage, publish, archive, exchange, compile, and integrate increasingly complex and interesting biodiversity datasets.

Thanks for your time and patience in trying to figure this out. Curious to hear your insights.

[pleary]

I agree it would be good to eventually add more data to the archive we publish to GBIF, but I'd like to separate that functionality from the issue at hand which is using DwC-A for the data we're making available specifically for GloBI. Based on what we learn from generating data for you, we can extend the archive we publish GBIF, but I'd prefer to do that in the context of a separate issue, in the iNaturalist repo, that we can work on separately.

At this point we've removed the API you depend on, and I'd like to settle on a replacement sooner than later so you can continue to get new data. Any objections with the hybrid proposal I made to use for now? If you don't object to what I proposed, I'll implement those changes and set up the archive to be generated monthly.

Separately, if you'd like to create an issue in the iNaturalist Github repo for discussion extending the archive we publish to GBIF, feel free to do that.

[jhpoelen]

Hey @pleary thanks for taking the time to respond.

As mentioned above, I do have objections to your proposed hybrid approach. Rather than formulating them as objections, I chose to formulating them as suggestions:

1. merge GBIF and GloBI DwC-A products
2. extend DwC-A with resource relationships
3. extend DwC-A with observations fields as measurementOrFact
4. use measurementOrFact as a bridge between the "core" (the observation in this case) and the observation field and specify the kind of annotation via the ResourceRelationship extension.

[pleary]

1. I see this as out of scope for this issue. We would like to continue to generate a separate DwC-A for Globi for now
2. This is part of what I proposed, so sounds like we're in agreement there
3. This hasn't come up before, but we can switch from using our custom observation_fields type to measurementOrFact
4. This is a component of 3, right?

We won't be doing 1 right now, but can work on it if an issue is created in https://github.com/inaturalist/inaturalist . If we implement 2, 3, and 4, would that be sufficient to resolve this issue?

[jhpoelen]

@pleary thanks again for engaging in discussion.

Yep, having 2,3,4 would be great progress.

Argument for implementing 1. is twofold: first, get more eyes on a single product helps to tease out errors more quickly. The second one is more strategic - exposing resource relationships to the GBIF universe would create an excellent showcase of iNaturalist for others to follow.

Curious what you think about these argument. . . my gut tells me that we might save time and come up with healthier data integration. But hey, I might have been drinking too much coffee this morning.

Curious to see how you'll proceed.

[jhpoelen]

Addendum -

In your hybrid solution, you are suggesting to effectively publish two datasets - the occurrences, and the taxa. Would it be an idea to actually package them as separate DwC-As? That way, I can declare that the occurrence DwC-A depends on the taxon DwC-A, and like will be able to re-use existing tooling.

With this, there's no need to introduce any other layer like I proposed earlier (like measurementOrFact).

So, we'd have two endpoints -

dwca-globi-taxa.zip
dwca-globi-occurrences.zip

with dwca-globi-occurrences.zip -[:depends on] -> dwca-globi-taxa.zip

Ideally, the globi-taxa dwc-a would also include all the identifiers mappings to other taxonomies (e.g., itis, catalogue of life etc.)

[jhpoelen]

So, a revised list of suggestions is:

1. merge GBIF and GloBI DwC-A products
2. extend DwC-A with resource relationships where occurrence ids are related to taxon ids
3. extend DwC-A with observations fields as measurementOrFact
4. use measurementOrFact as a bridge between the "core" (the observation in this case) and the observation field and specify the kind of annotation via the ResourceRelationship extension.
5. publish taxa.csv in its own separate dwc-a (with meta.xml, eml.xml etc).

Where 1. would be implemented once 2. and 5. are working smoothly in iNaturalist <> GloBI for a while.

@pleary Thanks for being patient with me as I am trying to avoid rework or adding integration workflows.

[jhpoelen]

A first result using iNaturalist occurrence and taxa DwC-As yielded in over 490k indexes interactions.

See attached results after running

elton pull globalbioticinteractions/inaturalist
elton interactions globalbioticinteractions/inaturalist | pv -l | gzip > interactions.tsv.gz

due to upload constraints only the first 100k interaction claims were included in attached examples.

interactions-first100k.tsv.gz

note, however, that the current solution is putting much strain on the java heap. More work is needed to optimize memory usage of indexing this corpus of interaction claims.

[jhpoelen]

DwC-A based iNaturalist records appears to have been indexed by GloBI as of Oct 5 2023.

thanks to @kueda and @pleary for turning a 2019 idea into a 2023 reality.

via https://www.globalbioticinteractions.org/?accordingTo=globi%3Aglobalbioticinteractions%2Finaturalist&interactionType=ecologicallyRelatedTo

we found:

suggesting that https://www.inaturalist.org/observations/36909546 was successfully indexed via DwC-A instead of the custom integration.

Please do note that for some reason, fewer records were indexed (511k prior, now 496k) after migration to the DwC-A integration.

[jhpoelen]

Included are the inaturalist observation ids prior and after dwca integration switch

inaturalist-GloBI-indexed-observation-ids-2023-10-05.tsv.gz
inaturalist-GloBI-indexed-observation-ids-2023-10-05-dwca.tsv.gz

with

diff\
 <(cat inaturalist-GloBI-indexed-observation-ids-2023-10-05-dwca.tsv.gz | gunzip)\
 <(cat inaturalist-GloBI-indexed-observation-ids-2023-10-05.tsv.gz | gunzip)\
 | head -n20

yielding

3a4,6
> https://www.inaturalist.org/observations/10000096
> https://www.inaturalist.org/observations/10000097
> https://www.inaturalist.org/observations/10000098
8a12
> https://www.inaturalist.org/observations/100005785
11a16
> https://www.inaturalist.org/observations/1000067
18d22
< https://www.inaturalist.org/observations/100009361
20d23
< https://www.inaturalist.org/observations/10001043
35d37
< https://www.inaturalist.org/observations/100015102
38a41
> https://www.inaturalist.org/observations/100016584
43c46,47
< https://www.inaturalist.org/observations/100017880
---
> https://www.inaturalist.org/observations/1000180

[jhpoelen]

So, for some reason,

https://www.inaturalist.org/observations/10000096

was no longer indexed, while

https://www.inaturalist.org/observations/100017880

was added to the index, where it was previously not.

[jhpoelen]

@pleary @kueda Just wrote a little blog post on our use of the Resource Relationship extension. Let me know if you'd like me to make edits to improve / correct it -

Nov 3, 2023
Field Museum and iNaturalist Adopt Darwin Core Resource Relationship Standard to Share Species Interaction Records
The Field Museum in Chicago and iNaturalist capture detailed records on how species interact. They both showed their capacity to innovate by using the recently improved Darwin Core Resource Relationship extensions to publish their interaction records. By using this standards based approach, they facilitate access to the valueable biodiversity knowledge they keep, and provide examples for others to follow. More ...