updated baseline check

[timadriaens]

@adrienlatli checked the updated baseline (which now looks good for the crayfish spp.):

• some points have disappeared from the last baseline (in red on the map) (+/-20) only in Flanders. It 's mainly plant observations like ludwigia or hydrocotyle. These need to be checked why they disappeared from the baseline.
• new observations of this year (2021) have been added. These can be visible on the map but need to be separated from the baseline as this only takes into account the observations made until December 31, 2020.

For Wallonia the baseline is OK and validated.
@SanderDevisscher can we check and solve these issues?

[adrienlatli]

Hello Sander,

Tell me how i can help you for solving this problem.
Previously, I compared with Arcgis the last baseline send by Tim (in red on the map) with the new one (purple).
Some points disappeared. Do you have the same issue if you create the map in R without XY transformation ?

[SanderDevisscher]

• new observations of this year (2021) have been added. These can be visible on the map but need to be separated from the baseline as this only takes into account the observations made until December 31, 2020.

And thus be excluded from the export (csv/shp) ?

[SanderDevisscher]

Hello Sander,

Tell me how i can help you for solving this problem.
Previously, I compared with Arcgis the last baseline send by Tim (in red on the map) with the new one (purple).
Some points disappeared. Do you have the same issue if you create the map in R without XY transformation ?

@adrienlatli Can you somehow identify the missing records ? By ID perhaps ?

[adrienlatli]

Hello Sander,

Find below a datafiles with observations that have disappear in the new baseline (essentially VMM observations) :
Missing_point_in_new_baseline.txt

And find below another data file with the most recent observations that were missing in the last baseline:
Adding_point_in_new_baseline.txt
We found successfully the procambarus sp. observations and plant observations from Natagora in 2021 and the end of 2020.

For the baseline, missing observations from VMM should appear, also with observations from the other data owners, up to 2020. But observations after December 31, 2020 should be deleted.

[SanderDevisscher]

Find below a datafiles with observations that have disappear in the new baseline (essentially VMM observations) :
Missing_point_in_new_baseline.txt

I'll look into what went wrong with this data.

And find below another data file with the most recent observations that were missing in the last baseline:
Adding_point_in_new_baseline.txt
We found successfully the procambarus sp. observations and plant observations from Natagora in 2021 and the end of 2020.

This is logical since I did not receive any new Walloon data besides the data that's present on gbif. Can/may I consider this as a part of the Demna dataflow ? or should I create a new dataflow for Natagora data ?

For the baseline, missing observations from VMM should appear, also with observations from the other data owners, up to 2020. But observations after December 31, 2020 should be deleted.

Since I also need the data for the website https://iasspeciesdashboard.shinyapps.io/Riparias-baseline-maps/ I'll have to look into were I can implement this filter.

[SanderDevisscher]

I've Identified the step in the flow where the error does occur. I'll see what goes wrong and try to fix it.

[SanderDevisscher]

I've fixed the cause for the missing data but I'm waiting for a response whether or not to include the natagora data as demna or as independent to complete the update.

[adrienlatli]

This is logical since I did not receive any new Walloon data besides the data that's present on gbif. Can/may I consider this as a part of the Demna dataflow ? or should I create a new dataflow for Natagora data ?

In my opinion, you have to create a new dataflow for natagora data but only for observations in Flanders. Observations made by Natagora in Wallonia will appear in the DEMNA database when we make data aggregation.

[SanderDevisscher]

The Natagora data is all present on gbif so they should be present in the baseline.

[timadriaens]

@adrienlatli noticed lacking crayfish data for Faxonius limosus in the baseline with more data available on waarnemingen.be for the period considered (1/1/2010-1/1/2020) https://waarnemingen.be/species/8887/maps/?start_date=2010-01-01&interval=15552000&end_date=2020-01-01&map_type=grid10k

Can we do a quick check to see what went wrong here? Do we need to update the gbif data extraction @SanderDevisscher?

[SanderDevisscher]

@tim Tim Adriaens ***@***.***> I used approved data only. Does the difference appear mostly with older approved observations? Or more with newer (non) approved observations ? Op do 24 feb. 2022 12:24 schreef Tim Adriaens ***@***.***>:

…

@adrienlatli <https://github.com/adrienlatli> noticed lacking crayfish data for Faxonius limosus in the baseline with more data available on waarnemingen.be for the period considered (1/1/2010-1/1/2020) https://waarnemingen.be/species/8887/maps/?start_date=2010-01-01&interval=15552000&end_date=2020-01-01&map_type=grid10k [image: image] <https://user-images.githubusercontent.com/16116743/155515031-a710cb42-928c-42f0-b5c7-b146d36e62bb.png> Can we do a quick check to see what went wrong here? Do we need to update the gbif data extraction @SanderDevisscher <https://github.com/SanderDevisscher>? — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAAQOBMZUTSVBKI64ZNGZLU4YIQBANCNFSM5BADCBBA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[adrienlatli]

Hello Sander,

The link with the validated observations:
https://waarnemingen.be/species/8887/maps/?start_date=2010-01-01&interval=15552000&end_date=2020-01-01&only_approved=on&map_type=grid10k

We have less observations than previously but many of them are in the Riparias area but not in the baseline.

[SanderDevisscher]

@adrienlatli can you confirm these observation are included in this waarnemingen.be dataset on gbif ?

ifso some gbifid's to troubleshoot would be appreciated

[adrienlatli]

@SanderDevisscher
I'have not checked all the crayfish data, but I found at least 1 observation of faxonius limosus included in the 2 datasets (waarnemingen dataset : https://waarnemingen.be/observation/173441526/ and in the gbif "waarnemingen dataset": https://www.gbif.org/occurrence/2274200746).

[SanderDevisscher]

@adrienlatli thanks for the examples. I'll look into what goes wrong.

[SanderDevisscher]

I found the suspected cause of the missing Faxonius limosus observations. Somehow the species was not included in the gbif download and thus no waarnemingen.be data was included. I'll look into a fix.

[SanderDevisscher]

@adrienlatli Orconectes limosus was included in the download but the waarnemingen.be data uses Faxonius limosus and was thus omited. I'll change the download to include both taxa.

[adrienlatli]

@SanderDevisscher : Thanks a lot for your work.
Just a small question, for the GBIF download you automatically did not take into account all the names (synonyms and basionyms) of each taxa ?
For examples, orconectes limosus = faxonius limosus = astacus limosus...

[SanderDevisscher]

Just a small question, for the GBIF download you automatically did not take into account all the names (synonyms and basionyms) of each taxa ?
For examples, orconectes limosus = faxonius limosus = astacus limosus...

It does download synonyms when you download the accepted species, in this case Faxonius limosus. The download considers all (including orconectes & astacus) as Faxonius. When you download only the synonym, like we did, the download gets limited to that synonym.

[timadriaens]

the usual hassle with taxonomic matching to the gbif backbone. It will be so much easier when we have published the RIPARIAS target species checklist and the Union List as checklists on gbif (work in progress).

[SanderDevisscher]

@adrienlatli is this more in line with what it should be ?

[SanderDevisscher]

I checked and the example you provided is now present in the baseline

[adrienlatli]

Looks great Sander. It 's ok for me :-)