calculation of indicators > request for detailed documentation

[alitka]

Related to the comment by Davide Artaseni (2020/05/29) - #3670

We can provide a detailed documentation with the workflow and formula for the calculation of indicator.

We would appreciate, if you could provide us the concrete calculation formular of all indicators (the slides you have provided are not detailed enough). Only on this basis we are able to do a proper analyse of the provided (indicator) results this year, and as a next step to update and improve our metadata entries.

Thanks and best regards.

[dartasensi]

Dear @alitka,

I will now forward your request to the team, and I will keep you posted.

Kind regards,
Davide

[MarcoMinghini]

Dear @alitka,
can you better detail what information on the calculation of indicators (and which indicators in particular) you would need, in addition to the information already included in the COMMISSION IMPLEMENTING DECISION (EU) 2019/1372 and the presentation we attached to the Monitoring and Reporting page?

[alitka]

Dear @MarcoMinghini,
Please excuse my late repsonse.

To get a better understanding of the results we did some calculations on our side. In particular with regard to the indicator DSi2.x (Monitoring of the conformity of spatial data sets) and NSi4.x (Monitoring of the conformity of network services) we have identified significant deviations from your values.

The values as of December 20, 2020 were calculated using the following assumptions:

• for DSi2.x: all inspire-relevant datasets or series with a conformity-statement equal to this string: "VERORDNUNG (EG) Nr. 1089/2010 DER KOMMISSION vom 23. November 2010 zur Durchführung der Richtlinie 2007/2/EG des Europäischen Parlaments und des Rates hinsichtlich der Interoperabilität von Geodatensätzen und -diensten" AND in the same iso-element (gmd:report) the pass-element equal to the string "true"
• for NSi4.x: all inspire-relevant services with a conformity-statement equal to this string: "VERORDNUNG (EG) Nr. 976/2009 DER KOMMISSION vom 19. Oktober 2009 zur Durchführung der Richtlinie 2007/2/EG des Europäischen Parlaments und des Rates hinsichtlich der Netzdienste" AND in the same iso-element (gmd:report) the pass-element equal "true"

For a better understandig of these approach, I'll give an example:
NSi4.1 = Percentage of the discovery services that are in conformity with Commission Regulation (EC) No 976/2009 as regards the Network Services

• JRC value for NSi4.1 = 100%
• DE value for NSi4.1 = 0%

Therefore we would appreciate some insight into your calculation process.
Thanks in advance and kind regards.

[alitka]

Dear @MarcoMinghini,
Any news to our above observations?
Thanks in advance and kind regards.

[dartasensi]

@MarcoMinghini I think I will take it from here. :)

Dear @alitka,
May I ask you (as double check) to provide the fileIdentifier of the metadata describing the Discovery Service (from which that snippet is taken)?
So I will then go into our stored MR2020 harvest session and check your findings.

Apart from that, I can confirm that the formula you exposed for both the indicators are correct, and I will go further.
On the conformity statement section: since most of these sections usually contain errors (eg. copied and paste incorrectly, with lot of spaces and/or wrong characters, or in a different language from the md lang), we are currently checking for the numbered strings (ie. "1089" AND "2010", "976" AND "2009") in order to be as much as inclusive as possible.
On the degree of conformity section: if expressed correctly (as you highlighted) we stored the value as-is.

In the meantime, I will start checking your specific case as soon as possible.

KR,
Davide

[alitka]

Dear @dartasensi,
The fileIdentifier of the metadata describing the Discovery Service is = 7b88a292-cc04-0ae9-5d00-aaac167fe33c
And thanks for checking your findings regarding this specific case.
Kind regards.

[dartasensi]

Dear @alitka,

thanks again for raising this issue and providing the discovery service metadata.

It is indeed a bug, which only appears in a specific scenario and thus affects a small portion of the service metadata. When it occurs, the impact of this bug is always an over-estimation of the conformity (i.e. a false positive), so the opposite can never happen.
We added the task in our backlog and will proceed to the fix based. Updates will be provided in this discussion.

Kind regards,
Davide

[alitka]

Dear @dartasensi,

Thanks for your reply.

We are still struggling to reproduce your results. Even if we match the title with "1089" AND "2010" we get significantly different results. Also, this pattern matching approach introduces other kinds of errors. For example the title VERORDNUNG (EU) Nr. 1253/2013 DER KOMMISSION vom 21. Oktober 2013 zur Änderung der Verordnung (EU) Nr. 1089/2010 zur Durchführung der Richtlinie 2007/2/EG hinsichtlich der Interoperabilität von Geodatensätzen und -diensten would get matched and becomes a false positive.

However, we would like to double-check our results with your findings. For this reason we would appreciate, if you could provide us your raw data. Based on this we can easily comprehend, which records you have marked as true or false.

Alternative, if possible, could you share with us the (relevant) code snippet responsible filtering and calculating this specific result. Based on this we could easily compare your approach to ours and spot differences.

Thanks in advance and best regards.

[alitka]

Dear @dartasensi,
Just a little reminder for you, that we are still have an interest to compare your results with our findings.
Thanks and best regards!

[alitka]

Dear @dartasensi,
We would appreciate if you would answer us.
Thanks and best regards.

[dartasensi]

Dear @alitka

I would like to thank you again for the efforts spent in checking the MR2020 results.
As a result of this, we discovered this bug, and of course we are aware that it might have affected some indicators on the MR2020 run, even if in a "positive" way for all the Member States.
We would like to clarify that the bug consists in a wrong interpretation of the results available in that section, when a single metadata presents various conformities to different regulations. There is a now verified risk that the metadata would be recognized as conformant, and consequently counted for the indicator, if a positive degree of conformity (ie. <gmd:pass>[...]true[...]</gmd:pass>) is identified to at least one regulation.
Obviously, this bug affected all the MS indicators at the same level.

We are committed to solve this for the next upcoming MR2021.

Kind regards,
Davide on behalf of JRC INSPIRE Support team

[jrc-inspire]

As reported during the 67th MIG-T Meeting, the INSPIRE Geoportal is currently in a transition phase for the adoption of a fully-redesigned backend based on GeoNetwork. For this reason, we will limit the bug-fix/improvement actions on the current Geoportal to those issues having the highest impact and that can be addressed with the available resources.
However, we have taken note of the potential issue in the current Geoportal and will do our best for addressing it in time for the upcoming INSPIRE Monitoring, although we cannot ensure that this will be feasible.

[alitka]

In our opinion, this issue or bug isn't related to the INSPIRE Geoportal. As Davide wrote that the bug consists in a wrong interpretation of the results available in [the conformity statement] section [...]. Therefore, it has to be fixed for the MR2021, otherwise it will lead to incorrect results for the indicators DSi2.x and NSi4.x.

[alitka]

Unfortunately, our concerns about this issue are still present and are directly affecting the 2021 monitoring results.

Based on your calculation, the monitoring indicator DSi2 is 74% for Germany, but we are not able to reproduce this value. According to our own analysis, we get a value of 59.76% for DSi2.

We did some testing, and were able to reverse engineer your calculation method to match your results for DSi2. We would like to demonstrate this on a concrete example with the metadata record 0004ad7f-8a61-694b-0057-8720fd1fc5f2.zip:

title	date	pass
VERORDNUNG (EG) Nr. 1205/2008 DER KOMMISSION vom 3. Dezember 2008 zur Durchführung der Richtlinie 2007/2/EG des Europäischen Parlaments und des Rates hinsichtlich Metadaten	2008-12-04	true
VERORDNUNG (EU) Nr. 1253/2013 DER KOMMISSION vom 21. Oktober 2013 zur Änderung der Verordnung (EU) Nr. 1089/2010 zur Durchführung der Richtlinie 2007/2/EG hinsichtlich der Interoperabilität von Geodatensätzen und -diensten	2013-12-10	false
VERORDNUNG (EU) Nr. 102/2011 DER KOMMISSION vom 4. Februar 2011 zur Änderung der Verordnung (EU) Nr. 1089/2010 zur Durchführung der Richtlinie 2007/2/EG des Europäischen Parlaments und des Rates hinsichtlich der Interoperabilität von Geodatensätzen und -diensten	2011-02-05	false
VERORDNUNG (EG) Nr. 1089/2010 DER KOMMISSION vom 23. November 2010 zur Durchführung der Richtlinie 2007/2/EG des Europäischen Parlaments und des Rates hinsichtlich der Interoperabilität von Geodatensätzen und -diensten	2010-12-08	false

In this metadata record, there are 4 report blocks, but only the last block is relevant for DSi2. The elements title and pass are correctly specified. Using our calculation method, this dataset is NOT compliant, because the pass element is set to false in the relevant block. Thus, it does not contribute towards DSi2 and results to the above-mentioned 59.72%.

However, if we change our calculation method so that ANY pass element must be true per metadata record, disregarding a pass elements affiliation to a block, this dataset would become compliant and would count towards DSi2. With the changed calculation method, we score a DSi2 of 73.72%, which is much closer to your 74%.

We would be pleased to hear from you if the assumed calculation method is actually applied, and if so whether this is intentional or not.

[alitka]

Dear @MarcoMinghini and @dartasensi,
Any news to our above observations?
Thanks in advance and kind regards.

[jrc-inspire]

Dear @alitka,

apologies for the late reply.
Thanks a lot for your further investigation and time spent about this issue.
We can confirm this conformity bug, that you initially discovered and disclosed with us, is still present in the latest evaluation process of the MR2021.
During the overwhelming activity that hit our team between November and December, prior to the M&R deadline, this bug was fixed and scheduled for deploy. Unfortunately, we realized this might create a serious decrement for most of the member states' endpoints, therefore we decided to keep this "optimistic" estimation.
This estimation will be fixed surely in the next MR2022, when we will use the Geonetwork solution for the M&R process. Hopefully soon, we're planning to release this solution as part of the opensource Geonetwok release package, so you will be able to install it locally, check and fix eventual bugs.

Regards,
the JRC INSPIRE support team

[alitka]

Dear @jrc-inspire support team,
Could you elaborate on the release of the Geonetwork release package and what exactly it allows us to run and test locally?
Thanks and regards!

[jescriu]

Dear @alitka,

Apologies that we missed to replied you timely.

In case you might want to test it locally, the new Geonetwork package is publicly available in the official GeoNetwork opensource community project on GitHub:

• Harvester console components: https://github.com/geonetwork/geonetwork-remoteharvester
• Harvester console UI: https://github.com/geonetwork/core-geonetwork/tree/csw-remote-harvester ('csw-remote-harvester' branch)

I am closing this issue, since it was related to the previous version of the INSPIRE Geoportal (Classic INSPIRE Geoportal).