Improve communication of service status to users

[annakrystalli]

I feel we need good communication channels set up with service maintainers to be able to quickly figure out whether problems are from bugs in the package or service issues.

We should discuss at the next meeting.

I think at the very least a list of contact details for each service is necessary but would be interesting to explore whether there are better options.

[salvafern]

I'll talk to the coordinator of EMODnet Biology today. This is something we can try to do via EMODnet Central Portal (e.g. an alert when a service change etc).

I added some time ago an automatic test to check every single service. So in case that one of the service endpoints change we would be alerted in at least one day thanks to the CD/CI magic. But we don't have such a thing for individual layers. Maybe we want to also check the layers that we use in examples, vignettes etc?

[JoBeja]

The EMODnet webservices are being monitored centrally through https://monitor.emodnet.eu/ and there is a list of all webservices in https://emodnet.ec.europa.eu/en/data
These two pages are always up to date as they are part of the EMODnet Central monitoring procedures. So, it's not necessary to have a list of contacts, but come up with a way for the R package to have access to the monitor tool info to ascertain whether the services are operational or not. There is also no need to notify the thematic lots as this is done via the Central Portal monitoring procedures (VLIZ takes care of this via JIRA and email).

I have suggested to Salva that he contacts Tim Collart as he might be able to provide some suggestions on how to best implement an up to date list of the EMODnet webservices, without needing manual intervention.

Anna, I'm happy to provide more detailed explanations if required.

Hope this helps.

[salvafern]

Already looking at using the REST API from https://monitor.emodnet.eu/ to retrieve the available services as JSON (and also tells if they are down!). But it is giving some problems, I'll keep you posted.

@tim-collart Do you have any suggestions about how can we retrieve the OGC services of EMODnet in a machine-to-machine way? Can we do any web scrapping from https://emodnet.ec.europa.eu/en/data or rather try to persue https://monitor.emodnet.eu/? What of those two is more up-to-date?

[annakrystalli]

Thanks for the information @JoBeja! I agree that automatically checking the service through the packages would be ideal but being able to point users to the monitoring site in the meantime as a first place to troubleshoot errors will also be very useful. I'll add a note to the README.

[annakrystalli]

One observation though, for context for where this all started also.

A user opened an issue on Friday (#23) regarding calls to service geology_seabed_substrate_maps failing (using EMODnetWFS::emodnet_get_layers(service = "geology_seabed_substrate_maps", layers = "seabed_substrate_1m", reduce_layers = T). This lasted intermittently till yesterday afternoon and we've been able to access data successfully since.

I'm now looking on the monitoring site you shared @JoBeja and I can't seem to find any information on the service the user was trying to access (https://drive.emodnet-geology.eu/geoserver/gtk/wfs). So sadly it wouldn't have been helpful for the user nor myself as I was trying to debug the issue yesterday (perhaps there is an obvious reason for this relating to the changes going on that I am not fully aware of @salvafern ?).

It would be good to use the issue we encountered yesterday as a case study to develop our strategy. Hopefully there will be a machine-to-machine way to perform the necessary checks on the services so we have good ways of separating service errors from package bugs.

[JoBeja]

OK, I see. So, from my understanding the monitoring tool checks the Services, not the individual layers, which is what the substrates maps is, right?
Maybe if you manage to code in the status of a service and something fails, you can direct users to the monitoring tool first before they create a ticket in Github? It might be that there's nothing wrong with the R package but with the actual webservice.
I wonder if when that layer was failing for geology if the WFS item was showing a red colour. This is what is currently happening with Chemistry for one of their WFS services, not sure if the same applies with a "faulty" layer within the same service though.

Also, I'm not sure if @tim-collart will have any idea on how you can query the individual services in a way that does not require manual intervention (so that the list of webservices is updated if one endpoint is removed or a new one added).

I think (and hope) that Tim might be able to give some suggestions, this is way beyond my expertise.

[annakrystalli]

So from my understanding, https://drive.emodnet-geology.eu/geoserver/gtk/wfs is the geology_seabed_substrate_maps service endpoint and in the call the user was trying to make, seabed_substrate_1m was the layer. So it maybe that the Geology service has multiple endpoints? Indeed in our README we list access to 17 service endpoints while the monitoring tool only lists 10 WFS services. There may well be something I'm misunderstanding here about the structure of the services but would be really great to clarify if I am (perhaps @salvafern has a better grasp?)

But indeed, the key for us (and users) is to be able to efficiently diagnose whether a problem we/they are having is a problem with the server or the package.

Hopefully Tim might be able to advise!

[salvafern]

You are right @annakrystalli, seems to me that the services in the monitor tool are not all the services available. According to the docs, the services are added manually, so I think this is not the most updated list of the service endpoints (although it gives some interesting data about their current status)

Not sure if the maintainers of each service have to notify the EMODnet Central Portal team whenever a service is updated. Probably Tim can say the best way to get an updated list of the services.

[bart-v]

There is no need to do this within an R package or similar.
The service at monitor.emodnet.eu already notifies the maintainers when a service is down.
This (+the communication with the maintainers) is the responsibility of the EMODnet Central portal.

If services are missing, we are happy to add them.

I propose you close this issue.

[annakrystalli]

Hi @bart-v,

As I mentioned in my comments above,this is about communication with users not the maintainers and it arose because users of the package opened an issue here when the problem was with the service.

So all we are trying to do in this issue is decide on a strategy to avoid such situations in future and ensure issues opened here (and effort to debug them) is exclusively regarding the package.

We are currently just discussing options.

[bart-v]

That is not what the issue title says...

[annakrystalli]

The topic evolved from the first comment. I've ammended the title.

[bart-v]

Good, thanks
Querying the status from the monitor tool, before the actual WFS call (or after a failed request), looks like a good way of informing the user of the status of the server.

[annakrystalli]

I've added a note in the README for now:

EMODnetWFS/README.md

Lines 69 to 70 in 58352d3

	If you experience problems, please consult the [EMODnet OGC monitor
	](https://monitor.emodnet.eu/resources?lang=en&resource_type=OGC%3AWFS) to check the status of services prior to opening issues in the package.

directing users to the OGC monitor.

As mentioned though, there are endpoints that are currently missing (e.g https://drive.emodnet-geology.eu/geoserver/gtk/wfs which is the service the user was trying to access). I can't imagine I have the authority to add them (I imagine maintainers need to add them so the correct contact emails are associated with them?). Or is this something you can help with @bart-v ? As mentioned in a comment above, in our README we list 17 service endpoints while the monitoring tool only lists 10 WFS services.

[tim-collart]

Hi all,

A number of issues/questions raised:

Listing all the EMODnet Services:
The OGC/INSPIRE intended way, which i feel we should follow, to list all services available is to get it from a CSW discovery service, which should have a record describing each service with an ISO 19119 metadata XML document. For EMODnet, such a listing could be retrieved by doing a type=service filter on the Central Portal GeoNetwork CSW and parsing the XML returned. While most services are described there (see https://emodnet.ec.europa.eu/geonetwork/emodnet/eng/catalog.search#/search?facet.q=type%2Fservice) not all are there yet. This is because

1. not all portals have yet created a (valid) ISO 19119 metadata for each of their services (work in progress)
2. the central portal GeoNetwork is harvesting the service records (@bart-v can tell you how often this is done) from the Thematic portal CSW's. But not all portal have a CSW at this moment (work in progress) and some of them are just hosting the ISO19119 document on an http server (thus it is not currently harvested by the Central Portal GeoNetwork). @bart-v, I think GeoNetwork has the option to harvest metadata records from a file hosted on an http server? Maybe we can look into setting this up to get the services listed as complete as we can?

For those reasons, for now we are listing all services on a github page (https://github.com/EMODnet/Web-Service-Documentation) which we have asked the thematic portals to keep updated by issuing pull requests (i.e. manually). But the goal in the end is to have all service metadata described in the Central Portal GeoNetwork, which should be the way to get the metadata on them. However, at the end of the day, in case services change (e.g. URL endpoint), the updating of the metadata document, which will then be harvested by the Central Portal GeoNetwork and available in the CSW, will remain a manual process for the Thematic Portals. A good approach to make sure what is reported in the metadata is accurate, would then be to read the actual service endpoints monitored by monitor.emodnet.eu from the Central Portal CSW as well.

Checking for uptime of the services:
Indeed, monitor.emodnet.eu is currently only checking if GetCapabilities document are returned without error/exceptions. While one strategy would be to use the monitor.emodnet.eu tool to report any service issues to users, I don't feel this is a good design for a number of reasons:

1. your R package will become dependent on an additional service
2. the information in the tool is only updated every 10 min (which is not frequent enough for your use case)
3. as mentioned above, the services are not checked at layer/feature level.

As such, I would propose that for your use case (alerting users in case of service issues) to do the error checking of the service within the R code itself, so when a users runs a query on an EMODnet service, the issue will be detected in real time on the client side, and reported appropriately to the user. This is what we are doing for the European Atlas of the Seas (for the EMODnet WMS services). At first glance, I would say you can do
i) check the HTTP status code
ii) if an exception is returned, you can echo the OWS exception reported by the service.
As it's a widely used standard, perhaps this kind of OWS exception handling is already implemented in an R package somewhere, which you can use (maybe check : https://cran.r-project.org/web/packages/ows4R/index.html)

Sorry for the long post but hope this helps. Happy to answer any questions

PS: One more confusion I want to clear up. The number of services listed on https://github.com/EMODnet/Web-Service-Documentation is different from the ones listed https://monitor.emodnet.eu/. This is because several portals have split their services over multiple GeoServer workspaces, which are listed separately on the github page (they cover different kinds of data), but they are essentially the same server, therefore they are only listed once in monitor.emodnet.eu.

[annakrystalli]

Thanks so much for the detailed info @tim-collart . Very helpful.

[bart-v]

Answers to @tim-collart questions

• EMODnet Central Portal GeoNetwork is harvesting on a daily basis, more or less
• Yes, harvesting or uploading a static XML file (on a HTTP server) is possible

Agree largely with user notification proposal by @tim-collart

The monitor tool can certainly be expanded to do more, i.e. check availability of layers, inspect correct WFS responses etc.
But this needs to be discussed first, internally within the EMODnet CP team.

[salvafern]

Had a chat with @annakrystalli today:

@annakrystalli will work on #26

• @salvafern: Get to know of any mailing lists of EMODnet to know when new services are added or edited (@tim-collart and @bart-v probably you know more about this?)
• @salvafern or @annakrystalli: Move test-services to a separate GitHub Action for now. Eventually we may remove them. For now we don't want them to make the R-CMD-Check to fail
• @salvafern: Add function to handled errors in case that a WFS call does not work: Check if there is internet, then if the monitor tool is working, and finally return the status of the WFS service to the user in case this is down. E.g. add around here:

  EMODnetWFS/R/layers.R

  Line 78 in 58352d3

  error = function(e) {

[salvafern]

Hi @tim-collart and @bart-v , do you know if there is any mail list of EMODnet that could we use to know when new services are added or edited?