Support for SPARQL 1.1 Federated Query (SERVICE)

[AndreaWesterinen]

Is there any timeline for federated query support?

[hannahbast]

@AndreaWesterinen Getting federated queries to work with QLever is high on our list. After all, the easy interoperability between different datasets is one of the hallmarks of RDF. For example, two large knowledge graphs, which are predestined to be combined via federated search are Wikidata (17B triples) and OpenStreetMap (10B triples). There are three stages:

1. The most basic form of support is as follows: (1) Incorporate the SERVICE syntax into the SPARQL parser; (2) during query execution send the SERVICE query to the respective endpoint; (3) parse the result into QLever's internal representation. Items (1) and (2) are simple. Item (2) is equivalent to parsing a VALUES clause, which is already implemented in QLever, except that IRIs that are not part of the knowledge graph are currently ignored. (1) + (2) + (3) are relatively easy to do ("just work") but this solution has two performance problems; see the next two stages. We suspect that currently existing implementations of SERVICE queries have the same performance problems, so that already Stage 1 would yield an implementation that is on par with the state of the art.

2. If the result returned by the remote endpoint is relatively small, the result can be sent and parsed quickly. Specifically, QLever can parse over 1 million triples (which roughly corresponds to 100 MB in Turtle format) per second. The limiting factor is then either this or the network bandwidth. This is a problem shared by all currently existing implementations of SERVICE queries that we are aware of. To alleviate this problem, if both endpoints run on Qlever, the result could be sent in binary format, using QLever's internal (fixed-size) IDs instead of IRIs and literals. This requires an intermediate service that translates IDs from one internal vocabulary to another. However, that translation would have to be done only once for each pair of endpoints, and not once per query. With an efficient implementation and a high-bandwidth network connection, this solution has the potential to realize federated queries with a performance similar to the ideal performance when the two datasets had been merged together in one endpoint.

3. There is one other potential performance problem, which, however, will probably be rarely encountered in practice. Namely, when the remote endpoint returns IRIs or literals that are not part of the vocabulary of the endpoint that issued the query, these IRIs or literals will receive special IDs that are linked to out-of-vocabulary strings. Such strings can become part of a SPARQL query result (and typically do for federated queries). However, QLever currently does not implement JOIN operations on such strings. The reason is that SPARQL queries requiring this rarely occur in practice and even more rarey for federated queries (JOIN operation on entities from the remote endpoint are naturally part of the SPARQL query executed on the remote endpoint). Of course, to fulfill the SPARQL 1.1 standard, JOIN operations on out-of-vocabulary entities should be implemented, and there is also no problem doing so. But they would be significantly slower than JOIN operations on in-vocabulary entities.

[WolfgangFahl]

Any progress on this? See https://phabricator.wikimedia.org/T197530#8884556 where i am asking the WDQS service to allow federated queries against QLever

[hannahbast]

@WolfgangFahl A first version of SPARQL 1.1 Federated Query (SERVICE) is implemented since #793, that is, for over half a year already. On https://qlever.cs.uni-freiburg.de you find many example queries using it. For example:

https://qlever.cs.uni-freiburg.de/wikidata/r7KAiM (movies and their IMDb rating, SERVICE to qlever:imdb)
https://qlever.cs.uni-freiburg.de/osm-planet/pgQkPa (Spanish-speaking countries, SERVICE to qlever:wikidata)
https://qlever.cs.uni-freiburg.de/wikimedia-commons/fwdZ1M (WC pictures of humminbirds, SERVICE to qlever:wikidata)

The basic functionality is all there and it works fine. Missing features that would be very easy to add are:

1. SILENT not yet supported (the semantics is: ignore the SERVICE clause when the respective query fails)
2. A variable endpoint is not yet supported, it has to be a fixed IRI (which is the typical use case)
3. There is no timeout yet for the SERVICE query

The entities returned by the remote endpoint will usually not be part of the vocabulary of the endpoint that launched the query. For values (like "42"^^^xsd:int) that is not a problem at all. New IRIs or literals will be added to the so-called local vocabulary of the endpoint. The following restrictions currently apply. We will fix them eventually, but that requires a bit more work.

4. When the query surrounding the SERVICE query also adds IRIs or literals to the local vocabulary (which a typical query does not do), it will not work. There will be an error of the sort that QLever does not yet support the merging of two non-empty local vocabularies
5. When the ORDER BY columns contain IRIs from the local vocabulary, the placement of those IRIs or literals will likely be incorrect. That is usually not an issue though, because the IRIs or labels from the remote endpoint are typically just "payload"

Last but not least, there is the efficiency aspect. The current implementation is as efficient as any other SERVICE implementation out there. Eventually, we want to make it much more efficient in two respects:

6. When the query surrounding the SERVICE query produces a small variable binding, that binding should be passed to the remote endpoint (in the form of a VALUES clause). That would be easy to add
7. Currently, the result from the remote endpoint is requested as TSV and parsed into QLever's internal representation. Eventually, we want to enable two QLever endpoints to exchange internal QLever IDs. This would speed up SERVICE queries that involve very large results by serveral orders of magnitude

[hannahbast]

PS: Your request on https://phabricator.wikimedia.org/T197530#8884556 does not require SERVICE capabilities on behalf of QLever. It just requires that QLever implements the standard SPARQL API, which it does for a long time already.

However, note that the IRIs in your example query are wrong, they should be <https://qlever.cs.uni-freiburg.de/api/dblp> and <https://qlever.cs.uni-freiburg.de/api/wikidata> . Without the api, it's just the URIs of the webpage.

EDIT: There are now also aliases <https://qlever.cs.uni-freiburg.de/<name>/sparql> for <https://qlever.cs.uni-freiburg.de/api/<name> because SPARQL enpoint IRIs usually end in /sparql.

[WolfgangFahl]

Thx- i have updated the phabricator task and reopened it.

[WolfgangFahl]

There is a followup problem in https://phabricator.wikimedia.org/T339347
_
Blazegraph is submitting (afaict) Accept: application/sparql-results+xml to qlever as part of the federated query
qlever is responding that it doesn't know how to respond in that format.
Blazegraph knows how to write application/sparql-results+json for normal api responses, but I'm not sure if it can read that format or how to tell it to use that here
_

• could you please look into it?