Potential for custom SPARQL service implementation (like the Wikidata mwapi service)

[trueg]

As you know we are currently in the process of moving away from Blazegraph as the Wikidata query service backend. As part of that I would like to know if I understand the SPARQL parser correctly when I assume the following:

In order to add support for a custom SERVICE one would need to

• create a new subclass to MagicServiceQuery
• handle the new services's IRI in SparqlQleverVisitor::visit(Parser::ServiceGraphPatternContext*)
• extend QueryPlanner::GraphPatternPlanner
• create a new Operation to actually execute the custom service

This sounds quite complex already but am I missing a step?

(This is a theoretical mostly intended to fuel an argument against adding a custom service and in favour of query rewriting in a proxy)

[aindlq]

I'm not a qlever developer, but just want to say that an ability to have an easy way to implement SERVICE extensions would be a really great feature. Ideally, maybe even have them dynamically linked.

Query rewrite or in a way external federation engine sounds good in theory but very often perform sub-optimally in practice. I've been using some of the existing ones like the one described here https://link.springer.com/chapter/10.1007/978-3-319-69548-8_17 and here https://rdf4j.org/documentation/programming/federation/ . And it kind of works nice for very simple use cases. But as soon as you have something as simple as external full text search or external vector index search, then you run into many issues and pay the price with much higher latency, lower throughput and incomplete search results.

Text search is a good example, even something simple like: Give me all the buildings in Italy that have a mention "baroque" in linked text. Can't be executed with query rewrite/federation, because what would you do? Get 10k "barouque" results from external full text search and then join them with "Buildings in Italy", or do you get 10k "Buildings in Italy" and send them to full text search for filtering? Both options are extremely sub-optimal because the results that one get will be far from perfect. With in-engine SERVICE extension one can potentially have access to query plan and result estimates that can help with optimization, as well as can properly handle LIMIT, etc.

[sueun-dev]

Your understanding is close, but there are a couple of additional integration points.

For a new magic SERVICE type, the full path is usually:

1. Add a new parsed-query type (typically a MagicServiceQuery subclass) with addParameter + validate.
2. Add/route a service IRI in MagicServiceIriConstants.h and SparqlQleverVisitor::visit(ServiceGraphPatternContext*) so it dispatches via visitMagicServiceQuery<YourType>(...).
3. Add the new type to GraphPatternOperation's variant list.
4. Extend QueryPlanner::GraphPatternPlanner::graphPatternOperationVisitor(...) (and often a dedicated visitX) to translate parsed config into an executable plan node.
5. Implement or wire the execution Operation used by that planner branch.

So yes, it is non-trivial, and your proxy-rewrite argument is technically reasonable for many cases.

[trueg]

Thank you for the explanation. :)

[hannahbast]

@trueg Sorry I missed this. I just stumbled upon https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Migration/label_and_mwapi_service_replacement_evaluation, where it says

This patch will most likely become outdated regularly, requiring potentially complex rewrites and fixes whenever Qlever internals change, even slightly

I don't understand this. Whatever is implemented in QLever becomes part of the master and our continuous integration, exactly so that it does not become outdated. We should discuss how much sense this makes though. In my opinion, there are better options.

For the label service, I would recommend using standard SPARQL, and we have already done that for Scholia, see also https://phabricator.wikimedia.org/T418595

For the MediaWiki API, it seems to me that this could be made into a standard SERVICE relatively easily. You just have to modify it to accept SPARQL requests (which are all of a very similar kind) and return one of the standard media types like application/sparql-results+json). Alternatively, the data from MediaWiki could be made available under its own endpoint, as we already doing on https://qlever.dev/wikimedia-commons; see https://github.com/qlever-dev/qlever-control/blob/main/src/qlever/Qleverfiles/Qleverfile.wikimedia-commons

My strong impression is that all this non-standard stuff in Blazegraph were hacks to work around inefficiencies in Blazegraph. They are not needed anymore. And when a hack is not needed anymore, adhering to standards is much preferable I think

[trueg]

Hi @hannahbast , thank you for the reply. I agree with you fully. The wiki page you are referring to lists possible solutions to the problem, then lists their potential problems and shortcomings. One of the (bad) solutions would be custom SERVICE extensions in Qlever that are maintained as patches (since they are way too specific to go into vanilla Qlever). We will of course not do this. As stated in the wiki page the better options are to rewrite label service requests to standard SPARQL (we do that in our proxy), and to use a separate service for the MWAPI requests, exactly like you describe it.
I was under the impression that the wiki page made clear which solutions we prefer, maybe that is not the case. Please let me know, then I will try to improve it.
As for your comment on Blazegraph hacks: yes.