Use a skip scan based iterator for listing graph names in TDB2 (GH-1639)

[rvesse]

Adds a new BPTreeDistinctKeyPrefixIterator that allows iterating only records which are considered distinct based on a portion of their key. It is effectively a skip scan based iterator that can avoid reading portions of the B+Tree where all records share the same key prefix. This is used to improve performance of DatasetGraphTDB.listGraphNodes() and SolverLibTDB.graphNames(), in some cases dramatically so.

Used a couple of different test scenarios with calling listGraphNodes():

• 100k named graphs comprising a total of 125 million quads
  • On 4.6.1 this takes an average of 21,480 milliseconds
  • With this patch this is reduced to 18,235 milliseconds
• 10 named graphs comprising a total of 177 million quads
  • On 4.6.1 this takes an average of 27,856 milliseconds
  • With the previous patches (Use distinctAdjacent for TDB 2 graph iteration (GH-1639) #1641 and Improve "GRAPH ?g {}" #1642) this was reduced to an average of 25,185 milliseconds
  • With this new patch this is reduced to around 5 milliseconds 😄

This resolves #1639

[afs]

It is great to see this but it is now very close to 4.7.0.

[Aklakan]

Thanks for this work! I recently had to fight with postgresql because native skip scans are a missing issue there. The performance improvement from several seconds/minutes down to a few milliseconds is what I experienced there. Also, e.g. one of the postgres extensions greatly advertised their skip scan implementation in this blog post; this may be an inspiration for how to advertise this feature when its done.

[rvesse]

It is great to see this but it is now very close to 4.7.0.

Yeah I think this one will have to push out and be given more time to be refined and settled down. The change breaks a few tests so there's clearly some corner cases I am not catching correctly yet!

[SimonBin]

very nice work!!

SELECT ?e {
  GRAPH ?g {
    ?e spatial:withinBoxGeom ("POLYGON((19.49 50.62,26.87 50.626,26.87 46.43,19.49 46.43,19.49 50.62))"^^geo:wktLiteral 10)
  }
}

0.2 seconds

SELECT (count(distinct ?thing) as ?count)
WHERE {
    graph ?g {
        ?thing a/rdfs:subClassOf* coy:Powerplant .
    }
}

0.1 seconds

select distinct ?g { graph ?g {} } unchanged

[SimonBin]

The change breaks a few tests

didn't see any?

[rvesse]

The change breaks a few tests

didn't see any?

Already fixed them with my force push

[SimonBin]

the quick + dirty way to make select ?g { graph ?g {} } fast:

AKSW@2b064df?w=1#diff-e82707e0fbbe7ca027b63ecbac2d24d5607432e492061ff8d1cc3f5c6318e2feR162-R167

not sure what to do about the filter though.......

[Aklakan]

Imho there should also be some optimization for mapping the common query DISTINCT ?g { GRAPH ?g { ?s ?p ?o } } (and possibly common variants of it - such as counting distinct graph names) to effectively OpDatasetNames so that the skip scan can be leveraged further.

For example,

SELECT DISTINCT ?g { GRAPH ?g { ?s ?p ?o } }

should be evaluated efficiently w.r.t. the presence of possibly empty graphs as

SELECT ?g {
  GRAPH ?g { }
  FILTER EXISTS { GRAPH ?g { ?s ?p ?o } } # Note: Won't be expand infinitely because
                                          # we are not requesting DISTINCT ?g in the filter element
}

Of course this should then be managed as a follow-up issue to this one, but my questions right now are:

• Would it make sense to handle it as a query rewrite (alternatively it could be e.g. part of the OpExecutor but then the logic is less reusable)?
• Where would be the best place for that? Maybe a new optimizer under org.apache.jena.sparql.algebra.optimize?
• Is there a mechanism to ask a DatasetGraph's metadata for whether empty graphs are disallowed? In that case the FILTER EXISTS could be omitted although I suppose it'd only be a minor improvement and maybe not really worth the effort. Yet, maybe there is already a context attribute?

[rvesse]

Let's not get ahead of ourselves here, there's still some bugs in this feature to be ironed out before it's ready for merging i.e. DON'T expect it for 4.7.0

I've been working on some low level test cases for the new skip scan and it's definitely broken for some cases right now and until that's been addressed you won't see this in main

[afs]

the common query DISTINCT ?g { GRAPH ?g { ?s ?p ?o } }

Is it? Or is it a mistake for GRAPH ?g {}?

Would it make sense to handle it as a query rewrite

Adding too many "maybe" optimizations slows down fast, small queries. (We know this from BSBM.)

GRAPH ?g {} is distinct graph names.

The implementation exception might be DatasetGraphMapLink and DynamicDatasetGraph.

In DatasetGraphMap.listGraphNodes:

    // Hide empty graphs.

so it looks like it is a matter of copying that to DatasetGraphMapLink (caveat inference graphs).

DynamicDatasetGraph might be able to do better:: do listGraphNames on the wrapped dataset and filter. But this might be worse. Numbers matter.

The right thing to do is address in the implementations in a separate PR, starting with some test cases.

[Aklakan]

Is it? Or is it a mistake for GRAPH ?g {}?

I really meant GRAPH ?g { ?s ?p ?o} . On *cough* DBpedia:

• GRAPH ?g { } -> (after disabling the checkboxes) no results
• GRAPH ?g { ?s ?p ?o } -> expected results

So the "portable" query is the spo variant.

[afs]

On cough DBpedia:

The SPARQL endpoint or youR local load into TDB2 with this PR applied?

[Aklakan]

The SPARQL endpoint or you local load into TDB2 with this PR applied?

On the SPARQL endpoint - the graph patterns in my post are links to DBpedia.

[Aklakan]

Adding too many "maybe" optimizations slows down fast, small queries. (We know this from BSBM.)

Yes, it might be useful as an opt-in though; OpGraphNames typically results in listGraphNames so an algebra transform that injects OpGraphNames and FILTER EXIST to filter out empty graphs should perform well independent of the implementation - but I see that this is becoming a bigger discussion - so let's continue then in a separate issue.

[afs]

So the "portable" query is the spo variant.

It's not portable - it's a workaround. It might be a poor choice on another store.

There's an issues list for Virtuoso and the user list for Virtuoso is on SourceForge - has it been reported?
The users list is where they ask for bug reports judging by the SO responses of "use the emailing list".

We're not here to fix DBpedia. It has several deviations from the specification.

Jena is open and you can submit reports - that can be overused. Email would be better.
Time spent looking at Jena code was wasted. Sigh. That's another step to 4.7.0 delayed.

[rvesse]

Now 4.7.0 is out we should be able to get this reviewed and merged so that users have time to start testing the updated SNAPSHOTs with this improvement

[Aklakan]

We have the skip scan in use in a dataset with around 1 billion triples and graph listings are super fast 👍

What would be eventually needed is also make this feature publicly accessible in the various Tuple/DatasetGraph interfaces.
My proposal looks like this but maybe someone has already better ideas:

// D = domain tuple type (e.g. Quad or Tuple<NodeId), C = component type (e.g. Node or NodeId)
interface TupleMatcher4<D, C> {
  TupleStreamer<D, C> find(C g, C s, C p, C o, boolean distinct, int ... projectedColumns);
}

interface TupleStreamer<D, C> {
  Iterator<C> asComponents(); // e.g. Node or NodeIds
  Iterator<D> asDomainTuples(); // e.g. Quad
  Iterator<Tuple<C>> asGenericTuples();
}

This way a request for e.g. distinct predicates:

SELECT DISTINCT ?p { GRAPH ?g { ?s ?p ?o } }

could then map to a find() call such as

Iterator<Node> distinctPredicates = datasetGraph.find(ANY, ANY, ANY, ANY, true, 2).asComponents();

Furthermore, I wonder if the way TDB indexes data would be suitable for a skip scan for the case to retrieve a resource's distinct predicates:

SELECT DISTINCT ?p { GRAPH ?g { <concreteS> ?p ?o } }

The background is, that we have resources with 4mio+ statements (yeah not that usual) where a scan for distinct predicates takes seconds - maybe with the skip scan it would also be possible to speed this case up?

[rvesse]

We have the skip scan in use in a dataset with around 1 billion triples and graph listings are super fast 👍

What would be eventually needed is also make this feature publicly accessible in the various Tuple/DatasetGraph interfaces. My proposal looks like this but maybe someone has already better ideas:

// D = domain tuple type (e.g. Quad or Tuple<NodeId), C = component type (e.g. Node or NodeId)
interface TupleMatcher4<D, C> {
  TupleStreamer<D, C> find(C g, C s, C p, C o, boolean distinct, int ... projectedColumns);
}

interface TupleStreamer<D, C> {
  Iterator<C> asComponents(); // e.g. Node or NodeIds
  Iterator<D> asDomainTuples(); // e.g. Quad
  Iterator<Tuple<C>> asGenericTuples();
}

I don't suspect that this sort of low level execution optimisation is ever going to bubble up into the high level end-user APIs like DatasetGraph

I don't disagree that this iterator could be used to optimise execution of other query patterns but the goal here is to start small and incrementally improve.

This way a request for e.g. distinct predicates:

SELECT DISTINCT ?p { GRAPH ?g { ?s ?p ?o } }

could then map to a find() call such as

Iterator<Node> distinctPredicates = datasetGraph.find(ANY, ANY, ANY, ANY, true, 2).asComponents();

Furthermore, I wonder if the way TDB indexes data would be suitable for a skip scan for the case to retrieve a resource's distinct predicates:

SELECT DISTINCT ?p { GRAPH ?g { <concreteS> ?p ?o } }

The background is, that we have resources with 4mio+ statements (yeah not that usual) where a scan for distinct predicates takes seconds - maybe with the skip scan it would also be possible to speed this case up?

So as I used to tell a solution architect I worked closely with in a past $dayjob that optimisation is fundamentally a trade off. The goal of a general purpose optimiser is to apply optimisations that are generally useful to most users and most data yielding performance improvements for the general case.

Detecting whether an optimisation is applicable or not has a non-zero cost to it and for some query/dataset patterns that are unusual, e.g. a subject with 4 million statements, a general purpose optimiser shouldn't try to optimise for that because it's so outside it's normal expectations.

I would suggest for this kind of optimisation, where you have a specific query pattern that runs poorly on your dataset(s), you consider creating your own custom optimiser (based off ARQ's default one) adding in optimisations for specific query patterns you need specialised optimisation for. You can transform those into custom Op instances (derived from ARQs OpExt extension point) and provide suitable eval() implementations that can call into the relevant low level APIs as appropriate.

That gives you a way to experiment with some of these things outside of the Jena codebase and then potentially contribute them back later if they prove to be of more general value. But right now it seems like there's a lot of stuff that's very specific to your use cases that may not be generally applicable.

[afs]

One unused import.