LMDB very slow with server/workbench

[maxstolze]

We created a sail lmdb repository via console on the RDF4J server and tested it against native and memory store and it so 10 times slower.
This is our template:

#
# Rdf4j configuration template for a sail-lmdb repository
#
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rep: <http://www.openrdf.org/config/repository#>.
@prefix sr: <http://www.openrdf.org/config/repository/sail#>.
@prefix sail: <http://www.openrdf.org/config/sail#>.
@prefix lmdb: <http://rdf4j.org/config/sail/lmdb#>.
@prefix sl: <http://www.openrdf.org/config/sail/lucene#>.
@prefix ssc: <http://rdf4j.org/config/sail/shacl#> .

[] a rep:Repository ;
   rep:repositoryID "{%Repository ID|memory%}" ;
   rdfs:label "{%Repository title|Max Test Repo%}" ;
   rep:repositoryImpl [
      rep:repositoryType "openrdf:SailRepository" ;
      sr:sailImpl [
         sail:sailType "rdf4j:ShaclSail" ;
         ssc:parallelValidation true;
         sail:delegate [
          sail:sailType "rdf4j:LmdbStore"
          ];
      ];
   ].

What are we doing wrong/missing?
(Already fixed the dependency LWJGL bug before)

Thanks in advance!

[hmottestad]

Is it 10x slower than the NativeStore?

What exactly is 10x slower? Loading data? Querying? Deleting? Updating?

If you use the LmdbStore directly, without the ShaclSail, is it still slow?

[fkleedorfer]

here it is. I can make it bigger if needed.
https://gist.github.com/fkleedorfer/ffdb53f2419ed750158680b250ba84b9

[fkleedorfer]

Maybe you can also provide some kind of generator function for Java? I thought about using your example data and simpling duplicating the items (while following certain rules).

I don't see an easy way for us to do that. It's much faster to generate the data on our side as needed.

[kenwenzel]

Thank you for the test data. I've implemented a test for both the NativeStore and the LmdbStore:
https://github.com/kenwenzel/rdf4j/tree/GH-3982-lmdb-query-optimization

Unfortunately, I get almost the same results for your example query. Do you bind some of the variables when you execute the query?

LmdbStore:

NativeStore:

[fkleedorfer]

Do you bind some of the variables when you execute the query?

@maxstolze correct me if I am wrong - but it seems to me that this query can only be executed in our code with a binding for ?root_standard (and nothing else).

The dataset I provided will not produce interesting results there because the 'standard' in question has not been copied. Here is another dataset: https://gist.github.com/fkleedorfer/3b03ab4612f470b4cf99b3bf26bd575b

... Meant to be added to the first one.

It contains a 'standard' that has been copied. (so there are ?x mms:copyOf ?y triples in there. I think binding ?root_standard to urn:uuid:fed35f18-bd2e-4cfe-a6ea-bbb351b49e75 might lead to our observed behaviour.

[kenwenzel]

OK, this is really helpful. I'll first try to improve the existing algorithm for computing the cardinalities before moving to something like HLL.

[kenwenzel]

@maxstolze Which indexes are you using with the LMDB store. The default configuration?

[fkleedorfer]

We did not change the indexing.

[kenwenzel]

OK. Maybe you can try with "spoc,ospc,psoc" until the issue ist fixed.

[kenwenzel]

@fkleedorfer @maxstolze Maybe you can try #4183 with your data?

[kenwenzel]

The fix is included in RDF4J 4.2.0.

[maxstolze]

Thank you! It's now as fast as NativeStore with our setup!

[kenwenzel]

@maxstolze Thank you for the information. How many triples has your data set?

[maxstolze]

[kenwenzel]

Ah ok, that is not much. Probably the LMDB store mainly pays off if the statements don't fit completely into RAM.
Nonetheless, it would be interesting if you could collect some profiling information with VisualVM.