Add TermsCardinalitySets as an index

[pietercolpaert]

This pull request introduces a new index called TermsCardinalitySets, speeding up distinct look-ups of subject, predicate, objects and/or graphs.

In the index, we maintain an overview of their cardinalities of how many times they have been used in that position. While currently to exploited, this could also be useful towards speeding up certain counts.

Full test coverage should be reached, but I lack the typescript coverage experience to understand how the last line can be covered as well. Can you help with that?

[rubensworks]

Thanks for the PR @pietercolpaert!

I had a quick look at the proposed API, but It's quite a bit different to what I had in mind, which might be caused by our different requirements.

While we already had quad indexes, this PR adds unary indexes. But there is also a need for binary and ternary indexes.
As such, I think the most sustainable way forward to implement this is to extend or abstract the IRdfStoreIndex interface, given their common logic.

But the current implementation with TermsCardinalitySets obviously works, and solves your use case. So one option might be to use your fork of this library until I've managed to look into implementing this index abstraction.

[pietercolpaert]

Thanks! That works for me. Can we however agree already on the method names then, such as getSubjects() in order to get a distinct list of subjects?

[rubensworks]

Can we however agree already on the method names then, such as getSubjects() in order to get a distinct list of subjects?

I'm not too sure to be honest. If we follow this approach for binary and ternary access as well, then will need a huge amount of methods for handling all possible combinations of access. So we may need something more abstract. But I don't have a clean way in mind atm. Possible something using QuadTermName as params.

[pietercolpaert]

How about getDistinct(term1: QuadTermName, term2?: QuadTermName, term3?: QuadTermName, term4?: QuadTermName)

[rubensworks]

How about getDistinct(term1: QuadTermName, term2?: QuadTermName, term3?: QuadTermName, term4?: QuadTermName)

I don't think that will be sufficient to capture queries such as S? or ?P on SP index.

[pietercolpaert]

Right... How about:

getDistinct(terms: QuadTermName[], quad?: QuadPattern)

That way you can then bind the variables to the distincts you want?

i.e.

getDistinct(['subject'], [null, df.namedNode('rdf:type'),df.namedNode('foaf:Person')])

Would translate into:

SELECT DISTINCT ?s WHERE {
   ?s a foaf:Person .
}

[rubensworks]

Perhaps, something like that could work.
But I'm not 100% certain yet, I feel like additional requirements may pop up once I start designing the architecture for handling these different indexes into comunica 😅