Problems generating or using empty HDT files

[osma]

I noticed that the hdt-java tools cannot handle empty HDT files i.e. files with zero triples.

Trying to generate a HDT file based on an empty N-Triple file fails:

$ touch empty.nt # create an empty N-Triples file
$ rdf2hdt.sh empty.nt empty.hdt
Converting empty.nt to empty.hdt as null
Exception in thread "main" java.lang.IllegalArgumentException: Adjacency list bitmap and array should have the same size
	at org.rdfhdt.hdt.compact.bitmap.AdjacencyList.<init>(AdjacencyList.java:50)
	at org.rdfhdt.hdt.triples.impl.BitmapTriples.load(BitmapTriples.java:207)
	at org.rdfhdt.hdt.triples.impl.BitmapTriples.load(BitmapTriples.java:224)
	at org.rdfhdt.hdt.hdt.impl.HDTImpl.loadFromModifiableHDT(HDTImpl.java:377)
	at org.rdfhdt.hdt.hdt.HDTManagerImpl.doGenerateHDT(HDTManagerImpl.java:107)
	at org.rdfhdt.hdt.hdt.HDTManager.generateHDT(HDTManager.java:129)
	at org.rdfhdt.hdt.tools.RDF2HDT.execute(RDF2HDT.java:106)
	at org.rdfhdt.hdt.tools.RDF2HDT.main(RDF2HDT.java:167)

Another way of triggering the same exception is to generate the zero-triple HDT file using hdt-cpp (which works) and then attempt to use it using hdtsparql.sh:

$ touch empty.nt # create an empty N-Triples file
$ rdf2hdt empty.nt empty.hdt # make a HDT file out of it using rdf2hdt from the hdt-cpp suite
$ hdtsparql.sh empty.hdt "select * {?s ?p ?o}"
Exception in thread "main" java.lang.IllegalArgumentException: Adjacency list bitmap and array should have the same size
	at org.rdfhdt.hdt.compact.bitmap.AdjacencyList.<init>(AdjacencyList.java:50)
	at org.rdfhdt.hdt.triples.impl.BitmapTriples.mapFromFile(BitmapTriples.java:372)
	at org.rdfhdt.hdt.hdt.impl.HDTImpl.mapFromHDT(HDTImpl.java:260)
	at org.rdfhdt.hdt.hdt.HDTManagerImpl.doMapIndexedHDT(HDTManagerImpl.java:62)
	at org.rdfhdt.hdt.hdt.HDTManager.mapIndexedHDT(HDTManager.java:93)
	at org.rdfhdt.hdtjena.cmd.HDTSparql.main(HDTSparql.java:38)

While one can argue about the usefulness of empty (i.e. zero triples) HDT files, I don't think this special case should trigger an exception. I noticed this while writing unit tests for my application; the tests exercise some special situations, and one of them happens to generate an empty NT file which will then be converted to HDT and queried using hdtsparql.sh.

[D063520]

this seams to work in the current version

@mielvds can you check on your side?

[D063520]

In fact inside the java version everything is working .... but it is not compatible with the C++ version. Empty file created with the c++ version

empty file created with the java version:

[D063520]

So for the direction

1. compressing an empty HDT file with c++
2. opening the hdt file with java version
   my guess is that the c++ version is wrong. Here:

hdt-java/hdt-java-core/src/main/java/org/rdfhdt/hdt/triples/impl/BitmapTriples.java

Line 214 in 594e5f6

adjY = new AdjacencyList(seqY, bitmapY);

the seqY has length zero but bitmapY has length 1. I'm 90% sure this is encoded in the HDT file wrongly.

Why then it opens in c++ and not in java? The c++ version is not making the same check as java

https://github.com/rdfhdt/hdt-cpp/blob/332a9cc2d5273e76b9daad366f7d2f80adb6b3fc/libhdt/src/sequence/AdjacencyList.cpp#L38

hdt-java/hdt-java-core/src/main/java/org/rdfhdt/hdt/compact/bitmap/AdjacencyList.java

Line 49 in 594e5f6

if (array.getNumberOfElements() != bitmap.getNumBits()) {

that is why in c++ we can search over it and not in java.

Moreover if we remove the check in java we can open the file and it contains no triples.

@mielvds : should I open an issue in the c++ version?

[D063520]

For the direction:

1. compressing an empty HDT file with java
2. opening the hdt file with c++ version

I think the best is to debug it in the c++ version.
@mielvds I would move this also to the c++ version therefore .....

[mielvds]

Nice findings @D063520 ! Yes, let's bounce this over to the C++ version. and close this issue

[D063520]

Ok, I'm moving the first issue ....