Mapdb2 and pumpSource problem

[adridadou]

Hi,
In my quest to work with MapDB2, I have another issue.
I currently use pumpSource with MapDB1 and everything works fine. But when I use MapDB2, I get this error:
java.lang.AssertionError: keys are not sorted: [315522, 315522, 646732, 774812, 878225, 1045003, 1170222, 1264551, 1314408, 1381461]
at org.mapdb.BTreeMap$BNode.checkStructure(BTreeMap.java:353)
at org.mapdb.BTreeMap$DirNode.checkStructure(BTreeMap.java:428)
at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:702)
at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:668)
at org.mapdb.Store.serialize(Store.java:163)
at org.mapdb.StoreDirect.put(StoreDirect.java:409)
at org.mapdb.Pump.buildTreeMap(Pump.java:456)
at org.mapdb.DB.createTreeMap(DB.java:988)
at org.mapdb.DB$BTreeMapMaker.make(DB.java:785)
....

What seems strange for me is that in MapDB1, there is a test that checks that the key orders is correct and here the AssertionError comes not from the Pump code but rather from checkStructure.

the BTree config I use is :
val cubeData:BTreeMap[Long,String] = db.createTreeMap("data")
.valueSerializer(Serializer.STRING_ASCII)
.pumpSource(CubeMapDbIterator(data)).make()

I assume the CubeMapDBIterator is correct as it works properly with MapDB1.

Any idea?

[jankotek]

There is duplicate: 315522, 315522 . There is extra method on createTreeMap which ignores duplicates.

Message is missleading, so I change this issue to update documentation.

[adridadou]

I saw that and I thought of that too.
But two issues here:

1. in MapDB 1 it works and I don't use ignoreDuplicates
2. I tried with ignoreDuplicates and I still have the issue. It looks like it creates a value two times for some reason. I will try to create a test case and send it to you when I can isolate and reproduce the issue.

[jankotek]

What mapdb version is that?

[jankotek]

Also I miss read the message

java.lang.AssertionError: keys are not sorted: [315522, 315522, 646732, 774812, 878225, 1045003, 1170222, 1264551, 1314408, 1381461]

Is not generated by datapump, but is internal assertion in BTree. It is possibly unrelated to input data

[adridadou]

Yes the message comes from BTree and not from the PumpData thing.
The version that works is 1.0.6 and the one that does not is a pull from yesterday

[jankotek]

Bug in data pump than. There were many changes in BTreeMap and Pump probably needs more testing.

[adridadou]

Good! I will try to create a test to reproduce the issue.

[jankotek]

Thanks. BTW in your case use this to get bettter result.keySerializer(BTreeKeySerializer.LONG/INT)

[adridadou]

:) Look at my other issue. I tried but got an issue too. Basically I cannot use a serializer without giving explicitly a Comparator and it is not clear how to work with it.

I tried that in MapDB1 with ZERO_OR_POSITIVE_LONG and I got an issue that I got a NPE. But I suspect the issue comes from the fact I am in Scala.

[adridadou]

I will close this issue and I can always reopen it if needed. I see that you did a lot of work on this part and I guess the issue is no more or at least not in the same form.