Migrating from 3.x to 4.x: Database file is corrupted

[sea-steve]

Trying again to migrate from 3.x to 4.x. I've checked out the 4.x branch and did the refactoring in my codebase to use 4.x. When I initially tried to start my app, I saw the following exception when it tried to open the database:

Caused by: java.lang.ClassNotFoundException: org.h2.mvstore.MVMap$MapBuilder

After adding the H2 dependency (2.1.214), I now see the following error when trying to open the database:

Caused by: org.dizitart.no2.exceptions.NitriteIOException: Database file is corrupted
	at org.dizitart.no2.mvstore.MVStoreUtils.openOrCreate(MVStoreUtils.java:83)
	at org.dizitart.no2.mvstore.NitriteMVStore.openOrCreate(NitriteMVStore.java:53)
	at org.dizitart.no2.NitriteDatabase.initialize(NitriteDatabase.java:231)
	at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:107)
...
Caused by: org.h2.mvstore.MVStoreException: The write format 1 is smaller than the supported format 2 [2.1.214/5]
	at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:1004)
	at org.h2.mvstore.MVStore.getUnsupportedWriteFormatException(MVStore.java:1066)
	at org.h2.mvstore.MVStore.readStoreHeader(MVStore.java:886)
	at org.h2.mvstore.MVStore.<init>(MVStore.java:463)
	at org.h2.mvstore.MVStore$Builder.open(MVStore.java:4082)
	at org.dizitart.no2.mvstore.compat.v1.UpgradeUtil.tryUpgrade(UpgradeUtil.java:59)
	at org.dizitart.no2.mvstore.MVStoreUtils.tryUpgrade(MVStoreUtils.java:178)
	at org.dizitart.no2.mvstore.MVStoreUtils.openOrCreate(MVStoreUtils.java:69)
	... 52 more

It seems that H2 is complaining that since the database is v1, H2 v2 cannot open it? My understanding is that 4.x has explicit code to upgrade a database when opening it (presumably to support migration from 3.x to 4.x), do you know why this isn't working? Or how I can fix this or work around it? The same file opens without error under 3.x.

Here is my 4.x code for opening the db:

      storeModule = MVStoreModule.withConfig()
        .filePath(App.DB_FILE_PATH)
        .compressHigh(true)
        .recoveryMode(true)
        .autoCommit(true)  
        .build();

    Nitrite db = Nitrite.builder()
        .loadModule(storeModule)
        .loadModule(new JacksonMapperModule(new JavaTimeModule())) // optional
        .openOrCreate();

[anidotnet]

Not sure what causing this issue. You can take a look at the upgrade test code here -

nitrite-java/nitrite-mvstore-adapter/src/test/java/org/dizitart/no2/NitriteTest.java

Line 424 in 072909e

public void testReadCompatibility() throws IOException {

One thing I'll suggest, to use minimal store config, as there are config changes as well in H2 v1 vs v2.

I'll caution you against the upgrade functionality, all kind of v1 db might not be upgradable to v2. H2 itself does not support the upgradation (no idea why). Here, it is a mere effort to try upgrade on common use cases only.

[sea-steve]

Thanks for the suggestions. I'll give them a try.

Failing those, if the Nitrite upgrade logic is for common use-cases only, what do you suggest for people with (apparently) non-common use-cases?

[anidotnet]

In un-common use cases, unfortunately, there is no upgrade path yet. User have to destroy the old database and re-create the new one. That's what H2 is also suggesting.

[sea-steve]

Looks like H2 recommends export/import using their export/import tools. I'll take a look at that...

https://www.h2database.com/html/migration-to-v2.html

[sea-steve]

Also, the "Database file is corrupted" message is bogus: the open/upgrade code creates a new (empty) database file whose name is the old db file appended with "_new" (e.g., "my-nitrite.db" -> "my-nitrite.db_new"). The open/upgrade starts to populate this new file, but if the upgrade/open fails (as it did in my case), the "_new" file is not deleted and on subsequent attempts to open "my-nitrite.db", it tries to open the existing "my-nitrite.db_new" file and fails (with the exception/stacktraces I shared -- "Database is corrupted"). The "_new" file should really be deleted on failure (or the code should check for an existing "_new" file and, if it exists, create a new uniquely named file)...

If I delete the "_new" file and try to start my app, here is the actual error that's happening during the open/upgrade (I abbreviated a long list of numbers following "-84, -19, 0,..."):

Caused by: java.lang.IllegalArgumentException: Could not deserialize [-84, -19, 0,... [2.1.214/0]
	at org.dizitart.no2.mvstore.compat.v1.mvstore.DataUtils.newIllegalArgumentException(DataUtils.java:925)
	at org.dizitart.no2.mvstore.compat.v1.NitriteDataType.deserialize(NitriteDataType.java:363)
	at org.dizitart.no2.mvstore.compat.v1.NitriteDataType$SerializedObjectType.read(NitriteDataType.java:1774)
	at org.dizitart.no2.mvstore.compat.v1.NitriteDataType.read(NitriteDataType.java:525)
	at org.dizitart.no2.mvstore.compat.v1.NitriteDataType.read(NitriteDataType.java:408)
	at org.dizitart.no2.mvstore.compat.v1.mvstore.Page$Leaf.readPayLoad(Page.java:1483)
	at org.dizitart.no2.mvstore.compat.v1.mvstore.Page.read(Page.java:596)
	at org.dizitart.no2.mvstore.compat.v1.mvstore.Page.read(Page.java:232)
	at org.dizitart.no2.mvstore.compat.v1.mvstore.MVStore.readPage(MVStore.java:2200)
	... 61 more
Caused by: java.lang.ClassCastException: class org.dizitart.no2.mvstore.compat.v1.Compat$NitriteId cannot be cast to class java.lang.Comparable (org.dizitart.no2.mvstore.compat.v1.Compat$NitriteId is in unnamed module of loader 'app'; java.lang.Comparable is in module java.base of loader 'bootstrap')
	at java.base/java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:393)
	at java.base/java.util.concurrent.ConcurrentSkipListMap.readObject(ConcurrentSkipListMap.java:1255)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1101)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2442)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2280)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1760)
	at java.base/java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2625)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2476)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2280)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1760)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:538)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:496)
	at org.dizitart.no2.mvstore.compat.v1.NitriteDataType.deserialize(NitriteDataType.java:360)
	... 68 more

This is very similar to the errors I remember seeing the last time I tried to migrate from 3.x to 4.x. Any thoughts?

[anidotnet]

You are right. The unfinished new file needs to be cleaned up. Is it possible for you to share the data file? I could not reproduce it at my end.

[anidotnet]

Necessary clean up code has been added to 4.2.0-SNAPSHOT.