Store header is corrupt

[ghost]

This is the error which is triggered everytime on compacting db.

Caused by: org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "Closing" [90028-198]
Caused by: java.lang.IllegalStateException: Store header is corrupt: split:30:/data/data/org.briarproject.briar.android.debug/app_db/db.mv.db [1.4.198/6]

Database:
https://code.briarproject.org/briar/briar/-/blob/master/bramble-core/src/main/java/org/briarproject/bramble/db/H2Database.java#L60

Compact and close:
https://code.briarproject.org/briar/briar/-/blob/master/bramble-core/src/main/java/org/briarproject/bramble/db/H2Database.java#L116

https://code.briarproject.org/briar/briar/-/blob/master/bramble-core/src/main/java/org/briarproject/bramble/db/JdbcDatabase.java#L429

Replacing "SHUTDOWN COMPACT" with "SHUTDOWN" does not trigger any error.
Removing CIPHER=AES does not trigger any error.

Error started from 1.4.198 and is present with the lastest 2.0.204.

Full trace log:
db.trace.txt

[manticore-projects]

Greetings.

Unfortunately I observe the same problem with SHUTDOWN DEFRAG on 2.0.204 -- it repeatingly corrupts the database.

1. Create a 2.0.204 DB from SQL script
2. Run a lots of DMLs over it
3. Call SHUTDOWN DEFRAG (without shutting down the DB)
4. Re-open the DB will fail

What seems to work is:

1. Create a 2.0.204 DB from SQL script
2. Run a lots of DMLs over it
   3a) Close the DB
   3b) Open the DB and Call SHUTDOWN DEFRAG
3. Re-open the DB will succeed

[ghost]

Can confirm. Corrupts with DEFRAG also

[manticore-projects]

As far as I understand it, COMPACT is a synonym for DEFRAG.

[andreitokar]

Yes, COMPACT is now a synonym for DEFRAG, they used to mean different things in the past.
I can see that this command was broken (only) for encrypted databases, since it has been re-implemented in 1.4.198.
It never worked since then, because it opens a new store to put compacted data into, but do not have access to encryption key to apply. It's simply long gone by then. Not sure how to deal with this problem without compromising security.

@manticore-projects, it looks like the problem you have, is different (unless your database is also encrypted).
I wonder if it is a new problem, and 1.4.200 was OK, or it's been there from day one, (because AFAIK, there were no changes in this area). So, if you are not using encryption, and can provide some reproducible test case ...

[manticore-projects]

@manticore-projects, it looks like the problem you have, is different (unless your database is also encrypted).

Not encrypted. And it is a FRESH DB created from SQL Script, with 2.0.204.

I wonder if it is a new problem, and 1.4.200 was OK, or it's been there from day one, (because AFAIK, there were no changes in this area).

I have observed similar corruptions since 1.4.199 but never have been able to pin-point it.
This time, it is different because: 1) I can reproduce this issue reliably and 2) I have the DB as SQL Script, which I am able to create the DB from and it does not seem to have any issues -- unless I call that SHUTDOWN DEFRAG.

So, if you are not using encryption, and can provide some reproducible test case ...

I have this reproducible test case and would love to render all needed assistance to solve this. There is a caveat though: the DB SQL Script is 200 MB (gzipped) and the DDLs I have been running over are Program Logic.

How shall I make that available to you?

[andreitokar]

@manticore-projects, Maybe Google drive? Or any file sharing platform, really.
Does it have to be that big? Have you tried to cut that script in half?
If you don't want it to be public - you can encrypt it with pgp, my public key is here
my-pgp.pub.txt, or you can get it from MIT repo:

gpg --keyserver pgp.mit.edu --recv-keys E57AD3B89C8AEE8F996044EDDDCBA18A81E46A97

[wangwufan]

Maybe it's just that the password is wrong...

[manticore-projects]

@andreitokar, thanks for helping!

I believe, that my particular concern has been addressed by you already in 2.2.219 -- I have not seen this corruption since (and I run quite a lot of H2 databases on different sides).

That said: Why not just block/disable DEFRAG on encrypted database file? It can be explained, why it was not supposed to work and corrupting DBs is pretty much the worst case from an end-users point of view. Better to disable the feature instead of breaking the DB.