Sqlcipher Crash after upgrade from 3.x to 4.x

[bazimogmbh]

After migrating the sqlcipher from v 3.5.9 to v 4.2.0 , we are seeing a lot of crashes, mostly on
Galaxy S7, S8, A5(2017) & J7 Pro devices, with android version 8 & 9 installed.
The upgrade seems to work well for all the other users.

This is the hook used for the migration

 SQLiteDatabaseHook hook = new SQLiteDatabaseHook() {
        public void preKey(SQLiteDatabase database) {}
        public void postKey(SQLiteDatabase database) {
            singleValueFromQuery(database, "PRAGMA cipher_migrate");
        }
 };

public static String singleValueFromQuery(SQLiteDatabase database, String query) {
        net.sqlcipher.Cursor cursor = database.rawQuery(query, new String[]{});
        String value = "";
        if (cursor != null) {
            cursor.moveToFirst();
            value = cursor.getString(0);
            cursor.close();
        }
        return value;
    }

The database exists, the password is correct, but the migration fails it seems and the app crashes. When i provide the users an apk with old v3.5.9 sqlcipher build after they have installed the new build with v4.2.0, everything works again.
Here is the Stack trace.

Caused by net.sqlcipher.database.SQLiteException
file is not a database: , while compiling: select count(*) from sqlite_master;
net.sqlcipher.database.SQLiteCompiledSql.native_compile (SQLiteCompiledSql.java)
net.sqlcipher.database.SQLiteCompiledSql.compile (SQLiteCompiledSql.java:91)
net.sqlcipher.database.SQLiteCompiledSql. (SQLiteCompiledSql.java:12)
net.sqlcipher.database.SQLiteProgram. (SQLiteProgram.java:18)
net.sqlcipher.database.SQLiteQuery. (SQLiteQuery.java:1)
net.sqlcipher.database.SQLiteDirectCursorDriver.query (SQLiteDirectCursorDriver.java:60)
net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory (SQLiteDatabase.java:2012)
net.sqlcipher.database.SQLiteDatabase.rawQuery (SQLiteDatabase.java:1898)
net.sqlcipher.database.SQLiteDatabase.keyDatabase (SQLiteDatabase.java:2647)
net.sqlcipher.database.SQLiteDatabase.openDatabaseInternal (SQLiteDatabase.java:2577)
net.sqlcipher.database.SQLiteDatabase.openDatabase (SQLiteDatabase.java:1243)
net.sqlcipher.database.SQLiteDatabase.openDatabase (SQLiteDatabase.java:1210)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SQLiteDatabase.java:1300)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SQLiteDatabase.java:1288)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SQLiteDatabase.java:1280)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SQLiteDatabase.java:1325)
com.pixelcrater.Diaro.storage.sqlite.helpers.SQLiteMgr.getCipherEncryptedDb (SQLiteMgr.java:83)
com.pixelcrater.Diaro.storage.sqlite.SQLiteAdapter. (SQLiteAdapter.java:10)
com.pixelcrater.Diaro.storage.StorageMgr.getSQLiteAdapter (StorageMgr.java:44)
com.pixelcrater.Diaro.storage.StorageMgr. (StorageMgr.java:4)
com.pixelcrater.Diaro.MyApp.onCreate (MyApp.java:145)

We also noticed, that a lot of users were complaining about how slow the app has become, and after running some tests, we found out that using this makes sqlcipher faster again :
encryptedDb.execSQL("PRAGMA cipher_memory_security = OFF");

[developernotes]

Hi @bazimogmbh,

Can you detail how the password/key material is created/sourced?

[bazimogmbh]

Hello @developernotes,

it is a MD5 Hash value (32 chars) of some random stuff we generate and results into something like e.g. 5f069735546a1e07738ac92d06d256fb

[developernotes]

Hi @bazimogmbh

As a follow-up, a few more questions:

1. Do you have a copy of a database with a known password that is failing to perform a cipher_migrate?
2. What is the return value from singleValueFromQuery on any failures you've experienced?
3. Have you ever used any non-standard runtime configuration options for SQLCipher?

This information will be helpful to understand what the cause may be.

[bazimogmbh]

Hi @developernotes,

1. I do have a copy of the database with its password, i can provide you with that. I copied it to one of my devices as well as android emulator, the migration runs without any issues whatsoever, and everything works as expected. So the issue seems to be with these devices. I will try to get my hands on them next week. It seems that even the fresh install on these devices causes a crash.

2. I did not track that yet, will do it soon and will let you know.
3. We did not ever used any non-standard runtime configuration options for SQLCipher

[mreid-projects]

I've experienced the same crash after upgrading from v3.5.4. to v4.1.3

[sjlombardo]

@mreid-projects - are you also using random bytes as key material and using PRAGMA cipher_migrate to upgrade the database? If so, please review this guidance for using random values as keys. You may also consider upgrading to SQLCipher 4.2.0 which improves PRAGMA cipher_migrate to handle keys containing non-terminating zero bytes.

[developernotes]

Hello @bazimogmbh

I copied it to one of my devices as well as android emulator, the migration runs without any issues whatsoever, and everything works as expected.

Because of this, the issue may exist with the device itself. Do you have any of the reported devices to verify the behavior on?

I did not track that yet, will do it soon and will let you know.

We look forward to hearing the result value returned from the pragma cipher_migrate command.

[mreid-projects]

@sjlombardo - I have tried upgrading to SQLCipher 4.2.0 and using PRAGMA cipher_migrate, as shown below:

SQLiteDatabaseHook hook = new SQLiteDatabaseHook() {
    @Override
    public void preKey(SQLiteDatabase database) {
    }

    @Override
    public void postKey(SQLiteDatabase database) {
        database.execSQL("PRAGMA key = '" + key + "';");
        database.execSQL("PRAGMA cipher_migrate;");
    }
};
Database db = new EncryptedDatabase(SQLiteDatabase.openOrCreateDatabase("DB.db", key, null, hook));
return new DaoMaster(db).newSession();

...but I get the following error:

No implementation found for void net.sqlcipher.database.SQLiteDatabase.dbopen(java.lang.String, int) (tried Java_net_sqlcipher_database_SQLiteDatabase_dbopen and Java_net_sqlcipher_database_SQLiteDatabase_dbopen__Ljava_lang_String_2I)
Could not dispatch event: class com.*.LoginResponse to subscribing class class com..LoginViewModel
java.lang.UnsatisfiedLinkError: No implementation found for void net.sqlcipher.database.SQLiteDatabase.dbopen(java.lang.String, int) (tried Java_net_sqlcipher_database_SQLiteDatabase_dbopen and Java_net_sqlcipher_database_SQLiteDatabase_dbopen__Ljava_lang_String_2I)
at net.sqlcipher.database.SQLiteDatabase.dbopen(Native Method)
at net.sqlcipher.database.SQLiteDatabase.openDatabaseInternal(SQLiteDatabase.java:3)
at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:10)
at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:7)
at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:5)
at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:3)
at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:3)
at com.*.infrastructure.di.modules.GreenDAOModule.provideDAO(GreenDAOModule.java:8)
at com.*.infrastructure.di.modules.GreenDAOModule_ProvideDAOFactory.get(GreenDAOModule_ProvideDAOFactory.java:3)
at com.*.infrastructure.di.modules.GreenDAOModule_ProvideDAOFactory.get(GreenDAOModule_ProvideDAOFactory.java:1)

[developernotes]

Hi @mreid-projects

The java.lang.UnsatisfiedLinkError with no implementation found is often caused by either a ProGuard configuration issue, or improper project integration. I see you are also using GreenDAO, so the integration there may also be something that you investigate.

[mreid-projects]

Sqlcipher is calling the native method for SQLiteDatabase.dbopen(java.lang.String, int) but that method does not exist in the Android SQLiteDatabase class.
How can that be a GreenDAO problem?

[developernotes]

Hi @mreid-projects

We publish an AAR package (the latest here), we do not publish GreenDAO. You didn't mention whether you are using ProGuard, but as I mentioned above, that can often strip symbols from packaging producing runtime errors you are experiencing. As to whether the issue exists with GreenDAO which appears to be bundling SQLCipher for Android, that was speculative.

You might try running the SQLCipher for Android test suite and compare your project integration. What CPU architecture are you experiencing this issue on?

[mreid-projects]

I am testing without Proguard and am currently pulling in version 3.5.4 of sqlcipher, for use by GreenDAO. The device that I'm testing with is a Nokia 3.1 which is running Android 9 (64-bit) on an ARM Cortex-A53.

The test suite says "SQLCipher for Android runs on Android 4–Android 8, for armeabi, armeabi-v7a, x86, x86_64, and arm64_v8a architectures."
Does the current version of sqlcipher support Android 9?

[developernotes]

Hello @mreid-projects

We will update that message, we support a minimum of API 14 on 32-bit platforms, and API 21 on 64-bit platforms, currently including Android 9.

[mreid-projects]

ok thanks.

Does version 3.5.4 support 64-bit Android targets?
I've just looked at my apk and can only see libsqlcipher.so in the 32-bit folders