Row too big to fit into CursorWindow

[laurent22]

Expected Behavior

The query with large data should succeed without an error.

Current Behavior

When trying to a do a SELECT on large rows, the query fails with error:

Row too big to fit into CursorWindow

Steps to Reproduce (for bugs)

This is affecting several users of Joplin:

laurent22/joplin#1771
laurent22/joplin#1690

and it happens when trying to read or write to the database large text data (some users mention 4MB).

It seems to have started only recently so it could be due to a React Native upgrade or other change to the Android build.

What I'm wondering is, is there any way to prevent this error, for example some option that can be set to increase the available memory?

The Joplin desktop application uses the same architecture (but of course with a different sqlite package) and this error doesn't happen, so it seems it's possible to prevent it. 4MB is a bit large but not huge either so it seems it should work, at least with an option.

Your Environment

• React Native SQLite Storage Version used: 3.3.10
• React Native version used: 0.59.10
• Operating System and version (simulator or device): Android (device)
• Link to your project: https://github.com/laurent22/joplin

[jgreen210]

I'm seeing the same error:

SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1

...but it only started happening after upgrading the test device to android 9.0 (from 8 I think).

React Native SQLite Storage Version used: 3.3.3 (which is obviously a little old now).
React Native version used: 0.59.5
Operating System and version (simulator or device): Android 9.0 (device)

Android 9.0 is implicated by the joplin tickets too.

[jgreen210]

These links discuss this problem and suggest some solutions:

https://stackoverflow.com/questions/51959944/sqliteblobtoobigexception-row-too-big-to-fit-into-cursorwindow-requiredpos-0-t
https://medium.com/androiddevelopers/large-database-queries-on-android-cb043ae626e8

[jgreen210]

There's a warning before the error:

CursorWindow: Window is full: requested allocation 2402739 bytes, free space 2096651 bytes, window size 2097152 bytes

We've got BLOBs in each row, some of which are around this size. This is for an app with many existing users, so we can't just make the rows smaller somehow.

[andpor]

can you attach full native debug log.

[bashen1]

@andpor
craftzdog/react-native-sqlite-2#57 (comment)
For those who encountered this problem, add following code to your MainApplication.onCreate in MainApplication.java:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

Note that it requires Android 9 (API level 28).

this work for me.

[jgreen210]

https://stackoverflow.com/questions/11340257/sqlite-android-database-cursor-window-allocation-of-2048-kb-failed/50725977#50725977 mentions the above sCursorWindowSize fix. It will affect all code using CursorWindows across the whole application, so could easily make your application use too much memory. It's changing the value of this field: https://android.googlesource.com/platform/frameworks/base/+/f86bea9b4277d9fe2c1b8bfae872a8fad15c4dc1/core/java/android/database/CursorWindow.java#48, so will generate a warning:

https://developer.android.com/distribute/best-practices/develop/restrictions-non-sdk-interfaces
https://android.googlesource.com/platform/frameworks/base/+/pie-release/config/hiddenapi-light-greylist.txt#2021

https://stackoverflow.com/questions/11340257/sqlite-android-database-cursor-window-allocation-of-2048-kb-failed/56887043#56887043 is an android P specific fix, that would allow a different window size to be used in each case, but requires a change to be made to the react-native-sqlite-storage application.

The ideal solution is to change your application to keep the row size small, although that's generally not possible for an already-released application where users have large rows and upgrade to android 9.

I've still not worked out why this only affects us (and others) on Android 9 and above - our rows can be too large for the default cursor window size, and it looks like this should have always been the case on every platform.

The cursor window size defaults to the value of the com.android.internal.R.integer.config_cursorWindowSize android platform resource, which has been 2MB for some time (probably), for at least AOSP builds: https://stackoverflow.com/a/35012963

[laurent22]

Anyone found a working fix for this? I've applied the one below, but users are still having the same error.

CursorWindow.class.getDeclaredField("sCursorWindowSize").setAccessible(true);
CursorWindow.class.getDeclaredField("sCursorWindowSize").set(null, 50 * 1024 * 1024); // 50 MB

[ahmed-ismail-nt]

Just hit this as well.

The logs are below in case they help anyone.

The SQL query executed is there as well and is executed when my app rehydrates the redux state (based on prsn's redux-persist-sqlite-storage package). The error only occurs the first time the app is opened after an upgrade. Otherwise, closing and re-opening the application doesn't trigger the error.

The default 2MB limit for the CursorWindow applies here.

10-31 18:45:00.388  5561  5622 W CursorWindow: Window is full: requested allocation 2124434 bytes, free space 2096720 bytes, window size 2097152 bytes
10-31 18:45:00.389  5561  5622 E SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT value FROM store WHERE key=?
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin: SQLitePlugin.executeSql[Batch](): failed
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin: android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=0, totalRows=1
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:859)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:836)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:149)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.sqlite.SQLiteCursor.getCount(SQLiteCursor.java:137)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.AbstractCursor.moveToPosition(AbstractCursor.java:220)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at android.database.AbstractCursor.moveToFirst(AbstractCursor.java:259)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at org.pgsqlite.SQLitePlugin.executeSqlStatementQuery(SQLitePlugin.java:812)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at org.pgsqlite.SQLitePlugin.executeSqlBatch(SQLitePlugin.java:712)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at org.pgsqlite.SQLitePlugin.access$100(SQLitePlugin.java:49)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at org.pgsqlite.SQLitePlugin$DBRunner.run(SQLitePlugin.java:927)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
10-31 18:45:00.390  5561  5622 E unknown:SQLitePlugin:  at java.lang.Thread.run(Thread.java:764)

[roycechua]

For anyone having issues with this error you might want to check my comment here craftzdog/react-native-sqlite-2#57 (comment)
That worked for me.

[roycechua]

@andpor
craftzdog/react-native-sqlite-2#57 (comment)
For those who encountered this problem, add following code to your MainApplication.onCreate in MainApplication.java:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

Note that it requires Android 9 (API level 28).

this work for me.

You have to add the necessary imports

import android.database.CursorWindow;
import java.lang.reflect.Field;

[aqeebimtiaz]

@andpor
craftzdog/react-native-sqlite-2#57 (comment)
For those who encountered this problem, add following code to your MainApplication.onCreate in MainApplication.java:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

Note that it requires Android 9 (API level 28).

this work for me.

I got an error for DEBUG_MODE, had to comment that out for a quick fix.

BTW Thanks, @roycechua23 for pointing the import issues.

[bambinoua]

@aqeebimtiaz

I got an error for DEBUG_MODE, had to comment that out for a quick fix.

You may use this code:

if (BuildConfig.DEBUG) {
  e.printStackTrace();
}

[aqeebimtiaz]

@bambinoua Thank you! 💯 I'll give it a try!

[danieldanielecki]

@andpor
craftzdog/react-native-sqlite-2#57 (comment)
For those who encountered this problem, add following code to your MainApplication.onCreate in MainApplication.java:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

Note that it requires Android 9 (API level 28).
this work for me.

I got an error for DEBUG_MODE, had to comment that out for a quick fix.

BTW Thanks, @roycechua23 for pointing the import issues.

I got Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. for the DEBUG_MODE part, so had to strip it off and it works!

[matt-dalton]

@andpor
craftzdog/react-native-sqlite-2#57 (comment)
For those who encountered this problem, add following code to your MainApplication.onCreate in MainApplication.java:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

Note that it requires Android 9 (API level 28).
this work for me.

I got an error for DEBUG_MODE, had to comment that out for a quick fix.

BTW Thanks, @roycechua23 for pointing the import issues.

Out of interest, is there any risk associated with increasing this limit so much (from 4MB -> 100MB)? Is this potentially going to cause other issues that we should watch out for?

[laurent22]

For the record I've implemented that fix a while back:

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  if (DEBUG_MODE) {
    e.printStackTrace();
  }
}

And we're still getting the error now and then, so I'm not sure it's doing anything at all.