Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug-Upgrading Cloud Firestore SDK causes crash loop for some users #370

Closed
highthunder opened this issue Apr 22, 2019 · 8 comments

Comments

@highthunder
Copy link

commented Apr 22, 2019

[READ] Step 1: Are you in the right place?

Yes, this is an issue with Cloud Firestore's internal on-device sqlite database and this repository appears to contain all of the code required to manage that database.

[REQUIRED] Step 2: Describe your environment

  • Android Studio version: 3.4
  • Firebase Component: Cloud Firestore
  • Component version: 17.0.2 upgrading to 18.2.0
  • Gradle version: 4.10.1
  • com.android.tools.build:gradle:3.3.2
  • Compile/Target SDK version: 28

[REQUIRED] Step 3: Describe the problem

Immediately after releasing an app update which included the library upgrade we started to see users crash looping, seemingly starting when the app first restarted after updating.

Crashlytics Stack Trace

# Crashlytics - plaintext stacktrace downloaded by Developer at Mon, 22 Apr 2019 19:19:01 GMT
# URL: https://fabric.io/maven-machines-inc/android/apps/com.mavenmachines/issues/5c6af9dbf8b88c29634c7cec/sessions/5CBE11CA008800014C5BB70A3A4E0045_DNE_0_v2?
# Organization: Maven Machines, Inc.
# Platform: android
# Application: Maven Machines
# Version: 3.28.0-7 (177)
# Bundle Identifier: com.mavenmachines
# Issue ID: 5c6af9dbf8b88c29634c7cec
# Session ID: 5CBE11CA008800014C5BB70A3A4E0045_DNE_0_v2
# Date: 2019-04-22T19:11:00Z
# OS Version: 7.1.1
# Device: Galaxy Tab E 8.0
# RAM Free: 41.8%
# Disk Free: 81.2%

#0. Crashed: main
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

--

Fatal Exception: java.lang.RuntimeException: Internal error in Firestore (18.2.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

Caused by java.lang.RuntimeException: java.lang.IllegalStateException: Couldn't read row 6987, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
       at com.google.firebase.firestore.util.AsyncQueue.enqueue(AsyncQueue.java:290)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$3.run(AsyncQueue.java)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at com.google.firebase.firestore.util.AsyncQueue$DelayedStartFactory.run(AsyncQueue.java:205)
       at java.lang.Thread.run(Thread.java:762)

Caused by java.lang.IllegalStateException: Couldn't read row 6987, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
       at android.database.CursorWindow.nativeGetString(CursorWindow.java)
       at android.database.CursorWindow.getString(CursorWindow.java:451)
       at android.database.AbstractWindowedCursor.getString(AbstractWindowedCursor.java:51)
       at com.google.firebase.firestore.local.SQLiteSchema.lambda$ensureSequenceNumbers$7(SQLiteSchema.java:369)
       at com.google.firebase.firestore.local.SQLiteSchema$$Lambda$7.accept(SQLiteSchema.java)
       at com.google.firebase.firestore.local.SQLitePersistence$Query.forEach(SQLitePersistence.java:431)
       at com.google.firebase.firestore.local.SQLiteSchema.ensureSequenceNumbers(SQLiteSchema.java:366)
       at com.google.firebase.firestore.local.SQLiteSchema.runMigrations(SQLiteSchema.java:114)
       at com.google.firebase.firestore.local.SQLiteSchema.runMigrations(SQLiteSchema.java:65)
       at com.google.firebase.firestore.local.SQLitePersistence$OpenHelper.onUpgrade(SQLitePersistence.java:283)
       at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:256)
       at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:163)
       at com.google.firebase.firestore.local.SQLitePersistence.start(SQLitePersistence.java:120)
       at com.google.firebase.firestore.core.FirestoreClient.initialize(FirestoreClient.java:238)
       at com.google.firebase.firestore.core.FirestoreClient.lambda$new$2(FirestoreClient.java:116)
       at com.google.firebase.firestore.core.FirestoreClient$$Lambda$2.run(FirestoreClient.java)
       at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$4(AsyncQueue.java:311)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$4.call(AsyncQueue.java)
       at com.google.firebase.firestore.util.AsyncQueue.enqueue(AsyncQueue.java:287)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$3.run(AsyncQueue.java)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at com.google.firebase.firestore.util.AsyncQueue$DelayedStartFactory.run(AsyncQueue.java:205)
       at java.lang.Thread.run(Thread.java:762)

#0. Crashed: main
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

Relevant Code:

Upgrade from:

    implementation 'com.google.firebase:firebase-core:16.0.1'
    implementation 'com.google.firebase:firebase-messaging:17.1.0'
    implementation 'com.google.firebase:firebase-firestore:17.0.2'
    implementation 'com.google.firebase:firebase-auth:16.0.2'

to:

    implementation 'com.google.firebase:firebase-core:16.0.8'
    implementation 'com.google.firebase:firebase-messaging:17.5.0'
    implementation 'com.google.firebase:firebase-firestore:18.2.0'
    implementation 'com.google.firebase:firebase-auth:16.2.0'
@google-oss-bot

This comment has been minimized.

Copy link
Collaborator

commented Apr 22, 2019

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

@schmidt-sebastian

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

@highthunder Thanks for reporting! This does sound a little scary and definitely looks like a problem in our SDK. We will look at this internally and hope to have an update for you soon!

@schmidt-sebastian

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

We believe that you are hitting an issue that is similar to #208. In SQLite, a CursorWindow cannot hold more than 2 MB of data, and it looks like during our schema migration even just reading a single column can hit this limit. The offending code is here: 58c2075#diff-7e86ec47aa1c080b4b33aee14ddd7489R272

We will get this fixed shortly. In the meantime, for the clients that failed the schema upgrade, you can continue to use the older version of Firestore. Optionally, you can also turn off persistence: https://firebase.google.com/docs/reference/android/com/google/firebase/firestore/FirebaseFirestoreSettings.Builder.html#setPersistenceEnabled(boolean)

@highthunder

This comment has been minimized.

Copy link
Author

commented Apr 23, 2019

Just to clarify, is this an issue with a single row being too large, or is it because the SDK is attempting to migrate a large number of rows at once?
Because all of the crashes we've observed have occurred when reading a row id around 7000.

@schmidt-sebastian

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

It's the combined size of the result set. In your case, it looks fetching the document names of like ˜ 7000 documents exceeds the maximum Cursor Window.

@schmidt-sebastian

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

I have been able to reproduce this and have a fix that addresses the issue in a local test. I will open a PR shortly.

We unfortunately missed the release train for the next release, so I suspect that this won't go out until late May.

@highthunder

This comment has been minimized.

Copy link
Author

commented Apr 24, 2019

The main reason we attempted to update this dependency was because a new feature we are trying to release relies heavily on whereArrayContains queries which are not available on the version we had previously been using.

Would there be any issues if we update our app to use version 17.1.0 so we can use whereArrayContains while also avoiding this troublesome database migration?

@schmidt-sebastian

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

17.1.0 does not yet contain the offending schema migration and should avoid these issues for users that have not yet updated. Be aware though - running 17.1.0 on clients that have already upgraded their database schema should work, but we haven't tested this very thoroughly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.