Pending mutation in mutation outbox breaks synchronization process.

[vitalnik]

Description

Recreated the issue in a simple Todo Flutter project. Testing it on Android 11.

Adding local record to the DataStore while throttling Internet connection is causing this exception:

Error ended observation of mutation outbox: DataStoreException{message=Timeout processing PendingMutation{mutatedItem=SerializedModel{id='c11922ab-8d0d-4d2b-97a2-5c42abb38ab1', serializedData={name=Task 154, createdAt=null, description=MOBILE, id=c11922ab-8d0d-4d2b-97a2-5c42abb38ab1, updatedAt=null}, modelName=Todo}, mutationType=CREATE, mutationId=a720ed51-5a06-11ed-ae33-7dfddaac416a, predicate=MatchAllQueryPredicate}, cause=null, recoverySuggestion=Check your internet connection.} at com.amplifyframework.datastore.syncengine.MutationProcessor.drainMutationOutbox(MutationProcessor.java:120) at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$startDrainingMutationOutbox$1$com-amplifyframework-datastore-syncengine-MutationProcessor(MutationProcessor.java:101) at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda3.apply(Unknown Source:4) at io.reactivex.rxjava3.internal.operators.observable.ObservableFlatMapCompletableCompletable$FlatMapCompletableMainObserver.onNext(ObservableFlatMapCompletableCompletable.java:97) at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201) at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)

The timeout exception makes sense to me since the connection is limited.

After establishing stable Internet connection all other local updates made to the Todo table not making it to the remote DataStore. Updates got enqueued but never processed. Amplify Hub issued the corresponding OutboxMutationEvent event.

Seeing this in Android Studio logcat after adding more local Todo records:
Mutation outbox has pending mutation for [id]. Saving the metadata, but not model itself

The only way I can get around this issue is to call DataStore.clear() and then DataStore.start() to reload everything, but in that case I'm loosing all local updates.

Categories

• Analytics
• API (REST)
• API (GraphQL)
• Auth
• Authenticator
• DataStore
• Storage

Steps to Reproduce

1. Create simple ToDo project using Amplify and Flutter.
2. Add some records to Local Storage (everything is synchronized to the remote DataStore)
3. Throttle network connection using some proxy, like Charles.
4. Add another record to the Todos.
5. The above DataStoreException exception is thrown.
6. Establish a stable connection now.
7. Try to add more local records.
8. No further local updates got synchronized to the remote DataStore.

Screenshots

No response

Platforms

• iOS
• Android
• Web
• macOS
• Windows
• Linux

Android Device/Emulator API Level

API 29

Environment

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.3.6, on macOS 13.0 22A380 darwin-x64, locale en-CA)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Android Studio (version 2021.3)
[✓] Android Studio (version 2021.3)
[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
[✓] IntelliJ IDEA Community Edition (version 2022.2.3)
[✓] VS Code (version 1.72.1)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

Dependencies

Dart SDK 2.18.2
Flutter SDK 3.3.6
simple_todo_5_sync 1.0.0+1

dependencies:
- amplify_api 0.6.9 [amplify_api_android amplify_api_ios amplify_core amplify_flutter aws_common collection flutter meta plugin_platform_interface]
- amplify_datastore 0.6.9 [flutter amplify_datastore_plugin_interface amplify_core plugin_platform_interface meta collection async]
- amplify_flutter 0.6.9 [amplify_core amplify_datastore_plugin_interface amplify_flutter_android amplify_flutter_ios aws_common collection flutter meta plugin_platform_interface]
- cupertino_icons 1.0.5
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]

transitive dependencies:
- amplify_api_android 0.6.9 [flutter]
- amplify_api_ios 0.6.9 [amplify_core flutter]
- amplify_core 0.6.9 [aws_common collection flutter intl json_annotation meta plugin_platform_interface uuid]
- amplify_datastore_plugin_interface 0.6.9 [amplify_core collection flutter meta]
- amplify_flutter_android 0.6.9 [flutter]
- amplify_flutter_ios 0.6.9 [amplify_core flutter]
- async 2.9.0 [collection meta]
- aws_common 0.1.1 [async collection http meta stream_transform uuid]
- characters 1.2.1
- clock 1.1.1
- collection 1.16.0
- crypto 3.0.2 [typed_data]
- http 0.13.5 [async http_parser meta path]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- intl 0.17.0 [clock path]
- json_annotation 4.7.0 [meta]
- material_color_utilities 0.1.5
- meta 1.8.0
- path 1.8.2
- plugin_platform_interface 2.1.3 [meta]
- sky_engine 0.0.99
- source_span 1.9.0 [collection path term_glyph]
- stream_transform 2.0.1
- string_scanner 1.1.1 [source_span]
- term_glyph 1.2.1
- typed_data 1.3.1 [collection]
- uuid 3.0.6 [crypto]
- vector_math 2.1.2

Device

Pixel 2

OS

Android 11

Deployment Method

Amplify CLI

CLI Version

10.3.2

Additional Context

No response

Amplify Config

{
"UserAgent": "aws-amplify-cli/2.0",
"Version": "1.0",
"api": {
"plugins": {
"awsAPIPlugin": {
"simpletodo5sync": {
"endpointType": "GraphQL",
"endpoint": "xxx",
"region": "us-west-2",
"authorizationType": "API_KEY",
"apiKey": "xxx"
}
}
}
}
}

[HuiSF]

Hi @vitalnik thanks for reporting this issue. it sounds like the sync engine is not recovering and reconnecting after the exception. As amplify-flutter library is not involved within the sync engine work flow, I will forward this issue to amplify-android repository for further triage.

[vitalnik]

Are there any updates on this issue? Or if you can suggest a workaround that would be greatly appreciated.

[AnilMaktala]

Hey @vitalnik 👋, we are testing out potential fix for this issue and we will report back with the updates soon.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.