http fails to execute after offline mutation #33

danielvtan · 2018-09-06T13:21:31Z

steps:
offline mutation
turn internet on
mutation fails

sdk version: 2.6.25 (latest)
device type: emulator

// appsync client instance
public static AWSAppSyncClient getInstance(final Activity context) {
if(client == null){
            client = AWSAppSyncClient.builder()
                    .context(context)
                    .cognitoUserPoolsAuthProvider(getUserPoolAuthProvider(context))
                    .region(Constants.APPSYNC_REGION)
                    .serverUrl(Constants.APPSYNC_API_URL)
                    .persistentMutationsCallback(persistentMutationCallback)
                    .build();
}
}
// mutation call
public void createCall() {

// other code
        Log.d(TAG, "create call");
        AppSyncFactory.getInstance(activity).mutate(createCallMutation, expected)
                .enqueue(new GraphQLCall.Callback<CreateCallMutation.Data>() {
                    @Override
                    public void onResponse(@Nonnull final Response<CreateCallMutation.Data> response) {
                        Log.d(TAG, "success");
                    }
                    @Override
                    public void onFailure(@Nonnull ApolloException e) {
                        Log.e(TAG, "Failed to make call", e);
                        
                    }
                });
    }

Please include a stacktrace if applicable.

D/AppSync: Internet connected.
D/AppSync: Checking if I need to process next originalMutation
D/AppSync: First check: Internet Available
D/AppSync: Second check: Persistent mutations queue is EMPTY!
D/AppSync: Processing next in queue: INMEMORY.
D/AppSync: Handling offline originalMutation.
D/AppSync: onFetch()
D/AppSync: onFailure()Failed to execute http call
D/AppSync: Got message to take action regarding mutation queue.
D/AppSync: Got message that a originalMutation process errored out.
D/AppSync: Checking if I need to process next originalMutation
D/AppSync: First check: Internet Available
D/AppSync: Second check: Persistent mutations queue is EMPTY!
D/AppSync: Third check: Inmemory mutations queue is EMPTY!
                          
 com.apollographql.apollo.exception.ApolloNetworkException: Failed to execute http call
                                                                                       at com.apollographql.apollo.internal.interceptor.ApolloServerInterceptor$1$1.onFailure(ApolloServerInterceptor.java:105)
                                                                                       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:148)
                                                                                       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                                                                                       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
                                                                                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
                                                                                       at java.lang.Thread.run(Thread.java:818)
                                                                                    Caused by: java.net.SocketTimeoutException: timeout
                                                                                       at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:593)
                                                                                       at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:601)
                                                                                       at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:146)
                                                                                       at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
                                                                                       at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                                       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                                       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                                       at com.amazonaws.mobileconnectors.appsync.sigv4.AppSyncSigV4SignerInterceptor.intercept(AppSyncSigV4SignerInterceptor.java:165)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                                       at com.amazonaws.mobileconnectors.appsync.retry.RetryInterceptor.intercept(RetryInterceptor.java:43)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                                       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                                       at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
                                                                                       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135)
                                                                                       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
                                                                                       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113) 
                                                                                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588) 
                                                                                       at java.lang.Thread.run(Thread.java:818)

The text was updated successfully, but these errors were encountered:

danielvtan · 2018-09-06T13:52:16Z

running the sample (no edits. just follow the instruction)
https://github.com/aws-samples/aws-mobile-appsync-events-starter-android

produces the same error.

steps:
turn off internet
create event
after creating event, turn internet on

rohandubal · 2018-09-06T18:11:18Z

Does this only happen on first offline mutation or all offline mutations? Do the normal mutations go through fine?

danielvtan · 2018-09-06T18:20:38Z

This happens to all of the offline mutation. Sometimes the first offline mutation go in.
Normal mutations are fine.

Take note. the aws appsync android sample also does not work. Same error.

I have a solution that is currently working but after about a hundred of call it fails.
too many open files error. or sometimes a syqlite error
i removed the client checking so now my app always creates a new appsync client.
doing this made it work not sure why. but eventually ffails so i need a real solution here.

//if(client == null){
            client = AWSAppSyncClient.builder()
                    .context(context)
                    .cognitoUserPoolsAuthProvider(getUserPoolAuthProvider(context))
                    .region(Constants.APPSYNC_REGION)
                    .serverUrl(Constants.APPSYNC_API_URL)
                    .persistentMutationsCallback(persistentMutationCallback)
                    .build();
//}
``

danielvtan · 2018-09-06T18:58:29Z

I tested a bit more.
Doing offline mutation closing the app and opening it up again then turning the internet makes it work.
So basically i think the persistent cache is working. The in memory is the one that fails.

I think that is also the reason why my temp solution works because it forces appsync to store the mutation in the persistent cache.

rohandubal · 2018-09-06T19:14:21Z

Thanks for the details @danielvtan I will investigate the issue.

danielvtan · 2018-09-09T17:44:03Z

I got a temporary fix working. I edited the appsync sdk a bit. Here is what i did.

I removed the in memory mutation by removing this line

//AppSyncOfflineMutationManager.java -> addMutationObjectInQueue
inMemoryOfflineMutationManager.addMutationObjectInQueue(mutationObject);

to replace the in memory mutation i always call this after a mutation.

//PersistentOfflineMutationManager
persistentOfflineMutationObjects = fetchPersistentMutationsList();
        persistentOfflineMutationObjectMap = new HashMap<>();
        for (PersistentOfflineMutationObject object: persistentOfflineMutationObjects) {
            persistentOfflineMutationObjectMap.put(object.recordIdentifier, object);
        }
        Log.d("AppSync", "There these many records in persistent cache: " + persistentOfflineMutationObjects.size());

the purpose of the code above is to ditch the in memory mutation entirely
after that here is the actual temp solution
in AppSyncCustomNetworkInvoker.java -> executeRequest i call this in the run function

OkHttpClient okHttpClient = (OkHttpClient) httpCallFactory;
okHttpClient.connectionPool().evictAll();

after doing this, offline and online mutation are finally working.
for the full explanation i am not really sure but i got the idea from this issue square/okhttp#3146 which is still open and not resolved

Please make an actual fix for this. thank you!

donaldarmstrong · 2018-11-15T01:26:02Z

I see the same exception... The mutation is longer available and our client and backend are out of sync.

D/AppSync: onFailure()Failed to execute http call
D/AppSync: Got message to take action regarding mutation queue.
D/AppSync: Got message that a originalMutation process errored out.
D/AppSync: Checking if I need to process next originalMutation
D/AppSync: First check: Internet Available
D/AppSync: Second check: Persistent mutations queue is EMPTY!
D/AppSync: Third check: Inmemory mutations queue is EMPTY!

2018-11-14 19:24:13.381 10200-10261/com.regiscorp.athena.pos E/TransactionSDKImpl: Failed to execute mutation com.apollographql.apollo.exception.ApolloNetworkException: Failed to execute http call at com.apollographql.apollo.internal.interceptor.ApolloServerInterceptor$1$1.onFailure(ApolloServerInterceptor.java:105) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:148) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) at java.lang.Thread.run(Thread.java:761) Caused by: java.net.SocketTimeoutException: timeout at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:593) at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:601) at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:146) at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at com.amazonaws.mobileconnectors.appsync.sigv4.AppSyncSigV4SignerInterceptor.intercept(AppSyncSigV4SignerInterceptor.java:182) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at com.amazonaws.mobileconnectors.appsync.retry.RetryInterceptor.intercept(RetryInterceptor.java:48) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) at java.lang.Thread.run(Thread.java:761)

scb01 · 2018-12-05T20:56:49Z

@danielvtan @donaldarmstrong

I am investigating this issue currently. My working theory at this point is that the mutation is being removed from the queue before it is successfully processed as reported in #40 and is lost if the operation fails due to a network error.

I am testing a couple of fixes locally that I hope to include in an upcoming release of the SDK. I will post back on this thread once I have more updates.

scb01 · 2018-12-19T04:41:06Z

@danielvtan @donaldarmstrong

Version 2.7.1 of the SDK contains the queuing fixes logic to address the issue that you reported. Please upgrade to this version and let us know how you fare.

scb01 · 2018-12-30T17:49:44Z

@danielvtan @donaldarmstrong
Looks like you are good to go. I'll close out this thread - please feel free to post back here or create a new thread if you are still running into issues.

danielvtan · 2018-12-31T17:56:04Z

@cbommas thanks for the fixes. Currently i can not test this as we are using a modified version of the sdk. We will test this for our next project.

desokroshan assigned scb01 Sep 10, 2018

frankmuellr added AppSync and removed AppSync labels Sep 28, 2018

frankmuellr added the bug Something isn't working label Oct 15, 2018

scb01 mentioned this issue Dec 8, 2018

AppSync deletes mutation before calling http that leads to lost of data #40

Closed

scb01 added the pending-response Issue is pending response from the issue requestor label Dec 19, 2018

scb01 added the closing-soon label Dec 26, 2018

scb01 closed this as completed Dec 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http fails to execute after offline mutation #33

http fails to execute after offline mutation #33

danielvtan commented Sep 6, 2018 •

edited

danielvtan commented Sep 6, 2018 •

edited

rohandubal commented Sep 6, 2018

danielvtan commented Sep 6, 2018 •

edited

danielvtan commented Sep 6, 2018

rohandubal commented Sep 6, 2018

danielvtan commented Sep 9, 2018 •

edited

donaldarmstrong commented Nov 15, 2018 •

edited

scb01 commented Dec 5, 2018

scb01 commented Dec 19, 2018

scb01 commented Dec 30, 2018

danielvtan commented Dec 31, 2018

http fails to execute after offline mutation #33

http fails to execute after offline mutation #33

Comments

danielvtan commented Sep 6, 2018 • edited

Please include a stacktrace if applicable.

danielvtan commented Sep 6, 2018 • edited

rohandubal commented Sep 6, 2018

danielvtan commented Sep 6, 2018 • edited

danielvtan commented Sep 6, 2018

rohandubal commented Sep 6, 2018

danielvtan commented Sep 9, 2018 • edited

donaldarmstrong commented Nov 15, 2018 • edited

scb01 commented Dec 5, 2018

scb01 commented Dec 19, 2018

scb01 commented Dec 30, 2018

danielvtan commented Dec 31, 2018

danielvtan commented Sep 6, 2018 •

edited

danielvtan commented Sep 6, 2018 •

edited

danielvtan commented Sep 6, 2018 •

edited

danielvtan commented Sep 9, 2018 •

edited

donaldarmstrong commented Nov 15, 2018 •

edited