Amplify deleted my resolvers and now my stack is stuck in UPDATE_ROLLBACK_FAILED

[cunneen]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

16

Amplify CLI Version

10.8.1

What operating system are you using?

Amplify CI/CD (amazon linux)

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made before the issue described was encountered. I've since attempted (unsuccessfully so far) to manually attach resolvers via the appsync console to rectify the issue.

Describe the bug

After a couple of failed amplify pushes, reverting my failed changes and re-pushing the last successful amplify push resulted in my stack having an UPDATE_ROLLBACK_FAILED state, because somehow the (generated) appsync resolvers are now missing.

How I got here

In case it helps explain how this happened, what I was trying to do (in our non-production stack) was to modify four existing appsync @keys (i.e. dynamodb GSIs) in our graphql schema, with the eventual goal of adding extra fields to the sortkeys.

Understanding that I'd need to delete and then re-create the GSIs, I commented out the @key statements , then tried to amplify push (via CI/CD),

i.e.:

index c06064e3..6326f8bc 100644
--- a/amplify/backend/api/bpedsysgql/schema.graphql
+++ b/amplify/backend/api/bpedsysgql/schema.graphql
@@ -2707,16 +2707,17 @@ Table: This table stores handwriting uploads log made by users.
 """
 type HandwritingLog
   @model(subscriptions: null)
-  @key(
-    name: "byActivity"
-    fields: ["activityID"]
-    queryField: "getHandwritingLogByActivityID"
-  )
-  @key(
-    name: "byUploadUserID"
-    fields: ["uploadUserID"]
-    queryField: "getHandwritingLogByUserID"
-  )
+  # # TODO: These indexes have been temporarily commented so that they will be removed upon deployment, so that we can then re-create them with changes.
+  # @key(
+  #   name: "byActivity"
+  #   fields: ["activityID"]
+  #   queryField: "getHandwritingLogByActivityID"
+  # )
+  # @key(
+  #   name: "byUploadUserID"
+  #   fields: ["uploadUserID"]
+  #   queryField: "getHandwritingLogByUserID"
+  # )
   @auth(
     rules: [
       {
@@ -2749,16 +2750,17 @@ and with this information the teacher will be able to match exceptions through t
 """
 type StudentHandwritingLog
   @model(subscriptions: null)
-  @key(
-    name: "byStudentID"
-    fields: ["studentID"]
-    queryField: "getStudentHandwritingLogByStudentID"
-  )
-  @key(
-    name: "byLogParentID"
-    fields: ["uploadID"]
-    queryField: "getStudentHandwritingLogByLogParent"
-  )
+  # # TODO: These indexes have been temporarily commented so that they will be removed upon deployment, so that we can then re-create them with changes.
+  # @key(
+  #   name: "byStudentID"
+  #   fields: ["studentID"]
+  #   queryField: "getStudentHandwritingLogByStudentID"
+  # )
+  # @key(
+  #   name: "byLogParentID"
+  #   fields: ["uploadID"]
+  #   queryField: "getStudentHandwritingLogByLogParent"
+  # )
   @auth(
     rules: [
       {
-- 

... then amplify push, but ... oops (my bad), I'd missed a reference the byLogParentID key/query, broken the schema and so the push failed with this message:

2023-12-22T09:51:09.154Z [WARNING]: ✖ There was an error initializing your environment.
2023-12-22T09:51:09.156Z [INFO]: 🛑 Key byLogParentID does not exist for model StudentHandwritingLog
                                 Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/
2023-12-22T09:51:09.176Z [INFO]: 
2023-12-22T09:51:09.176Z [INFO]: Session Identifier: cebc0b6d-774b-4bac-be2d-f7b4d4afc804

So, I fixed the schema and tried to push again:

index 6326f8bc..8458b431 100644
--- a/amplify/backend/api/bpedsysgql/schema.graphql
+++ b/amplify/backend/api/bpedsysgql/schema.graphql
@@ -2739,8 +2739,9 @@ type HandwritingLog
   fileUrl: String
   # the original file name that was processed from the activity folder.
   uploadedFileName: String
-  studentLogs: [StudentHandwritingLog]
-    @connection(keyName: "byLogParentID", fields: ["id"])
+  # # TODO: These indexes have been temporarily commented so that they will be removed upon deployment, so that we can then re-create them with changes.
+  # studentLogs: [StudentHandwritingLog]
+  #   @connection(keyName: "byLogParentID", fields: ["id"])
 }
 
 """
-- 

... and then amplify push ....

But no... this time I encountered Cannot perform more than one GSI creation or deletion in a single update :

UPDATE_FAILED               HandwritingLogTable \
    AWS::DynamoDB::Table       Fri Dec 22 2023 10:26:39 GMT+0000 (Coordinated Universal Time) Resource handler returned message: "Cannot perform more than one GSI creation or deletion in a single update" (RequestToken: 471e38c3-2061-cf78-2ed3-ff6bb0b78735, HandlerErrorCode: InvalidRequest)

So I edited cli.json to enable the enableIterativeGsiUpdates feature and tried another amplify push. Only to get the same error (!):

UPDATE_FAILED      HandwritingLogTable  \
    AWS::DynamoDB::Table       Fri Dec 22 2023 11:04:19 GMT+0000 (Coordinated Universal Time) Resource handler returned message: "Cannot perform more than one GSI creation or deletion in a single update" (RequestToken: 3f4c47b0-22a4-3052-7ca8-19b24b59646b, HandlerErrorCode: InvalidRequest)

"Maybe it's a sequencing issue" I thought, reasoning that perhaps the enableIterativeGsiUpdates feature needed to be enabled and pushed separately before attempting to change the GSIs.

this guy seemed to come to a similar conclusion and said that it worked for him.

So at this point, I'm adding the GSIs back temporarily so I can then try to remove them again before adding them back again. Obviously.

So I tried to "revert" all of the schema.graphql schema changes (which, since the previous deployments had failed, should result in no Appsync changes at all, right? RIGHT?)

WRONG.

No resolver found, apparently. When simply re-pushing my last successful push WITH NO CHANGES (other than the feature flag).

UPDATE_FAILED               QuerygetStudentHandwritingLogByLogParentResolver                                              AWS::AppSync::Resolver     Fri Dec 22 2023 11:33:30 GMT+0000 (Coordinated Universal Time) Resource handler returned message: "No resolver found. (Service: AppSync, Status Code: 404, Request ID: 74475317-ef65-49be-8b0c-b61d76bc5511)" (RequestToken: 8813f388-4e3b-b79e-773e-b07a2b29e73e, HandlerErrorCode: UPDATE_FAILED               QuerygetStudentHandwritingLogByStudentIDResolver                                              AWS::AppSync::Resolver     Fri Dec 22 2023 11:33:31 GMT+0000 (Coordinated Universal Time) Resource update cancelled
UPDATE_FAILED               ListStudentHandwritingLogResolver                                                             AWS::AppSync::Resolver     Fri Dec 22 2023 11:33:31 GMT+0000 (Coordinated Universal Time) Resource update cancelled
UPDATE_FAILED               GetStudentHandwritingLogResolver                                                              AWS::AppSync::Resolver     Fri Dec 22 2023 11:33:37 GMT+0000 (Coordinated Universal Time) Resource update cancelled
UPDATE_ROLLBACK_IN_PROGRESS amplify-bpedsys-develop-174413-apibpedsysgql-1FXK37D4UZQ8L-StudentHandwritingLog-AU2Q0XXR10Z3 AWS::CloudFormation::Stack Fri Dec 22 2023 11:33:38 GMT+0000 (Coordinated Universal Time) The following resource(s) failed to update: [ListStudentHandwritingLogResolver, GetStudentHandwritingLogResolver, QuerygetStudentHandwritingLogByStudentIDResolver, QuerygetStudentHandwritingLogByLogParentResolver].

Expected behavior

"Insanity is doing the same thing over and over and expecting different results".

I expected that pushing the exact same graphql schema that had worked before, would work again.

Reproduction steps

Please see the "describe the bug" section above.

Project Identifier

Running the provided command doesn't give a project identifier, nor does it seem to send a zip:

amplify diagnose --send-report

Learn more at https://docs.amplify.aws/cli/reference/diagnose/

✅ Report saved: /var/folders/67/334sg0_90q3dm7t1q0ncxwtm0000gn/T/bpedsys/report-1703346831036.zip

⠴ Sending zip

✖ Sending zip

Log output

the "push"-es were all performed within amplify CI/CD so that's where the logs are;

region is ap-southeast-2
amplify app ID is d31ul62nbvaagu
amplify app environment is dev
the build numbers referred to in the "describe the bug" section are: 1868 through to 1871.

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[cunneen]

As others have reported, using the appsync console to create (dummy) resolvers allowed me to then "continue update rollback" successfully, and my stack is now in a usable UPDATE_ROLLBACK_COMPLETE state. The disappearance of these resolvers and cloudformation's inability to roll back the changes is still a cause for major concern. If this happens on our production environment it will be disastrous; it would be comforting to know if this gets treated as a priority.

[giuliobracci]

As others have reported, using the appsync console to create (dummy) resolvers allowed me to then "continue update rollback" successfully, and my stack is now in a usable UPDATE_ROLLBACK_COMPLETE state. The disappearance of these resolvers and cloudformation's inability to roll back the changes is still a cause for major concern. If this happens on our production environment it will be disastrous; it would be comforting to know if this gets treated as a priority.

saved my day, thank you.

[8maxxam8]

@cunneen I am having a similar issue! After major changes in my graphQL schema and some layer. Can you please describe how you create "dummy" resolvers? Thanks a lot!🙏

[BradVidler]

This saved me as well. I was changing a schema model in my Amplify Flutter app and some resolvers disappeared. Specifically, I was trying to allow public subscriptions. Could not force the stack to rollback even when skipping resources. The process to fix it is:

1. Go to the AWS AppSync console.
2. Select your API.
3. Navigate to the "Data Sources" section.
4. Create a new data source of type "None". You can name it NoneDataSource.
5. Go to the "Schema" section in the AppSync console.
6. Locate the subscription type in your schema. If it doesn't exist, you will need to define it. For example:
   type Subscription {
   onCreateTodo: Todo
   @aws_subscribe(mutations: ["createTodo"])
   onDeleteTodo: Todo
   @aws_subscribe(mutations: ["deleteTodo"])
   }
7. For each subscription field, you will need to attach a resolver:
   Copy onCreateTodo/onDeleteTodo 1 at a time and paste into the filter on the right hand side where resolvers are. Click "Attach" and attach the NoneDataSource from earlier.
8. You can now roll back the stack in cloudformation. You don't need to skip these resources.

[cunneen]

Sorry @8maxxam8 for the delay in responding, and thanks @BradVidler for sharing his steps. From memory, my process was very similar but we weren't using subscriptions, and I didn't need to define any subscriptions in the graphQL schema.

I think my process was probably not very efficient, but here's how I did it.

1. Look at the error messages from the failed CloudFormation deployment to find which resolvers for graphQL types couldn't be created:

   1. In the AWS web console, in Amplify, navigate to your amplify application's back-end, and click View in CloudFormation
   2. In your CloudFormation stack, navigate to the Resources tab, and search for your GraphQL API's name (e.g. mine is called "bpedsysgql", so my amplify project has a amplify/backend/api/bpedsysgql folder. Click on the link under Physical ID, it will open a new browser tab for the CloudFormation sub-stack for your GraphQL API.
   3. Repeat the process: click on the Resources tab, and find the CloudFormation sub-stack that defines your DynamoDB table and related GraphQL type that failed, and click on the Physical ID link
   4. This time, click on the Events tab, and search through the error messages to figure out which relation in your GraphQL schema has missing resolvers.

2. In AppSync, go to the Schema tab, and in the right-hand pane, SCROLL to find the type whose field is missing a resolver. See the attached picture.

   • search doesn't seem to work
   • either use a large monitor with high resolution, or zoom out.
   

3. Create a new resolver for the field as per the attached picture:

   • I used "Unit Resolver" and "VTL"
   • Choose the appropriate table name from the list of data sources
   • Click "Create" and the "dummy" resolver should be created.
   

4. Repeat this process for each missing resolver.

5. Redeploy.

[8maxxam8]

@BradVidler @cunneen Thanks a lot for your help! This was indeed solving the problem I was facing!👍