SVC Init failed on getting repo, now tries to create all failed repos.

[h5aaimtron]

I was running a simple copilot init command to generate my manifest and prep for a service deployment, however; the proposed changes to the stack failed trying to get the arn of the ecr repo and rolled everything back. Of course, this left values in the Parameter Store because it wasn't a true rollback. I cleared those out and tried renaming the service. Same deal except this time it tried to create an ECR repo for the new service name and the previous service name. Now I'm stuck, unable to initialize my service.

[h5aaimtron]

I ran the svc delete to delete the failed service reference and it says its creating the repos then deleting them. It's really weird.

[Lou1415926]

Hello @h5aaimtron !

I have a few questions to help us understand the problem you are facing:

1. Is this a new app?
2. What version of copilot you are running?
3. What is the exact error that you are seeing?

Of course, this left values in the Parameter Store because it wasn't a true rollback.

Adding values to the parameter store is the last step of svc init; therefore, if the previous steps fail, this wouldn't have happened. I suspect the values that you saw in SSM were something left over from operations that happened before, not by this particular one.

[h5aaimtron]

Hi @Lou1415926 !

Not a problem, I'll do my best to answer them:

1. It's new in that its a week old with 2 services already added to it.
2. Version 1.24.0
3. Unable to retrieve Arn attribute for AWS::ECR::Repository, with error message null

On #3 there happening, it deletes the ECR repository but leaves the service name in the parameter store. It also attempts to create the ECR repositories from the previous attempts (changed the name a few times thinking there was an issue)

Even with the roll back in the above, the parameter store shows the service:

Note: this was the first time I used the name test-service so it could not have existed previously.

[Lou1415926]

Super weird indeed 🤔

Would you mind going to the CFN console, go to "StackSets" tab

Select finance-infrastructure and go to the Template tab:

I am curious to know what do you see under Metadata.Services?

Also, one additional question that may or may not be helpful 😂 : are finance-dev-sso and ucla-sso the "previous attempts" mentioned here "It also attempts to create the ECR repositories from the previous attempts (changed the name a few times thinking there was an issue)"?

[h5aaimtron]

Not a problem:

Services:

• chartmapperapi
• chat-mapper-api
• cloud-hub-sampl-spa-dotnet
• finance-dev-sso
• test-service
• ucla-sso

Yeah those are previous attempts that failed with the can't get ECR Repo error. After each error I attempted to delete the svc created, but they'd keep showing.

[Lou1415926]

Thank you!

I think I have some clues what the issues are now. Just a few more troubleshooting - could you:

1. Go to the CFN console at us-west-2
2. Look for a stack named StackSet-finance-infrastructure-<random>
3. Look at the Events tab and look for "Status Reason"

If possible, I'd like to know if there are some interesting "Status Reason" there. I think there should be a few "CREATE_FAILED" or "UPDATE_FAILED" event there. If you scroll down, you can probably find the "Status Reason" for the first fail. If you are comfortable sharing a screenshot of that portion, it'd be helpful as well!

[h5aaimtron]

This is where it started failing:

[Lou1415926]

Do you happen to have used permissions boundary when creating the app?

[h5aaimtron]

@Lou1415926 not to my knowledge. I never used that flag and probably wouldn't generally speaking.

[Lou1415926]

@KollaAdithya found out that if you have manually deleted one of your ECR repo (likely the repos for the two services that you created before and wanted to keep), you can get this Unable to retrieve Arn attribute error! Would you please confirm if you are missing any ECR repo in "us-west-2" ?

[Lou1415926]

I will first explain what I think the problems are (also for our reference!), and then I will talk about my recommended actions for you!

The Cause(s)

I think there are two independent issues at play:

1. One issue causes "previous attempts" to keep showing
2. The other issue caused the "Unable to retrieve Arn attribute for AWS::ECR::Repository" error

Previous Attempts Keep Showing

When you init a service, Copilot calls UpdateStackSet to update the template for the stackset, i.e. finance-infrastructure: it adds the service to Metadata.Services, as well as adding an ECR repository for that service. The UpdateStackSet call will update the template, as well as update the stackset instance - in your case, it should be a stack named StackSet-finance-infrastructure-<random> in us-west-2.

However, the UpdateStackSet doc says

Even if the stack set operation created by updating the stack set fails (completely or partially, below or above a specified failure tolerance), the stack set is updated with your changes. Subsequent CreateStackInstances calls on the specified stack set use the updated stack set.

To apply this In your situation:

1. Your stackset instance StackSet-finance-infrastructure-<random> fails to update for some reason[1], and is rolled back.
2. However, although the stackset instance fails, CFN keeps the template that has "finance-dev-sso" and "ucla-sso" for the stackset finance-infrastructure
3. Because UpdateStackSet does not error out, Copilot thought the operation succeed, and proceeded to add SSM.
4. When you run copilot svc init --name test-service, Copilot adds test-service to the list of Metadata.Services, which still has "finance-dev-sso" and "ucla-sso".

Unable to retrieve Arn attribute for AWS::ECR::Repository

The issue described above ⬆️ explains why Copilot keeps trying to create ECR repos from previous attempt, but it does not explain:

1. Why the stackset instance failed to update in the first place [1]
2. What is causing the error "Unable to retrieve Arn attribute for AWS::ECR::Repository"

Due to the finding in #4296 (comment), it is possible that you have deleted an ECR repo for one of the services. When you run copilot scv init, CFN still tries to grab the ARN of that repo, which is in fact already gone.

---

What you can do to recover

You will need to first identify which ECR repo(s) is(are) missing. If you are comfortable with deleting that service, then you can copilot svc delete --name <the service for which the ECR repo was manually deleted>.

After that, you should be able to delete the other unwanted services that you created along the way one by one (i.e. usla-sso, finance-dev-sso and test-service)

However, if you don't want to delete that service, then there would be a bit of work to do:

1. You need to create an ECR repo
2. You need to import that ECR repo to the CFN template, so that the CFN knows which repo to query for ARN. You can follow the instruction here
3. You can then delete the other unwanted service one by one

---

Improvement on our side

Due to UpdateStackSet, there are two confusing behaviors in Copilot:

1. A service still shows up in Metadata.Services although the attempt to init that service has failed
2. The SSM parameter is added even though the stackset instance has failed to update

One thing that we can do to improve this is to not rely on the error code returned by UpdateStackSet to tell if the update has completed. We can check the status of each individual stack instance instead.

[h5aaimtron]

What you've written makes sense, however; the failure to get the ecr arn happened without deleting an ecr repository. This was a "clean" app for the most part with 2 services loaded. That error arose on creating a brand new service. After that, the issues of previous attempts arose. To solve the issue I removed everything which also required a lot of manual stack removals and stackset removal, etc. I got a clean environment and started over. Everything seems fine now.

Edit to clarify. The error about ecr arn happened prior to the "previous attempts". Then it happened everytime after.